Batching Vs Streaming - John Deere's Journey to Scale & Process Millions of Measurements a Second

The John Deere data platform receives and processes millions of sensor measurements per second from machines around the world. In this talk, I will discuss the various technologies and architectures we have implemented and how we had to adapt to ever increasing volumes of data. I will explore our usage of Lambda/Kinesis, Spark, and Flink... where they worked, where they didn't, lessons learned and how our current architecture is a blend of all three.