The Internet of Things and Next Generation Big Data management with stream processing

Many of the growth charts and predictions for the Internet of Things  originated from a Cisco paper in 2011,  which stated that even by 2008 or so, there were already more connected devices than people on the planet, and that the number of connected devices would grow 25% year on year to get to 50 billion by 2020. More recent predictions from Gartner place Internet of Things growth at 26 billion by 2020, but importantly for the smart services market, breaking down the overall predicted market value for Internet of Things of $1.9 trillion dollars, to estimate a product and services industry of around $300 billion. Gartner says that the three most important drivers for Internet of Things projects are now new revenue streams and enabling new services, as well as operational efficiency and then improving existing services and the consumer experience.

Smart Services for Real-time Internet of Things applications

Gartner uses the term “Smart Services” to describe these new types of real-time services, meaning services that are capable of managing, monetizing and operating on connected devices and data. Smart services must run Big Data analytics and execute in real-time, be available 24×7,   and offer a platform for building multiple, integrated smart services. They must also be able to collect sensor and machine-generated data, and be able to support and integrate with existing enterprise systems and applications.

IoT-SmartServicesHowever, with estimates that by 2020 suggest that over 50 billion devices will be creating endless amounts of data. This presents a significant challenge as the volume of data generated will begin to outstrip the capacity to store the data and much of the data requires real-time interaction and response within a timeframe that is beyond today’s storage-based data management platforms. Therefore the opportunities across the Internet of Things, depend on our ability to deliver intelligent services across the network of connected devices, and to achieve this in real time.

Stream Processing An IOT Platform for Smart Services

Businesses need to be able to collect and manage their IoT data in real-time. Data must be collected, cleaned, analyzed, compared against trend data, visualized and analytics integrated with existing systems. The difference is that to achieve this in real-time and at the large data volumes, requires new technology such as stream processing.

A stream processor will:

  • analyze the arriving sensor and device data in real-time
  • provide comparisons against trend data by joining the real-time data with historical, stored data
  • provide real-time dashboards and alerts for visualization
  • provide continuous streaming integration with operational systems for process automation
  • and will streams the data through into existing Hadoop and data warehouse platforms.

Essentially, stream processing is to the Internet of Things, what Big Data and Hadoop has been to the Internet.