2014 in Review – A Big Year for Big Data. Bigger for the Internet of Things.

This past year has seen Big Data technology mature into Enterprise-class platforms capable of delivering value in the largest of organizations. Hadoop storage platforms and stream processing are now core components of the standard Big Data enterprise architecture, with traditional RDBMS and warehouse platforms regaining their position in the limelight. And the death of SQL has apparently had been much over-stated.

Yet the driving force and major item of interest for 2014 was not Hadoop or stream processing or machine learning, or any other technology for that matter, rather the rise of machine and the Internet of Things. Strange in many ways, not least because sensors have been around much longer than computers. You’d think there was nothing new to learn. However the emergence of the Internet as a global connectivity platform has been irresistible to thought leaders, innovators and entrepreneurs. The lure of centralized command and control, a new era of previously unimaginable apps and platforms, and the promise of connecting everything, has had its own gravitational pull.

So were does that leave us at the end of 2014? Here follows an entirely one-sided and biased perspective.

SQL

SQLstream s-Server remains the only SQL standards platform for data stream management. There’s a reason for that. It’s much simpler to build a proprietary platform based on Java frameworks and proprietary languages without the constraints of standards compliance. However, ultimately, this is a fools errand. Longevity and enterprise-class adoption require standards, stability, reliability, and importantly, a platform that can automate the hard stuff such as distributed query optimization. SQL doesn’t imply structured data, it’s equally applicable for processing unstructured data. It’s just the underlying data management technology that differs.

Real-time

Stream processing when all is said and done is just a technology, an enabler for other things. What matters to Enterprises is the ability to deliver business value faster, and to improve operational efficiency through real-time actionable intelligence and operational process automation. Businesses are increasingly concerned about their ability to deliver real-time systems, whether for competitive differentiation, operational efficiency or optimal customer experience. The technology is not the major issue, it’s what it offers to customers that really matters. Stream processing offers real-time systems and automated processes with the lowest cost of performance. It’s horses for courses.

Internet of Things

The combination of sensors and the Internet is moving us into a new age of interaction with and understanding of the physical world around us. With the promise of real-time apps for connected cars, home automation, personal health, the industrial internet and smarter cities, we’re all intrigued as to what the Internet of Things offers. Real-time is a key requirement. For example, the ability to respond to a sensor event in real-time, to deliver connected car infrastructure that offers real-time updates, but importantly, the ability to automate processes in real-time. This is where a data stream management platform comes into play. There are of course many other components required, but the Internet of Things is being built on the full panoply of Big Data, Internet and Cloud technologies.

Finally, what does the IT world hold for 2015. There’s plenty of predictions out there. The one thing that remains constant is the need for business value. Once we’re past the hype, enterprises are most interested in what delivers business value and a compelling ROI. Big Data is here to stay, so too stream processing and standards, but primarily as the enablers for the future, the Internet of Things, and all who sail in it.