Real-time Big Data analytics and millions of events per second

Visit our new website to find out more about real-time Big Data applications

Big Data is here to stay. The breadth of the term Big Data may change as it becomes as much a marketing imperative as the ‘Cloud’ word, but the requirement for ‘supercomputing lite’ processing for the non-supercomputing world of enterprise data is a must have.

The rise of Big Data has happened in parallel with the emergence of real-time operational intelligence, and the extension of real-time analytics into the world of real-time updates and process control. Much of the recent interest has focussed on how these two worlds merge into a single complementary solution.

The NoSQL BigData platforms offer massively scalable, resilient data processing over commodity hardware. Ideally suited to scaling large scale data problems over hundreds or thousands of servers. However, platforms such as Hadoop do not support, nor were designed to support, real-time streaming data processing and analytics. Their forte is the batch-based, highly scalable, store-compute loop of map/reduce.

That’s where SQLstream comes in. SQLstream collects and conditions real-time updates from sources such as log files, sensor networks and GPS events, and both integrates streaming data into and from Big Data stores, but also generates real-time analytics from the data as they stream past. The SQLstream architecture also has parallels to that of map/reduce. SQLstream uses Relational Streaming, which is a paradigm for processing streaming Big Data tuples using standard SQL queries. SQL offers strong potential for automatic optimization and distributed parallel processing of streaming data. Whereas platforms such as Hadoop execute batch queries over stored tuples, SQLstream and Relational Streaming executes continuous queries over arriving data.

We’re also at Structure Data this week in New York, where our CEO, Damian Black, will be presenting on the wider area of streaming Big Data and massive scalability. However, if you are attending, visit us for a demo of the ‘millions of events per second” program, and a demonstration of massively parallel stream processing on an Elastic Compute Cloud.