Strata 2014, stream processing and the future of Big Data
It was evident at the O’Reilly Strata Conference, Feb 11th through 13th, that stream processing is now a mainstream Big Data technology. In fact, judging by the level of interest, stream processing for real-time analytics over machine data is right up there as the main Big Data trend in 2014.
The tag line on our booth: “Big Data Stream Processors,” caused many of those walking by to stop and ask for more detail. We asked them to first describe their situation and needs, and we heard many stories about how data volumes, particularly machine data, are growing rapidly and that the time interval to react to the data keeps getting shorter. A constant theme was: “we can handle it now, barely, but it is clear that we are going to have to do something differently soon.”
Music to our ears because that’s where a stream processor comes in. Attendees at Strata are generally up on the latest Big Data developments, and for the first time, we didn’t need to spend any time defining streams and explaining how they are not just data coming from a database. Instead, customers know they have or will soon have Big Data streams and wanted to know how a stream processor could help them solve their problems.
Understanding Stream Processors
A stream processor sits in the data path between the sources of machine data and in front of the backing stores. So no rip and replace is required. Incoming data streams are analyzed and can be acted upon without delay; sub-second response times are easy for a SQLstream stream processor, even at a tens or hundreds of thousands, even millions of events per second. The incoming streams can be forked off and stored raw in whatever persistent storage is already in use. The processed streams can be stored as well for later restreaming (restreaming is the fast-forward analysis of stored time-series data).
“Sounds exactly like what we need,” we often heard, “but I’m not sure I’ve got time to ramp up on a new system.” We explained our use of ANSI-standard SQL in a high velocity streaming environment (not a big surprise given our name) and they realized that their existing technical resources could be comfortable supporting stream processing of their high volume, velocity and variety Big Data streams.
Streaming SQL for Storm
Strata attendees asked us about Apache Storm, and we explained our support for Storm installations, bringing higher performance and ease-of-use to the popular platform. While Storm isn’t necessary to use SQLstream, we believe that SQLstream ensures Storm implementations reach their full potential – faster performance, less hardware, dynamic updates and faster time to value. Read about the SQLstream Stream Processor for Storm here.
Announcing SQLstream 4.0
Existing users of SQLstream were happy to hear that we introduced version 4.0 at the show, liking very much improvements in performance, visualization capability, and interoperability. SQLstream 4.0 also includes our APIs for Storm integration and a new HTML5 real-time dashboarding tool. Read more about SQLstream 4.0 here, or download and try out s-Server 4.0 here.
Strata turned out to be a great show. It appears very clear that the future SQLstream has long imagined, one where streams of data are used intelligently to drive business value in real time, is starting to come true.