SQLstream 2.5 – Real-time stream computing eliminates Big Data performance and storage bottlenecks

With service and sensor data growing at 60% CAGR, having both the raw power and correct architecture for processing streaming data is essential. IDC released recently estimates for the size of the ‘Digital Universe’ – a term used to describe every electronically stored piece of data. According to IDC, stored data will reach 1.8 million petabytes (1800 exabytes) by the end of 2011.

As a recent article in the Economist points out, all of this data raises significant processing performance and storage issues. Conventional database technology requires data to be stored, cleaned and aggregated before being queried. With the volume of data growing so quickly, it has become cost prohibitive and technologically infeasible to process all data using conventional solutions.

But how much of the raw data actually needs to be stored? The value of individual data is often low, and the useful lifetime of the raw data short. However, the information content is potentially high – it’s just a matter of identifying the valuable information in the raw data.

Introducing SQLstream Server 2.5

For SQLstream, this is the future of data processing – real-time, continuous analysis of streaming data – generate operational business intelligence from live streaming data without first storing the data in a database.

For the latest release of SQLstream Server, SQLstream 2.5, we’ve focussed on the common business requirements that are required for the rapid adoption of real-time stream computing across all markets – performance, reliability and scalability. More specifically, SQLstream 2.5 offers:

• 10X performance improvement, benchmarked against live operational deployments on a single server installation.
• Scalability for mission critical applications with federated installations across multiple servers.
• Business critical reliability following an exhaustive stability and operational optimization program.

Of course, we’ve also addressed a range of important requirements across our customer base, in particular, additional input and output connectors built on the SQL/MED standard for integration, including:

• enhanced database insert/update/select Adapters.
• enterprise messaging integration using AMQP.
• enhanced Log File management and XML feed processing Adapters.

And last but by no means least, supporting the SQL:2008 standards-based streaming SQL language with new functions including:

• support for GROUP BY ORDER BY.
• new and enhanced data analysis functions for detecting unique events, such as early emit SELECT DISTINCT.
• support for the SQL HAVING function.
• and a new range of streaming statistical functions for calculating variance and standard deviation.

Most existing customers have already upgraded to SQLstream 2.5. Some examples of recent SQLstream 2.5 upgrades include customers in the following markets:

Environmental monitoring and event detection – integrating with AMQP, which provides the guaranteed delivery of real-time raw data from a large sensor network, SQLstream filters (using windowed aggregation) the raw sensor and applies event detection patterns in real-time, generating a continuous stream of environmental exceptions events.

Social gaming infrastructure – working with a new entrant in the on-line social gaming market, SQLstream monitors user activity and provides continuous real-time scoring updates – including real-time incremental updates of historical, aggregated game data maintained in a back-end data warehouse.