Analyzing Data in s-Server
Purpose of this document
This document describes the basics of analyzing streaming data in s-Server, including writing streaming queries, using windowed aggregation, and moving data around in s-Server.
This document is intended for architects and developers who will analyze data in s-Server.
Streams are infinite, in that they will continually integrate new data as it arrives. For example, if a log file is updated at 5 minute intervals, and you have created a stream that points to the log file, the stream will continue to integrate updates to the log file as long as it is running.
In basic cases, querying streaming data is not different from querying static data. To filter for all rows that meet a specific criterion, for example, you can use a WHERE clause, just as you would for static data. The topic Writing Simple Streaming SQL Queries provides a few simple examples of streaming queries.
In most cases, though, you will need to make some considerations for the time dimension of streams. These considerations fall into three general categories "how far into the past do I want this data?" or "over what limited time window do I want this data?" or "what number of rows do I want to analyze" (There are some exceptions to these general categories.) These involve using windows, and the topic Windowed Aggregation on Streams describes how to use windows to aggregate data. You can also use windows to correlate streams in order to join them, as described in the topic
Moving Data Around in s-Server
As you develop your s-Server streaming application, you'll find that it is important to be able to move data around. In s-Server, you use pumps to get data from one stream to another. A pump a SQLstream schema object that provides a continuously running INSERT INTO stream SELECT ... FROM query functionality. The topic Using Pumps to Create Processing Pipelines describes how to use pumps to structure your applications.
You will also often want to join two streams together, as described in the topic Creating Streaming JOINs.
Bringing it All Together in Query Patterns
The topic Query Patterns describes some common use cases for streaming SQL queries.
Viewing Demonstration Applications
To help you understand the concepts above, s-Server ships with demonstration applications. The topic Running Demo Applications describes how to use these.
Using UDXs and UDFs for More Complicated Analysis
Some business problems will require you to create customized User Defined Transforms or User Defined Functions. The topic Using UDXs and UDFs for Analysis offers an overview of doing so.