Reading Data into s-Server

<< Click to Display Table of Contents >>

Navigation:  Integrating Blaze with Other Systems >

Reading Data into s-Server

Previous pageReturn to chapter overviewNext page

Before you can query, analyze, or set alerts on your data, you need to read this data into s-Server. This integration guide describes a number of ways to ingest data. All data ingested ends up in an s-Server foreign stream. A foreign stream is an s-Server object that serves as a "tank" to hold streaming data. Once you have established a foreign stream, you can pump data from the foreign stream to native streams, where you can analyze data, set alerts on data, visualize data, and so on.

There are two basic categories for reading data into s-Server:

Reading data from RDBMS sources. You can use s-Server to read directly from RDBMS tables. Since these data are already in rows and columns, no additional parsing of data is necessary to get it into an s-Server stream. See Reading Data from RDBMS Sources in this guide.
Reading data from all other sources using the Extensible Common Data framework. This framework lets you identify a data source, such as a file from the file system, a network socket feed, an Apache Kafka topic, an Amazon Kinesis stream, an IBM MQ queue, an AMQP message. s-Server needs to parse data from these sources. In s-Server, parsing means to translate data into rows and columns. s-Server can automatically process data in a number of common formats: CSV, XML, JSON, key pair, and Google Protobuf. For other formats, you can make a custom parser using Regex (these are "regular expressions" for creating search strings). See Reading Data from Other Sources in this guide, as well as the specific topics Reading from the File System, Reading from Network Sockets, Reading from AMQP, Reading from Kafka, Reading from Kinesis, Reading from IBM MQ, Reading over HTTP, Reading over Web Sockets, and Reading from Web Feeds.

Configuring Input

For all of the above sources, you can configure input in three ways. This guide focuses on the third option, creating a foreign stream by hand in SQL, but the principles described within apply to both StreamLab and s-Studio.

1.Using StreamLab, a graphical interface, to set up a connection with a source. For more details, see Adding Sources in StreamLab in the StreamLab Guide. (You can still connect with sources that you've created in StreamLab using SQL.)
2.Creatin a foreign stream in s-Studio, SQLstream's Integrated Development Environment. For more details, see Creating Objects in s-Studio in the SQLstream s-Studio Guide.
3.Creating a foreign stream by hand in SQL, then running this SQL in s-Server using sqlline. This process is described below under topics for each source. The Streaming SQL Reference Guide provides configuration details in the topic CREATE FOREIGN STREAM. For information on using sqlline, see the Using sqlline to Run SQL in s-Server.