easy ingest
No hand-coding pipelines and flow; streaming ingestion allows all data to be moved continuously and in real time into and out of AWS through our high-performance bi-directional Kinesis Adapter.
ingestion and load
lower cost
Query-before-store reduces data flowing to the cloud and on-premise storage requirements, and the use of SQL allows for a maximum utilization of skills.
why SQL
real-time action
Live data streams are integrated and analyzed with historical data, record by record, and turned into actionable information within milliseconds.
automated actions
easy development
SQLstream Blaze offers an intuitive, interactive GUI based on open SQL and Java standards.
StreamLab
The combination of Amazon Kinesis Analytics and Guavus SQLstream makes it easier than ever to cost-effectively ingest, analyze, and manage streaming data on and between cloud and on premises.
Easy ingestion of data into and out of AWS: streaming data flows automatically from data sources, to the cloud, and from the cloud through a high-performance bi-directional Guavus SQLstream Kinesis Adapter developed jointly in collaboration between AWS and SQLstream.
Any data format, source or destination: all data is welcome – structured or unstructured, live or historical, in motion or at rest – from and to any data format, interfacing with a wide array of sources and destinations including Amazon Kinesis and Firehose, Hadoop, data warehouses, message buses (including Kafka), les, and devices.
Advanced data preparation and app development: parsing multiple, nested types of data is automatic and building complete apps from ingestion to analysis and into action, is made simple by accepting suggestions through an intelligent suggestions engine and intuitive visual tools.
Code compatibility: sophisticated SQL statements are automatically generated and allow users to cut-and-paste SQL queries for effortless utilization of common logic between SQLstream and Kinesis environments.
Accurate, contextual insight through batch or stream: live data streams are integrated and analyzed with historical data (on premises or in the cloud) for accurate business context, record by record, eliminating the risk of incomplete analysis for time sensitive decisions, due to batch or micro-batch processing.
Acting with millisecond precision: actions are operationalized through apps that deliver triggers and alerts based on real-time, record-by-record contextual analysis of live and historical data, including time-series and spatial operators.
Centralized streaming data visibility and security at global scale: data is viewed globally and flowing live through deployments using telemetry; data can be managed and marshalled for storage or analyzed without storing, for optimal value and security.
Zero downtime: continuous business operations are maintained while changing streaming application logic or queries, on-the-fly, while data is in motion.
The agent is started with code along the following lines. You can issue this code in SQLline or another JDBC client.
|
CALL SYS_BOOT.MGMT.START_AGENT_HOSTER(‘my_kinesis_agent’, ‘kinesis’, ‘SCHEMA_NAME=myschema& TABLE_NAME=mystream& PARSER=CSV& –The following are settings for the CSV parser. CHARACTER_ENCODING=ISO-8859-1& SKIP_HEADER=false& –The following are Kinesis-specific options AWS_REGION=us-west-1& AWS_PROFILE_NAME=my_profile& AWS_PROFILE_PATH=~/.aws/credentials& KINESIS_STREAM_NAME=test& KINESIS_APPLICATION_NAME=testrun3& STREAM_FANOUT=1& KINESIS_INITIAL_POSITION_IN_STREAM=TRIM_HORIZON& KINESIS_MAX_RECORDS_PER_GET=1500& KINESIS_SOCKET_TIMEOUT=-1& KINESIS_IDLE_TIME_BETWEEN_READS=-1′); |
The first parameter (“my_agent” above) is the name you are assigning to the agent for use in subsequent calls.
Second parameter (“kinesis” above) is the agent type – in this case kinesis.
Third parameter is either the path to a properties file or list of properties separated by ‘&’.
Agent Options
|
Option Name |
Description |
|
|
SCHEMA_NAME |
s-Server schema in which the native stream to which data will be read resides. You need to create this before running the agent. |
|
|
TABLE_NAME |
s-Server native stream to which data will be read. You need to create this before running the agent. |
|
|
KINESIS_STREAM_NAME |
Required. Name of Kinesis stream to read from. No default. |
|
|
AWS_REGION |
Required. Region id of Kinesis region. See http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region for more detalis. |
|
|
AWS_PROFILE_PATH |
Must point to a credential file on the s-Server file system with the following format: [default] aws_access_key_id = xxx aws_secret_access_key = yyy
This defaults to ”” – which goes to ~/.aws/credentials.
|
|
|
AWS_PROFILE_NAME |
Optional. Profile name to use within credentials file. Amazon supports multiple named profiles within a configuration file. If you have a named profile, you can reference it here. Defaults to default. Seehttp://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html |
|
|
KINESIS_INITIAL_POSITION_IN_STREAM |
LATEST for latest or TRIM_HORIZON for earliest. Defaults to LATEST. |
|
|
STREAM_FANOUT |
If > 1, instead of writing to TABLE_NAME will instead write to TABLE_NAME1 through TABLE_NAMEn depending on shard coming in. shard 1 will go to TABLE_NAME1. shard n will go to TABLE_NAMEn. shard n+1 will go to TABLE_NAME1 etc. |
|
|
KINESIS_APPLICATION_NAME |
Identifies client in cloud watch (defaults to sqlstream). Identifies a group of consumers processes that will collaborate to process output from the source Kinesis stream. If two agents read from the same Kinesis stream with the sameKINESIS_APPLICATION_NAME they will share out the Kinesis shards between them. Whereas if the two agents use two different KINESIS_APPLICATION_NAMEs they will both independently read all the data from the stream. |
|
|
KINESIS_MAX_RECORDS_PER_GET |
(default -1) if > 0 will read multiple records per request. (Usually a large number is better.) |
|
|
KINESIS_SOCKET_TIMEOUT |
Defaults to -1, which means leave at Kinesis setting. If set, will override Kinesis socket timeout in milliseconds. |
|
|
KINESIS_IDLE_TIME_BETWEEN_READS |
Defaults to -1, which means leave at Kinesis setting. If set, will override Kinesis time between reads in milliseconds. |
Stopping the Agent
To stop the agent, issue code along the following lines:
call sys_boot.mgmt.stop_agent_hoster(‘my_agent’);
The code below reads from a Kinesis stream called production-stream, and reads data into an s-Server stream called kinesis_sink_read_json in a schema called myschema. It passes options in for JSON parsing. See Reading JSON for more details on these options.
|
— Read data from the TARGET Kinesis stream; not normally used but can be helpful for debugging etc — Read into Sink_As_Source.kinesis_sink_read_json — Expand JSON into the columns as defined for the output — — agent_name = ‘target_listen_json’ — application_name = ‘sqlstream_3’
call sys_boot.mgmt.start_agent_hoster(‘target_listen_json’,’kinesis’,’PARSER=JSON& SCHEMA_NAME=myschema& TABLE_NAME=kinesis_sink_read_json& AWS_REGION=us-east-1& AWS_PROFILE_NAME=default& AWS_PROFILE_PATH=& KINESIS_STREAM_NAME=production-stream& KINESIS_APPLICATION_NAME=sqlstream_3& STREAM_FANOUT=1& KINESIS_INITIAL_POSITION_IN_STREAM=LATEST& KINESIS_MAX_RECORDS_PER_GET_RECORDS=-1& KINESIS_SOCKET_TIMEOUT=-1& KINESIS_IDLE_TIME_BETWEEN_READS=-1& PARSER=JSON& ROW_PATH=$& device_key_PATH=$.device_key& model_code_PATH=$.model_code& latitude_PATH=$.latitude& longitude_PATH=$.longitude& recorded_at_PATH=$.recorded_at& channel_PATH=$.sensor_readings[0:].channel& sensor_type_PATH=$.sensor_readings[0:].sensor_type& metric_value_PATH=$.sensor_readings[0:].metric_value& other_value_PATH=$.sensor_readings[0:].other_value& value_PATH=$.sensor_readings[0:].value& unit_PATH=$.sensor_readings[0:].unit’);
|
To read from Kinesis, you need to have an AWS configuration file set up. You can set up your aws_access_key_id and secret_access_key on the AWS Console as follows:
| 1. | Open the AWS Console. |
| 2. | Click Identity & Access Management |
| 3. | Click Users. |
| 4. | Click your User ID. |
| 5. | Create an Access Key. |
| 6. | When you create an access key, the AWS console will allow you to download a credentials file which will contain the values for aws_access_key_id and secret_access_key. |
Guavus SQLstream’s mission is to make real-time stream analytics easy to use and own with a one-stop-shop solution that performs the best, has the widest footprint, never turns off, and can be developed and customized by data scientists and engineers alike.
©2021 Guavus, Inc. All Rights Reserved. SQLstream® is a registered trademark of Guavus, Inc.