Adding a Protobuf Source

<< Click to Display Table of Contents >>

Navigation:  Using StreamLab > StreamLab Sources Overview >

Adding a Protobuf Source

Previous pageReturn to chapter overviewNext page

You can use StreamLab to parse files developed with the Google protocol buffer compiler. These files are compiled from .proto files. Before using StreamLab for protobuf, you need to first write .proto files and the compile them using the Java compiler that Google provides. See https://developers.google.com/protocol-buffers/ for more details.

The Google protocol buffer compiler lets you define optional, repeated, and required messages. In parsing these message types, the ECD framework allows a single repeated inner message field. If you specify a a message as repeated, then one row will be output per repetition. All non repeated fields will be duplicated between those rows.

To configure a Protobuf source:

1.On the Sources page, drag a Protobuf source from the left column into the center area.
2.Click the new Protobuf source.

The Edit Protobuf Source page opens.

3.Select a schema from the pulldown menu on the left.

sl_add_stream_source_sel_schema

4.Select a stream from the pulldown menu on the right. This is the stream into which lines from the Protobuf source will be read.
5.Select File System, Socket, AMQP 0.9.1, AMQP 1.0, or Kafka.
6.Next, enter configuration information for the Protobuf source:

Option

Definition

Schema Jar File

Required. Jar containing compiled java classes created with the Google protocol buffer compiler (protoc command), such as unitsql/concurrent/plugins/common/protobufData/protobufpackage.jar. Locations are relative to $SQLSTREAM_HOME. See https://developers.google.com/protocol-buffers/docs/javatutorial for more details.

Note: $SQLSTREAM_HOME refers to the installation directory for s-Server, such as /opt/sqlstream/5.0.XXX/s-Server.

Schema Class

Required. Class name of outer protobuf record created with the Google protocol buffer compiler (protoc command), such as protobuf.PackageProto.protobuf.PackageProto$protobufPackage. Locations are relative to $SQLSTREAM_HOME. See https://developers.google.com/protocol-buffers/docs/javatutorial for more details.

Columns

Lets you specify a path within the schema that maps to a column in the foreign stream. If these are not specified, StreamLab populates a column using a field with the same name in the outer level record .

Does the input contain a stream of multiple messages?

Indicates whether or not to treat messages as continuous stream. If checked, then protobuf messages will be sent to s-Server as they're received. This could make a difference for sources, such as files or sockets, which don't necessarily deliver data in complete chunks.

Are messages prefixed with their length?

Indicates whether or not are all records prefixed with a length field. Must be specified if MESSAGE_STREAM is set.

7.Test the source by clicking the Sample 5 Rows from Source button.
8.Click the Go Up arrow to exit the Edit Source page.

  sl_go_up_arrow

9.To parse the newly added source, add a guide using the newly added Line source.