Adding a Protobuf Source

<< Click to Display Table of Contents >>

Navigation:  »No topics above this level«

Adding a Protobuf Source

Return to chapter overview

You can use StreamLab to parse files developed with the Google protocol buffer compiler. These files are compiled from .proto files. Before using StreamLab for protobuf, you need to first write .proto files and the compile them using the Java compiler that Google provides. See for more details.

The Google protocol buffer compiler lets you define optional, repeated, and required messages. In parsing these message types, the ECD framework allows a single repeated inner message field. If you specify a a message as repeated, then one row will be output per repetition. All non repeated fields will be duplicated between those rows.

To configure a Protobuf source:

1.On the Sources page, drag a Protobuf source from the left column into the center area.

2.Click the new Protobuf source.

The Edit Protobuf Source page opens.

3.By default, StreamLab uses the project schema for the new source. You can also select a schema from the pulldown menu on the left.


4.Select a stream from the pulldown menu on the right. This is the stream into which lines from the Protobuf source will be read.

5.Select File System, Socket, AMQP 0.9.1, AMQP 1.0, or Kafka.

6.Next, enter configuration information for the Protobuf source:



Schema Jar File

Required. Jar containing compiled java classes created with the Google protocol buffer compiler (protoc command), such as unitsql/concurrent/plugins/common/protobufData/protobufpackage.jar. Locations are relative to $SQLSTREAM_HOME. See for more details.

Note: $SQLSTREAM_HOME refers to the installation directory for s-Server, such as /opt/sqlstream/5.0.XXX/s-Server.

Schema Class

Required. Class name of outer protobuf record created with the Google protocol buffer compiler (protoc command), such as protobuf.PackageProto.protobuf.PackageProto$protobufPackage. Locations are relative to $SQLSTREAM_HOME. See for more details.


Lets you specify a path within the schema that maps to a column in the foreign stream. If these are not specified, StreamLab populates a column using a field with the same name in the outer level record .

Does the input contain a stream of multiple messages?

Indicates whether or not to treat messages as continuous stream. If checked, then protobuf messages will be sent to s-Server as they're received. This could make a difference for sources, such as files or sockets, which don't necessarily deliver data in complete chunks.

Are messages prefixed with their length?

Indicates whether or not are all records prefixed with a length field. Must be specified if MESSAGE_STREAM is set.

7.Test the source by clicking the Sample 5 Rows from Source button.

8.Click the Go Up arrow to exit the Edit Source page.


9.To parse the newly added source, add a guide using the newly added Line source.