Configuring Network Socket as Input

<< Click to Display Table of Contents >>

Navigation:  Using StreamLab > StreamLab Sources Overview > Configuring Line, CSV, XML, and JSON Sources >

Configuring Network Socket as Input

Previous pageReturn to chapter overviewNext page

To read from a line, CSV, XML, or JSON file over Kafka, you need to configure the connection to Kafka. Kafka is an open-source, real-time publish-subscribe messaging framework. See http://kafka.apache.org/ for more details. You may want to consult with whoever has set up the Kafka messaging system in your environment.

To connect with Kafka, you need two pieces of information:

The name and port of the Kafka broker (this defaults to localhost:9092, but the source will not work if a Kafka broker is not working at this location).
The Kafka topic name from which you are reading.

The other configuration details below help manage the starting point for reading Kafka topics as well as the amount of data fed to StreamLab.

Format Name

Name

Name/Port

hostname:port of the Kafka broker. Defaults to localhost:9092. Used for getting metadata (topics, partitions and replicas). Actual socket connections are established using the information from this metadata. Use commas to separate multiple brokers.

Topic

Required. Kafka Topic

Starting Time

The time to start reading from specified topic. Options are LATEST, EARLIEST, or a long int representing a timestamp

(milliseconds since epoch). Defaults to LATEST.

Starting Offset

When to start reading from (Default is -1) as a long int representing a timestamp (milliseconds since epoch)

Partition

Number of Kafka partition to read from. Defaults to 0.

Buffer Size

Buffer size in bytes. Defaults to 1048576.

Fetch Size

Fetch size. Defaults to 100000.