Adding an External Data Stream Sink
This sink writes data in CSV (Comma Separated Value) format to an output.
|1.||First, select a sink output type. You need to select one of these for CSV, JSON, and XML sinks. This could be a file, network socket, AMQP, or Kafka location.|
|2.||Start by specifying the s-Server schema and stream names for the sink's stream. By default, StreamLab uses the project schema for the sink. You can also route guide data to the sink later by selecting the Routing Data to a Sink button in a guide.|
|3.||Next, fill in the list of columns and their SQL types. Click the + icon to add another column. Give the column a name, and select its type from the pulldown menu. You can use the Clipboard to copy column names and types from another form.|
|4.||Use the output selector to choose an output and fill in the parameters to connect to that source. (This sink uses the Extensible Common Data Adapter to support writing to a number of different output media.) For more information on the options for ECDA, see the topic Writing to Files with the ECD Adapter and Agent in the Integration Guide.|
|5.||Click the blue title to save the project. The title returns to white after you click it.|
|6.||Click the Go Up arrow to exit the Add CSV Sink page.|
In this mode the output is accessed directly from s-Server. Files are local to the s-Server host, and sockets and other network connections are relative to the s-Server host. When this mode is enabled, StreamLab submits your configuration for this sink directly to your instance of s-Server.
In this mode you can access sinks via a host running the ECDA agent. The agent is a Java application running on a host other than the s-Server host, configured to read data from your instance of s-Server. When this mode is enabled, StreamLab configures the stream that the remote agent will read from. To configure the remote agent,use the Export Properties button to get the contents of the properties file, and paste this into a new file on the remote agent server (for example, mysource.props).
From a command shell on the remote host, run
%$SQLSTREAM_HOME/../clienttools/EcdaAgent.commondataagent.sh --output --props mysource.props.
Note: $SQLSTREAM_HOME refers to the installation directory for s-Server, such as /opt/sqlstream/5.0.XXX/s-Server.
Everything you've specified here to configure the sink will now apply within the remote agent, so file names will be local to that host, and sockets and other network connections will be relative to that host.
The character used to separate columns defaults to comma and can be changed via the Column Separators(s) field. You can also change the Row Separator, which defaults to 000A. You can also change the Character Encoding and decide if you want a CSV header containing the column names to be written.
You control the XML format by specifying information in the fields below Character Encoding:
Document elements indicates a list of elements, separated by slashes, to use as the root of the XML document.
Row elements indicates a list of elements, separated by slashes, to add for each value output to the document.
Data elements indicates a list of elements, separated by slashes, to add for each value output to the document.
Data attributes indicates the name of an attribute to add to the output.
If no value for DATA_ELEMENTS is specified, then the column name is used as an element name (not an <attribute> name). So a column named foo would be in an XML element named /batch/row/foo if no values were specified.