Network Socket Agent (NSAgent)

<< Click to Display Table of Contents >>

Navigation:  Integrating SQLstream Blaze with Other Systems > Appendix A: Legacy Adapters and Agents > Legacy Agents >

Network Socket Agent (NSAgent)

Previous pageReturn to chapter overviewNext page

The Network Socket Agent ("NSAgent") is supplied either as part of the distributed SQLstream product or as part of the ClientTools download from the SQLstream website (via SQLstream-4.0.0-clienttools-linux.run or SQLstream-client-tools-4.0.0.-windows.exe). The Network Socket Agent records incoming packets and writes to one or more named SQLstream streams.

The Agent installs as a script, either nsagent.sh (Linux) or nsagent.cmd (Windows). Configuration works the same way on both platforms.

SQLstream's NSAgent will not start recording data until it is able to attach to at least one SQLstream s-Server instance.

Loss of Connectivity

If NSAgent loses connectivity to a stream it is writing to, it will log the loss of connectivity to its trace log and then attempt to reconnect to that stream until successful. (If it is connected to other streams, it will continue to record packets and write to other streams.) If a server gets sufficiently backed up trying to write to a stream, this will also be treated as a loss of connectivity.

Directory Structure

After installation, the Network Socket Agent directory contains a jar, a script, and a properties file:

NSAgent/
onsagent.sh (Linux) OR nsagent.cmd (Windows)
onsaparams.sh (Linux) OR nsaparams.cmd (Windows)
/lib/:
oLogFileAgent.jar
oLogFileAgentPath.jar
/trace/:
otrace.properties

Parameters

NSAgent uses the following parameters:

option

flag

value

Meaning

-h

--help

 

Display this help and exit

-ap

--agent-properties

<property-file-name>

Property file for socket parameters and packet recording/logging parameters

-cw

--connection-wait

<value>

Time in Milliseconds to wait between connection retries

-id

--source-id

<columnName:idString>

Source Identification

-sn

--stream-username

<username>

Username for connection to SQLstream server

-sp

--stream-password

<password>

Password for connection to SQLstream server

-su

--stream-uri

<uri>

JDBC URI for SQLstream server; e.g., jdbc:sqlstream:sdp://host:port

-schema

--schema

<schema-name>

Case-sensitive schema name

-sf

--on-stream-full

<wait|reconnect>

Should wait or reconnect when following streams are full. Can be specified per stream. Default is reconnect

-v

--verbose

 

Enable verbose output

stream-name

 

Unqualified stream name

(See also -schema arg)

(*) Agent-properties and optionally verbose should only be specified once.
The others can be specified multiple times.
One stream will be written to for each stream-name supplied.

NSAgent Options

Options can be defined as part of a stream or as a properties file for NSAgent. To use a properties file, you first create it with the file extension ".prop", then implement it using the -ap option:

nsagent -ap sampleProperties.prop

 

Option

Description

Possible values

BUFFER_COUNT

Number of buffers for receiving packets before blocking

 

BUFFER_SIZE

Size of buffers for receiving packets

 

ENCODING

Character encoding used for the input being read in.

UTF-8

Any Java Supported Encoding. Use encoding implemented by java.nio, see this link:

FILENAME_ DATEFORMAT

Java time format string for date conversion used during file rotation.

yyyy-MM-dd_HH:mm:ss

Uses java SimpleDateFormat

FILENAME_BASE

Target name for output and basename for file rotation
Filewriter appends an underscore to the specified name.

No default

FILENAME_ROTATED

Pattern specifying how to rename file at rotation time.

[FNB] = filename_base described above,

[RT] = filename_dateformat described above.

[FNB]_[RT]

 
Example:
my-[FNB]_rotated-at_[RT]

FILEPATH

Directory where resultant files are located

No default

HOST

Specifies which interface to listen to. It defaults to all interfaces. You can override this to 'LOCALHOST' to listen to only local connections, or a specific ip address, such as <168.212.226.204>.

Defaults to all interfaces.

MAX_IDLE_TIME

Optional timeout in milliseconds to force file rotation:
 
If no data has been written during the specified time, then a flush to the existing or current output file is forced and a new file begins, using the specifications provided in the filename options.

[Integer value]

(If no time unit is specified, milliseconds is used.)

MAX_TIME_BETWEEN_FLUSH

Specifies number of seconds after which a flush is done (formerly after every row)

Default: 1-second

(If no time unit is specified, milliseconds is used.)

PORT

Internet socket port number, that is, the port number at which a Foreign Stream will accept connections.

>0

PROTOCOL

Defines the communication protocol to be used to transfer data. Case insensitive.

TCP, UDP

(**)
RECORD_END _PATTERN

Defines the character[s] indicating the end of a record. Must be a single character if connection type is UDP. Only patterns that evaluate to a fixed number of strings of identical length are allowed.

Double character example, for TCP:

u\000A\000D

 

Single character example:

u\000A

 

OR'ed example:

(d|e)\000A\000D

(**)
RECORD_FIXED_SIZE

The size of the fixed portion of a record, for fixed-size records, or records with a length indicator.

 

RECORD_SEPARATOR _PATTERN

 

 

Defines the character[s] separating the records. Must be a single character if connection type is UDP. Only patterns that evaluate to a fixed number of strings of identical length are allowed.

Double character example, for TCP:

u\000A\000D

 

Single character example:

u\000A

 

OR'ed example:

(d|e)\000A\000D

(**)
RECORD_START _PATTERN

Defines the character[s] indicating the start of a record. Must be a single character if connection type is UDP. Only patterns that evaluate to a fixed number of strings of identical length are allowed.

Double character example, for TCP:

u\000A\000D

 

Single character example:

u\000A

 

(d|e)\000A\000D

(**)
RECORD_VARIABLE_SIZE_MAX

Maximum value of variable size

 

(**)
RECORD_VARIABLE_SIZE_BIGENDIAN

Boolean indicating whether high byte is first

 

(**)
RECORD_VARIABLE_SIZE_BYTES

Number of bytes used to specify size

 

(**)
RECORD_VARIABLE_SIZE_MIN

Minimum value of variable size

 

(**)
RECORD_VARIABLE_SIZE_OFFSET

Offset of variable size

 

ROTATION_ROW_TIME

(Treated the same as
ROTATION_SYS_TIME)

Time unit to trigger file rotation for rowtime rotation option, using ROWTIME changes, not wall-clock/system time.

Also, if a rowtime bound is received that indicates that any next row would cause rotation to happen, then rotation is performed without actually waiting to receive such a row.

Time units can be as shown in either Note A or B.

 

See Note C for examples of usage.

ROTATION_FILE_SIZE

Approximate file size to trigger ROTATION for FILE_SIZE rotation option (above);

Actual file sizes may be larger by the amount that can be written in the time specified by MAX_TIME_BETWEEN_FLUSH.

Default: 10m

k - kilobyte

m - megabyte

g - gigbyte

t - terabyte

no unit indicates bytes

Case insensitive

ROTATION_SYS_TIME

Time unit to trigger file rotation for system rotation (SYS_TIME) option, using wall-clock time, not ROWTIME changes.
 

Time units can be as shown in either Note A or B.

 

See Note C for examples of usage.

TYPE

Currently the only option is to Read from a socket and streams data.
Please contact Technical support if you need a WRITE option..

READER

(Case insensitive.)

Note A:
1h - hours
s - seconds
m - minutes
h - hours
d - days
w - weeks
Specifying no time unit indicates seconds s.

Note B:
MINUTE
HOUR
DAY
WEEK
MONTH

Note C:

(1)        When a rotation time t less than one day is specified, then
every day at midnight, midnight + t, midnight + 2t, etc.,
there will be a rotation as long as at least one row was output during that period.

(2)        When the rotation time t specified is an integral multiple N of 1 day,
rotation will always happen on midnight ever Nth day starting at Jan 1, 1900.
 
For example, Jan 1, 1900 happens to be a Monday - so if a rotation time of t=7d is specified,
rotation will always happen Sunday/Monday, 12:00 in the evening/morning.

(3)        When the rotation time t specified is greater than 1d, but not an integral multiple of 1d,
rotation will happen on regular intervals of size t starting at Jan 1,1900,
with no adjustments for daylight savings etc.

Example Stream

Here's an example using the adapter. It creates a named stream, RawIncomingDataStream with two columns: The first column, ip, is optional and gets filled with the connecting client's ip address. The second column, message, is filled with each row parsed from the incoming packets processed by NSAgent.

CREATE OR REPLACE STREAM "RawIncomingDataStream" (

ip VARCHAR(15) NOT NULL,

message VARBINARY(5120) NOT NULL)

SERVER "SocketReader"

OPTIONS (

type 'reader',

port '8871',

host '168.212.1.1' {Sets a specific address at which the NSAgent will listen}

protocol 'tcp',

buffer_count '100',

buffer_size '5120',

record_fixed_size '2',

record_variable_size_offset '0',

record_variable_size_bytes '2',

record_variable_size_bigendian 'true',

record_variable_size_min '16',

record_variable_size_max '5120',

record_start_pattern 'u\0002',

record_end_pattern 'u\0003'

) DESCRIPTION 'Command packets, as binary payload with a 2-byte

(0x02, aka STX) separator';

 
RECORD_START_PATTERN and RECORD_END_PATTERN can be specified instead of RECORD_SEPARATOR_PATTERN to delimit messages.

Using the NSAgent stream

Once you establish a named stream for NSAgent, such as "RawIncomingDataStream" in the example above, applications can query this stream like any other stream. See the topic SELECT clause in the SQLstream s-Server Streaming SQL Reference Guide for more details.

Example Properties File

A typical properties file, "sampleProperties.prop", could contain the following options:

PORT=8871
PROTOCOL=tcp
BUFFER_COUNT=100
BUFFER_SIZE=5120
RECORD_FIXED_LENGTH=2
RECORD_VARIABLE_SIZE_OFFSET=0
RECORD_VARIABLE_SIZE_BYTES=2
RECORD_VARIABLE_SIZE_BIGENDIAN=true
RECORD_VARIABLE_SIZE_MIN=16
RECORD_VARIABLE_SIZE_MAX=5120
RECORD_START_PATTERN=\u0002
RECORD_END_PATTERN=\u0003

 

You would then use the -ap option to specify that properties file, in a command similar to the following:

nsagent -ap sampleProperties.prop

 

along with other appropriate parameters as specified in the Parameters section above.

Packet Format

A packet in this protocol (in each direction) consists of two sections.

The first section is common to all messages and consists of fixed length fields. The fields within this section are shown in white in the table below.

The second section (shown in grey) is variable in length, based on the message type. This section may contain a number of pre-specified fields, in which case these fields will be variable length text separated by commas.

Note that all values within these fields must be represented and interpreted in Network Byte Order. This is defined as "Big Endian" within the TCP/IP Protocol suite.

The general command packet format consists of the following fields:

Start

Length

Time Stamp

Field Device ID

Message Type

Message Body

(**) NOTE: These eight options marked with double-asterisks are only applicable on varbinary tcp streams.

Trace properties

The following trace methods can be used to specify tracing levels:

•com.sqlstream.tools.nsagent.level

•com.sqlstream.plugin.level

 

The following table describes the size and content of each field:

Format

Size

Comments/Value

Encoding

Start

1 Byte

<STX> 0x02

Binary

Length

2 Bytes

(short integer)

Length of the total message excluding Start, Length and End.

Binary

Time Stamp

8 bytes

(long integer)

UTC time – the number of milliseconds since GMT 00:00:00, January 1, 1970. NB: Time within a field device is to be kept synchronised with the GPS time while a valid GPS connection is maintained.

Binary

Field Device ID

6 Bytes

The field device Id is a unique identifier associated with the field device itself. This is the 48-bit MAC Address (MAC-48) of the field device NIC, as defined by the Ethernet standard [3].

Binary

Message Type

2 Bytes

The first character is the command while the second character is a sub-command or response.

US-ASCII

Message Body

Depends on message type but may be zero length.

Optional. Message fields are comma separated

US-ASCII

End

1 Byte

<ETX> 0x03

Binary