Container Connections

Topics:

Overview of Container Connections

This page discusses the various types of connections between containers and the several ways to create and modify container connections. To learn about containers in general, see Container Overview.

Container Connection Types

You can set up two types of connections between StreamBase applications running in different containers:

  • Stream to stream container connections.

    In this configuration, you set up an application's input stream in one container to receive tuples from an output stream of another application or module in another container, possibly running on another StreamBase Server. Stream to stream connections can be of two types:

    • Asynchronous. In the default and recommended form of stream to stream container connections, the flow of tuples is set to run asynchronously, with a queue automatically set up to manage the flow through the connection.

    • Synchronous. You can optionally specify low latency, direct connections between container streams without the benefit of queue management. This option offers a possible performance improvement for certain circumstances. See Synchronous Container Connections.

  • JMS container connections are connections from container streams to and from your site's Java Message Service (JMS) infrastructure.

    In this configuration, you set up one StreamBase Server to send tuples to a JMS server, and another StreamBase Server to receive those tuples from the JMS server. This feature allows you to use the reliable delivery feature of JMS to ensure delivery of StreamBase tuples between applications and containers. This feature is discussed on a separate page, Connecting Container Streams Through JMS.

  • CSV file container connections.

    In this configuration, you connect a container's output stream to a URI that specifies the absolute path to a CSV file. All tuples sent on this port are then written to the specified file. You can also connect a URI to a CSV file to an input stream. In this case, the input stream is read from the CSV file. CSV file container connections are discussed on a separate page, Connecting Container Streams To a CSV File.

Ways to Set Up Container Connections

StreamBase offers fours ways to set up container connections. The first two methods establish the configuration of containers and container connections at server startup time:

  • In the StreamBase Server configuration file, typically named sbd.sbconf.

  • In StreamBase Studio, in the Containers tab of the Run dialog for editing StreamBase launch configurations.

The next two methods for setting up container connections can be used interactively, after StreamBase Server has started:

  • From the command line, using the sbadmin addContainer and sbadmin modifyContainer commands.

  • Using StreamBase Manager's Add Container command in the context menu for servers and containers, as described in Context Menu Actions for Servers.

Setting Up Container Connections

When you add or modify a container, you can specify a connection to or from a stream in another existing container, even if on a remote StreamBase Server instance. Specify a container connection with the following syntax when using the sbadmin addContainer and sbadmin modifyContainer commands.

Note

The syntax for expressing all forms of container connections is always destination=source, like an assignment statement in some programming languages. 

container.instream=othercontainer.outstream

If the target container is running on a remote StreamBase Server, specify a full StreamBase URI enclosed in parentheses: 

container.instream=(sb://remotesbhost:9955/app3.outstream)

Using a URI in a container connection statement works in the other direction as well. For example: 

(sb://remotehost:9995/app2.instream)=container.outstream

There are three stream connection cases:

  • Containers share the same input stream.

  • A container receives its input from the output of another container.

  • Containers share the same output stream (magic union).

When you use the sbadmin addContainer or sbadmin modifyContainer commands, you can specify a container connection at the same time. The syntax is: 

sbadmin addContainer container-name application.[sbapp|ssql|sbar] connection1 connection2 ...

The following example cascades the output through all of the following three applications: 

sbadmin addContainer A app.ssql
sbadmin addContainer B app.ssql B.InputStream1=A.OutputStream1
sbadmin addContainer C app.ssql C.InputStream1=B.OutputStream1

A tuple enqueued into container A is passed to B, then processed by B and passed to C for further processing.

When Input and Output Streams are shared, the streams are still owned by the original container. This means that any input or output to the shared stream ultimately needs to go through the original container. When tuples are passed between containers they are passed using the same queuing mechanism used between parallel modules. This means that the tuple is queued for the other containers. When the tuple is queued, the original container must wait for the tuple to be pushed onto the queue, but will not be blocked by the processing of the tuple (which is done in another thread). Because of the queuing architecture, there is no guarantee of the order of tuples when they are shared between multiple containers. In the above example, containers A and B share the stream A.InputStream1. When a tuple is enqueued into A.InputStream1 the tuple is first passed to all containers that share that stream (in this case, container B) and then it is processed to completion by itself.

Nothing prevents you from enqueuing directly to containers B and C while those containers are still receiving input from their upstream containers. Because upstream tuples are queued, any input directly into container B and C will be randomly interleaved with input from container A.

When a container is removed, the input or output to dependent containers disappears. The dependent containers continue to function, but their input or output is removed. In the example above, if container B is removed, it means that A is suddenly not sharing the input with container A and that container C is suddenly no longer receiving input from upstream containers.

Schema Matching for Container Connections

The schemas of the outgoing and incoming streams are matched in different ways for different cases:

For Container Connections on the Same Server

For stream to stream connections between containers running on the same StreamBase Server, the connected streams must have equivalent schemas. That is, the connected streams must share exactly the same field names, field types, sizes, and field positions between the two streams. Thus, if the two streams share the same field types, sizes, and names, but are in different order, StreamBase cannot map the streams together.

For Container Connections Between Servers

For stream to stream connections between containers running on separate StreamBase Servers, fields in the outgoing stream are matched by name against fields in the incoming stream. Fields whose names match must have the same data type in both streams. Any fields in the outgoing schema whose names don't match the incoming schema are not streamed. Any fields in the incoming schema whose names don't match anything in the incoming schema are set to null.

For example, consider an outgoing stream with schema (a int, b (x int, y int, z int)) that is connected to an incoming stream with schema (b (y int, z int), c int). The incoming stream does not see fields a or b.x. Fields b.y and b.z are passed from incoming to outgoing, and field c in the incoming stream is set to null.

Dynamic Container Connections

Use sbadmin modifyContainer to dynamically add or remove a container connection from an existing container: 

sbadmin modifyContainer container-name [addConnection | removeConnection] connection

Synchronous Container Connections

By default, connections between containers are set up to run asynchronously, with a queue to manage the flow of tuples through the connection, as described in Setting Up Container Connections.

StreamBase also supports synchronous container connections, which are low latency, direct connections between containers without queue management. An application might use a synchronous container connection to improve response time of the communication between HA monitoring applications in clustered and highly available StreamBase applications.

Specify a synchronous container connection by using := (colon-equals) instead of = (equals) in your container connection assignment statement. For example: 

sbadmin addContainer A app1.sbapp
sbadmin addContainer B app2.sbapp
sbadmin modifyContainer addConnection B.incoming:=A.outgoing

Specifying synchronous container connections may or may not improve the overall speed of your application. It is possible to degrade your application's speed by, for example, setting up a synchronous connection to a container that blocks. In general, use synchronous container connections for the same reasons you would designate an operator or adapter to use parallel execution. See Execution Order, Concurrency, and Parallelism for more information on data parallelism.

StreamBase detects and prevents any attempt to make a synchronous container to connection to:

  • Any stream in the system container

  • An endpoint that would cause a loop in processing

Related Topics