Debugging Feature: Dequeuing Data from Intermediate Streams

Beginning with StreamBase 3.7, you can dequeue data from an intermediate stream. This feature is for debugging only, to examine the value of data at an intermediate point in the application, before it has been fully processed and before it has arrived on an Output Stream. This feature is not for inserting new data into the "middle" of an application. Use an Input Stream to declare an entry point for data.

Note: In 3.7.0, to use this feature, you had to start the StreamBase Server process (sbd) with the -d option. For example:

>sbd -d Split.sbapp
[notice] sbd at smart-lap:10000; pid=3148; Listening

In 3.7.1 and later releases, you can continue to do that; or you can start the server in its standard (non-debugging) mode, provided that you set the following JVM argument to true:

-Dstreambase.codegen.intermediate-stream-dequeue=true

The default for this JVM argument is false. JVM arguments can be configured in the server's sbd.sbconf.

You can dequeue data from any output port of any operator in an application using a string of the form:

out:operatorName_N

...where operatorName is the name of the operator and N is the number of the output port for that operator.

Your client can dequeue from intermediate streams in the same way that you dequeue from Output Streams. This is a feature that you will use in a client with the StreamBase API. Note that StreamBase Studio does not provide graphical views into the intermediate stream values; although, in the Test/Debug perspective, the Debugger View lets you step through the tuples that enter each operator.

Consider the Split.sbapp application diagram. This sample is installed with the StreamBase kit:

To verify the intermediate stream names, use an sbc list command. For example:

>sbc list -a
stream  INTRUSION_TooManyIPsForUser
stream  INTRUSION_TooManyUsersForIP
stream  IPandUserLogin
stream  out:CheckIPsForAUser_1
stream  out:CheckUsersInAnIP_1
stream  out:ProcessIPFirst_1
stream  out:ProcessIPFirst_2
schema  schema:IPandUserLogin
operator        CheckIPsForAUser
operator        CheckUsersInAnIP
operator        IPCountExceeded
operator        ProcessIPFirst
operator        UserCountExceeded
container       default
container       system
>

Compare the application diagram and the sbc list output. Notice the following:

• Because we are running the server in debug mode (or the previously described JVM argument is set), intermediate streams are known.
• Examples of the default naming convention for intermediate streams are seen in the list. An example: out:ProcessIPFirst_2.
• The output ports for the Filter operators, UserCountExceeded and IPCountExceeded, are connected to actual Output Streams, and thus do not have "intermediate streams." Of course, your client can dequeue from intermediate streams or actual Output Streams.
• In StreamBase Studio or in an editor, while you could view the XML source of the application diagram to verify the names of intermediate/output streams, the sbc list -a command is a convenient way to discover the names. If you view the XML source of an .sbapp, look for the stream value(s) associated with output port(s). For example:

    .
    .
    .
  <box name="ProcessIPFirst" type="split">
    <input port="1" stream="IPandUserLogin"/>
    <output port="1" stream="out:ProcessIPFirst_1"/>
    <output port="2" stream="out:ProcessIPFirst_2"/>
    <param name="output-count" value="2"/>
    .
    .
    .
  
  

We mention the discovery of stream names for a reason. In some cases, the name of an intermediate stream can vary from the default convention, depending on the history of edits to the application. For example, consider this next installed sample, AggregateByDim.sbapp:

When you run this sample in debug mode (or the JVM argument is set, as previously described in this article), an sbc list -a command returns:

stream  AvgPricePSOut
stream  OutputStream1

stream  TradesIn
schema  schema:TradesIn
operator        Aggregate2Dimensions
operator        ConvertTimeToSeconds
container       default
container       system

By default, the intermediate stream between the Aggregate2Dimensions and ConvertTimeToSeconds operators would have been named out:Aggregate2Dimensions_1. But at some point in this application's history, the Aggregate2Dimensions operator was connected to an Output Stream named OutputStream1, and was subsequently disconnected from that Output Stream (which was either removed or renamed to AvgPricePSOut).

Despite the edits, the original output stream name (OutputStream1) defined for Aggregate2Dimensions is still in use. In other words, don't assume that each operator's output stream name follows the default convention. Check the value with an sbc list -a command or view the XML source.

Recommendation: Use StreamBase Java Toolkit to Generate Dequeuers In your client, you can now dequeue from any of the streams. If you are an experienced application developer but are new to the StreamBase API, a recommended way to generate a working template of a StreamBase dequeuer client is to use the StreamBase Java Toolkit for Eclipse. After you complete the setup steps (please see Related Topics), generating templates for clients (and custom functions, embedded adapters, and Java operators) is easy. Here's a sample dialog from an installed StreamBase plugin within Eclipse:

And here's a sample screen of the generated template, which you can then customize:

Related Topics For related information, see:

• Getting Started with the StreamBase Java Toolkit for Eclipse 
• Using the StreamBase Java Toolkit to Write a Dequeuer Client 
• StreamBase API Reference Javadoc.

Back to Top ^