Runtime Tracing and Creating Trace Files

Introduction to Runtime Tracing

When enabled, runtime tracing writes tuple trace information to StreamBase Server's console or to a trace file, one file per container. If any component or module in your application uses concurrency features, then a separate trace file is created for each separate thread. (See Execution Order and Concurrency for more on concurrency.)

The default trace information is a single timestamped line per tuple. Each tuple is shown as received at each operator and stream in the application. Trace files can grow quite large very quickly, so runtime tracing is best used only for short bursts, for debugging only, to follow a tuple's progress through an application.

You can limit the number of operators and streams that are subject to tracing by using a regular expression to narrow the list of the components of interest.

Tracing does not occur unless you set up StreamBase Server for it and tracing is enabled. To set up for tracing, set the system property streambase.codegen.trace-tuples or environment variable STREAMBASE_CODEGEN_TRACE_TUPLES to true, as described below. Once set up, tracing is enabled by default when the application starts, unless you disable it at runtime with sbadmin modifyContainer containerName trace false. In that case, re-enable tracing with sbadmin modifyContainer containerName trace true.

Trace files are stored with UTF-8 encoding, which preserves any Unicode characters in tuple fields in the trace file.

Tracing Versus Dequeuing

Runtime tracing is similar to, but not the same as using sbc dequeue to dequeue tuples from any stream in your application, including intermediate streams, if enabled (see Intermediate Stream Dequeuing). The following table shows the differences between runtime tracing and dequeuing.

Feature	Standard Dequeuing	Runtime Tracing
See tuple output from ...	All explicit streams by default, and all intermediate streams in debug mode or if otherwise enabled.	All operators and streams (optionally limited by regular expression selection).
Timestamp shown for each tuple	No	Yes
Save output to a file or console	Must redirect console output or use an output adapter.	Specify with a runtime option.
Starts with StreamBase Server	No. It takes time to start the Server, then start a dequeuing client. The first few tuples can be missed.	Yes. Even the first few tuples are captured.
Can see tuple output in StreamBase Studio	Yes.	Yes. Trace files can be opened, searched, and run in Studio's SB Trace Debugger perspective.

Setting Up for Runtime Tracing

To use the runtime tracing feature, you must set up StreamBase Server to respond to tracing commands. Use one of the following methods:

Set the environment variable STREAMBASE_CODEGEN_TRACE_TUPLES to true.

Set the Java system property streambase.codegen.trace-tuples to true. Use the jvm-args or sysproperty directives in the server's configuration file, as described in StreamBase Server Configuration File XML Reference. For example:

<java-vm>
    <param name="jvm-args" value="-XX:MaxPermSize=128m 
       -Xms256m -Xmx512m 
       -Dstreambase.codegen.trace-tuples=true" />
</java-vm>

or

<java-vm>
    <sysproperty name="streambase.codegen.trace-tuples" value="true"/>
</java-vm>

Configuring Runtime Tracing

Configure runtime tracing with one of three methods:

Use the trace child element of the application element in the StreamBase deployment file.
Use the sbadmin addContainer command at runtime with its --trace* options.
Use the sbadmin modifyContainer command at runtime with its trace true and trace false subcommands, and the --trace* options.

All three methods have the same three options, as described in the following table:

`trace` element	sbadmin option	Default	Description
traceStreamPattern="`pattern`"	--traceStreamPattern="`pattern`"	none	Specify a regular expression `pattern` in the format of `java.util.regex.Pattern`. The pattern limits tracing to the operators and streams whose name matches the specified `pattern`. If not specified, tracing is active for all operators and streams in the application.
traceFileBase="`basename`"	--traceFileBase="`basename`"	none	By default, trace output is written to StreamBase Server's console. To redirect trace output to a file, specify `traceFileBase` with a `basename` argument. There is one trace file generated per parallel module. Specify a string to serve as the `basename` for trace files. Trace file names are generated as `basename+containername.sbtrace`. For modules running with parallel execution enabled, the module's path becomes part of the trace file name: `basename+containername.modulepath.sbtrace`. Tracing also creates a second file with extension `.sbtrace.meta`. This file, in XML format, contains the equivalent of the output of sbc describe. This file is automatically used by the Trace Debugger perspective when reading and displaying `.sbtrace` files.
traceOverwrite="true"	--traceOverwrite	false	For the `trace` deployment file element, specify "`=true`" or "`=false`" (default). For the sbadmin option, use the option without arguments to set the `true` state. Setting to `true` causes tracing to overwrite all trace files on StreamBase Serverstartup. By default, new trace information is appended to existing trace files for each traced container.
traceCompress="true"	--traceCompress	false	For the `trace` deployment file element, specify "`=true`" or "`=false`" (default). For the sbadmin option, use the option without arguments to set the `true` state. Set to `true` to specify that the generated trace file is to be compressed with gzip. If `true`, the trace file's extension becomes `.sbtrace.gz`. Compressed trace files are readable by Studio's Trace Debugger perspective.
traceBuffered="false"	--traceBuffered=true --traceBuffered=false	true	When set to `false`, the output trace file is flushed on every trace line. This slows down performance considerably, but makes it easier to use tracing as part of debugging a live application. When set to `true` (the default), tracing output is buffered before the trace file is written, to minimize tracing's drag on performance.

When using either the <trace> element of the deployment file or sbadmin addContainer, if the setup conditions are met, then tracing is enabled by default when the application starts. You can disable tracing at runtime with sbadmin modifyContainer trace false. In that case, re-enable tracing with sbadmin modifyContainer trace true. As soon as you disable or stop tracing, all trace files are closed.

Command Line Tracing Example

The following sequence of UNIX commands sets up for tracing, starts a container with tracing enabled, then disables tracing. Trace lines are written to a trace file named tr_default.sbtrace.gz and a metadata file named tr_default.sbtrace-meta.

In one terminal window, type:

export STREAMBASE_CODEGEN_TRACE_TUPLES=true
sbd BestBidsAsks.sbapp

In another terminal window, type:

sbadmin modifyContainer default trace  --traceFileBase="tr_" \
   --traceStreamPattern="Bids" --traceOverwrite
sbfeedsim -q NYSE.sbfs
Ctrl+C [to stop the feed simulation]
sbadmin modifyContainer default trace false

The following shows the same sequence of commands for Windows, using a StreamBase Command Prompt:

set STREAMBASE_CODEGEN_TRACE_TUPLES=true
start sbd BestBidsAsks.sbapp
sbadmin modifyContainer default trace --traceFileBase="tr_" 
   --traceStreamPattern="Bids" --traceOverwrite
sbfeedsim -q NYSE.sbfs
Ctrl+C [to stop the feed simulation]
sbadmin modifyContainer default trace false

In both UNIX and Windows, a trace file named tr_default.sbtrace is created in the sample's workspace directory. You can examine the file in a text editor or load it in the Studio Trace Debugger for analysis and interactive trace debugging.

You can stop and restart tracing while you send in manually enqueued tuples or a feed simulation. When ready to exit, type:

sbadmin shutdown

Deployment File Example

Use the deployment file trace element like this example:

...
<application container="default" module="BestBidsAsks.sbapp">
  <trace traceStreamPattern="Bids" traceFileBase="trace_" 
     traceOverwrite="true" traceCompress="true" />
</application>
...

This creates a trace file named trace_default.sbtrace.gz, which is overwritten each time the server restarts, and contains tracing information about all operators and streams with the string "Bids" in their names.

Format of Trace Files

Trace files have the .sbtrace extension and consist of one header line (shown below on two lines for publication clarity), plus one trace line per tuple received at each specified operator and stream. The following example shows the start of a trace file for the Best Bids and Asks sample shipped with StreamBase, running in the default container named default:

# ms since epoch, tupleid, operator/streamname, tuple-data - \
  trace started: 2010-11-04 16:09:22.200-0400 - 1288901362200ms
1288901370904,1,default.NYSE_Feed,33409,NLY,14.0,0,16.0,0,13033
1288901370906,1,default.out:Update_Bids_and_Asks_1,33409,NLY,14.0,16.0,14.0,16.0
1288901370906,1,default.out:IsNewBestAsk_1,33409,NLY,14.0,16.0,14.0,16.0
1288901370906,1,default.BestAsks,33409,NLY,16.0
1288901370906,1,default.out:IsNewBestBid_1,33409,NLY,14.0,16.0,14.0,16.0
1288901370906,1,default.BestBids,33409,NLY,14.0
1288901370906,2,default.NYSE_Feed,33620,NCC,28.5,0,30.5,0,13249
1288901370906,2,default.out:Update_Bids_and_Asks_1,33620,NCC,28.5,30.5,28.5,30.5
1288901370906,2,default.out:IsNewBestAsk_1,33620,NCC,28.5,30.5,28.5,30.5
...

Trace lines consist of at least four fields, described in the following table:

Field	Description
1	Timestamp in milliseconds representing the interval of time since the epoch.
2	Tuple ID.
3	Fully qualified name of the operator or stream output port.
4 through `n`	The comma-separated fields of the tuple as it exits the port described in field 3.

Back to Top ^