StreamBase New and Noteworthy

You can now connect three or more StreamBase Server instances in a highly available cluster. This change is implemented in two features, the HA Heartbeat Adapter, and the syntax for addressing multiple StreamBase URIs.

When setting up HA monitoring with the HA Heartbeat adapter, you must still designate one instance of the HA Heartbeat adapter as the heartbeat server. The change is that you can now connect more than one HA Heartbeat adapter as client to the heartbeat server. See HA Heartbeat Input Adapter.

When using the sbc utility or a custom-written client based on the StreamBaseClient API, you can now address two or more StreamBase Server instances in the same command by using a comma-separated list of StreamBase URIs. See the sburi reference page for details. The following example uses StreamBase URI shortcuts to request status information from four StreamBase Server instances running on four TCP ports on the default host, localhost:

sbc -u :9600,:9700,:9800,:9900 status

You can now connect two streams in containers on separate servers, passing through your site's Java Message Service (JMS) infrastructure. This feature allows you to use the reliable delivery feature of JMS to ensure delivery of StreamBase tuples. To use this feature, specify two container connections:

StreamBase supports JNDI-style JMS URIs, as described in Connecting Container Streams Through JMS. You can make the two JMS connections either in the server configuration files of two StreamBase Servers, or on the command line with the sbadmin addContainer command as described on the sbadmin reference page.

Release 6.1 added a second HA sample, which demonstrates the following:

See High Availability Samples.

A new operator was added to the Operators drawer in the Palette view in Studio. To each passing tuple, the Sequence operator adds a field that contains a unique identifier generated for that tuple. You can use the identifier field downstream as required by your application, such as to generate a keyword for tuples before storing them in a query table. You can link two or more Sequence operators together to share the state of the counter. See Using the Sequence Operator.

For StreamSQL, the functionality of the Sequence operator can be implemented with a dynamic variable, as described in Using Dynamic Variables to Implement a Sequence Operator.

A data directory is a file system location where a running StreamBase Server instance stores the files that implement disk query tables. Starting with release 6.1, you can specify a separate data directory location for each container, using either the new datadir server configuration file attribute, or interactively with sbadmin addContainer command options. See the StreamBase Server Configuration XML and sbadmin reference pages. This feature is illustrated in the new HA2 sample, described above.

A module directory is a file system location containing external application modules to be used at run time. You can now specify a separate module directory for each container, using the --moduleSearch option for the sbadmin addContainer command. See the sbadmin reference page.

The sbc list command now accepts a new option, –C container-name, which lists the streams and operators for the specified container. See the sbc reference page. For example, the following command lists the contents of the container named mainapp on the default server and port:

sbc list -C mainapp

Several new environment variables and Java properties were added to support fine-grained control of timeouts for clients connecting to StreamBase Servers. See Environment Variables and Java Properties.

The External Process Java operator was added in release 5.1, and the External Process Command Line operator was added in 6.0. These operators provide a way for StreamBase applications to run arbitrary operating system commands as if typed at the shell command prompt for the current operating system. This feature is especially useful in HA application contexts, where an application in one container might need to send an sbadmin command to an application in another container or on another StreamBase Server.

In previous releases, these operators did not appear in the Palette in Studio, but were delivered as JAR files that had to be added to the Java Build Path of your Studio projects. Starting with release 6.1, these operators now appear in the Global Operators & Adapters Palette drawer, and can be dragged into your EventFlow canvas like any other operator. Both operators are described on the same Authoring Guide page, Using the External Process Operators.

In release 6.1 in both operators, you can now specify the prefix prepended to input tuple fields, command argument fields, and environment variable fields, if you include these fields in the outgoing tuple. In the External Process Command Line operator, you can now optionally include the specified command line in the outgoing tuple.

The SB Demos perspective has a new member, the Real Time Profit and Loss demo.

Studio 6.1 introduced the ability to run separate applications or modules in separate containers when launched to run from Studio. Studio still runs only one instance of StreamBase Server at a time, but if you partition your application into separate containers for deployment, you can now run and test those multi-container applications in Studio. This feature also allows you to make stream to stream or other connections between containers at launch time.

The limitations are that the primary application still runs in a container named default, and that the Studio visual debugger cannot follow a debugging thread across container lines. See The Containers Tab for more information.

The Palette drawer called Global Adapters in previous releases was renamed to Global Operators & Adapters. This drawer now contains icons for:

StreamBase Studio 6.1 extended and improved the support for hierarchical data and nested schemas first offered in release 6.0.

See Schema Overview to understand the different schema-related terms.

All the schema editing tabs or dialogs in Studio 6.1 now support the tuple data type, and have the following usability improvements:

In the SB Test/Debug perspective, the Application Output view now has a filter field that accepts a regular expression to limit the display of rows in the view. The filter expression defines a strict string match, and must include wildcard characters to include parts of the row you are not filtering.

For example, when running the Best Bids and Asks sample, the filter *symbol=T?B* matches all rows containing the string symbol= followed by T, any character, then B, surrounded on both sides by any string:

Use the clear button to empty the filter field and restore the full output view.

The Feed Simulation view in the SB Test/Debug perspective now has a Pause button that lets you temporarily suspend a running feed simulation. When paused, the Pause button changes to a Resume button. The Pause/Resume button applies only to the selected feed simulation file in the view.

While using Studio's Visual Debugger on an application with breakpoints set, when the first breakpoint is reached, Studio prompts for permission to pause all the application's feed simulations. In most cases, you can respond Yes, and then step through your application using tuples already placed in the input queue by the feed simulation before it was paused.

In large applications with several input queues each fed by a different feed simulation, you might not want to have all the application's feed simulations paused. In this case, respond No, and then control which feed simulations are paused using the Pause/Resume button described above.

The StreamBase Studio SB Authoring perspective now includes the Reuters Schema Designer view, which allows you to browse and discover available RMDS data on your network, and to create a StreamBase schema from a selected set of discovered field identifiers. This view was previously available only in StreamBase Reuters Edition, but as of release 6.1, the view became part of standard StreamBase Studio. This view is described in Reuters Schema Designer View.

The RFA/Java Configuration wizard, previously available only in StreamBase Reuters Edition, is now part of standard StreamBase Studio. You start the wizard by running File → New → RFA/Java Configuration File, or by selecting the link in the top right corner of the Reuters Schema Designer View. This wizard lets you test RMDS connectivity and configure your system's RFA/Java configuration without leaving Studio. The wizard is described in Reuters RFA/Java Configuration Wizard.

Release 6.0 added support for customized Palette and canvas icons for your custom operators and adapters. In Release 6.1, this support was extended with a new option. In the Metadata tab of the EventFlow Editor, if you specify a 24x24 pixel image file in the Canvas Overlay field, that image is overlaid as before in the lower right corner of the operator's canvas icon. If you specify a 48x48 pixel image file, it now replaces the operator's canvas icon.

The sbadmin utility has the following new features and updates:

Release 6.1 introduced the following new functions to the StreamBase expression language: trim(), ltrim(), rtrim(), upper(), lower(), inet_aton(), and inet_ntoa().

There is now a two-argument version of round() that behaves much like the round() function in Microsoft Excel.

Documentation clarifications and updates were made for the count(), openval(), closeval(), and systemproperty() functions. The expression language is documented in StreamBase Expression Language and Functions.

In previous releases, two sets of server configuration file parameters in the data-source section for JDBC data sources were mutually exclusive. Starting with release 6.1, you can use the jdbc-batch-size and jdbc-reconnect-* parameters for the same JDBC connection.

The Reuters RMDS Subscribing Adapter and the (deprecated) Reuters Subscribing Adapter are now installed as part of the base StreamBase installation package. In previous releases, both adapters had separate installation kits. See Reuters RMDS Subscribing Input Adapter.

The Binary File Writer Output Adapter and the CSV File Writer Output Adapter both have a new Sync on flush option in the adapter settings tab in the Properties view. This option ensures that operating system buffers are written to disk on a flush event, but at the expense of a performance penalty. See Binary File Writer Output Adapter and CSV File Writer Output Adapter.

Log messages from embedded adapters (seen on the terminal that starts the containing StreamBase Server, or in the Studio Console view) are now prefixed with a string that identifies the adapter instance sending the message.

If the configuration file for a JMS Embedded Output Adapter specifies just one server and that server's specification contains just one destination, the schema of tuples processed by the adapter does not require server and destination fields (__JMSServerName and __JMSDestinationName).

The StreamBase sample formerly installed in sample/counter was removed. This sample demonstrated a way to add a sequence field to each passing tuple, but that functionality is now provided by the new Sequence operator described above.

When you load the operator sample package, the resulting Studio project includes sample EventFlow and documentation files to illustrate most operators and data constructs in the Studio Palette view. With release 6.1, this sample package includes samples for the Pattern and Sequence operators.

Release 6.1 includes a new sample in sample/ha2, which is described above in Enhanced High Availability Support and in High Availability Sample 2: Shared Disk Access.

In release 6.1, in the StreamBaseAdminClient class, modifyApplication() and modify() were deprecated. Use addContainer() instead.

In the Tuple class, the getFields() method was previously defined as being able to throw a TupleException. The getFields() method can no longer throw this exception, and the throw signature has been removed. If your code has a try-catch block specifically for this exception, remove the try-catch block.

In the Java, C++, and .NET APIs, the flushBuffer() method of the StreamBaseClient class was modified to return a helpful error message if called before enqueuing was enabled for a stream.

Starting with release 6.0, StreamBase Studio includes a visual debugger for EventFlow applications. In the SB Test/Debug perspective, standard Eclipse debugger views can now be used for debugging EventFlow applications, including the Debug, Breakpoints, and Variables views. You can set breakpoints on arcs in the application. When you run an application in debug mode, you can stop on breakpoints, pause the application, step through the application, and step into or step over modules. The Variables view shows tuple contents as execution passes through various processing stages. With the debugger running, an execution stack is preserved for the application as a whole, which allows you to review the steps of your debugging session and return to any step. The Variables view and breakpoint highlighting in the EventFlow diagram are coordinated with selections in the execution stack. See Using the Visual Debugger.

In previous releases, the Debugger View contained the command line debugger, sbdb. Starting with release 6.0, the Debugger View was removed from StreamBase Studio. The sbdb command is still delivered as a command line tool for use outside of Studio, but its use is deprecated and it may be removed in a future release.

Starting with release 6.0, you can develop unit tests for StreamBase modules to confirm that your modules return the expected result tuple given known input streams. You can create a unit test in Studio by creating a test file in .sbtest format, using the new graphical StreamBase Test Editor, or by recording the input and output streams of a running application and saving the results as a test file. You can edit saved tests in Studio with the Test Editor. The new sbtest command provides a way to run a saved test file from the command prompt. See Running Unit Tests.

StreamBase now supports profiling of StreamBase applications, which lets you gather runtime statistics for an instance of StreamBase Server over the course of an entire run. The sbmonitor window has a new profiling view. Use the new sbprofile command to gather profiling statistics from a running server, and to display a saved statistics file in CSV or simple HTML table format. See Profiling StreamBase Applications.

Starting with release 6.0, you can define schemas as named objects at the application level. Named schemas and their fields can be referenced in any component that takes a schema, and in expressions throughout EventFlow and StreamSQL applications. The EventFlow Editor now includes a Named Schemas tab that lets you store named schemas for reuse throughout your application. In StreamSQL, you can create named schemas using the new CREATE SCHEMA statement. See Schema Overview.

Unnamed schemas are still supported, and are now called private schemas in Studio and anonymous schemas in StreamSQL. A private schema is accessible only to the component in which it is defined, and to connected downstream components in the application flow.

StreamBase now supports hierarchical data by means of a new data type, tuple, which is a collection of one or more fields defined in an associated schema. A tuple schema can contain one or more nested schemas (including named schemas, described above) to an arbitrary depth. You can access tuple data in StreamBase expressions and in CSV data files. The Manual Input view in Studio's SB Test/Debug perspective shows a tuple field indented and containing its own input fields.

When developing custom Java clients, operators, adapters, or functions, you can now take advantage of the Java logging facility used internally by StreamBase code. The existing com.streambase.sb.util.Msg is deprecated and now delegates to the new logging facility. The new logging facility is based on SLF4J, which can be extended to support other Java logging interfaces; the default logging implementation is logback. The new logging facility is embedded in sbclient.jar, which makes it available for all custom StreamBase Java code. Start logging with a call to Operator.getLogger(). Studio's Java code generation wizards now include examples of using getLogger(). You can also use the logging facility in conjunction with the new Log Output Adapter described below. See Using Logging in Java Code.

This release has the following changes to the Sun JDK bundled with StreamBase.

For all platforms:

The handling of streams with wide schemas was improved, resulting in a significant StreamBase Server speedup for certain classes of applications when handling such streams.

You can now use StreamBase expressions in Java operators and adapters that you write to extend StreamBase. Your operators and adapters can access dynamic variables by means of expressions.

StreamBase EventFlow application files (.sbapp) and StreamSQL files (.ssql) are now UTF-8 encoded on all platforms. In previous releases, files were interpreted with cp1251 on Windows.

You can override the new default settings to make the server use a different encoding when reading files. Do this with the -Dfile.encoding parameter in the configuration file for the server. For example: <param name="jvm-args" value="-Xms64m -Xmx256m -Dfile.encoding=Cp1252"/>

The JDBC operator was enhanced to support stored procedures, and you can now specify an exact schema for a data source in both EventFlows and StreamSQL, instead of relying on automatic typechecking. When batching, you can now specify a timeout.

You can now use shortcuts when specifying a StreamBase URI in Java or C++ code that uses the StreamBaseURI class, or when specifying a URI for the sb* command-line utilities. The shortcuts are documented in the sburi reference page.

When using the sb* command-line utilities, in many cases, the URI from the failing StreamBase Server instance is included in the error message if a command fails. This is useful in high availability scenarios to determine the cluster member on which the error occurred.

The following features were added to support the new StreamBase HA and clustering solutions introduced in release 5.1:

In addition to the HA support changes described above, the sbmonitor command has the following new features:

The StreamSQL language was extended with the following features:

StreamBase Manager, a GUI alternative to the text-based sbmonitor command, was enhanced with new views. For Windows installations, StreamBase Manager is now an entry in the StreamBase section of the Start menu.

StreamBase Chronicle is an optional high-capacity event-store that can be used to provide persistent data storage in association with streaming applications. This release includes the following updates to Chronicle:

Contact StreamBase Systems for information on adding StreamBase Chronicle to your StreamBase implementation.

The following simple functions were added to the StreamBase expression language: coalesce_tuples(), format(), format_time(), parse_time(), tuple(), and the tuple constructor. The following aggregate functions were added: median(), withmin(), and withmax().

The min() and max() aggregate functions were extended to support any data type that can be compared with < or >, which includes int, double, long, timestamp, string, and blob. See StreamBase Expression Language and Functions for details.

Dynamic variables can now be used as bare values in aggregate expressions. In previous releases, you could write sum(K) + sum(x), or lastval(K) + sum(x), where K is a dynamic variable. Now you can write K + sum(x). The value of K at calculation time is used to find the result of the aggregate expression.

In EventFlow applications (but not yet in StreamSQL), dynamic variables can now be self-referencing in the update expression. This means update expressions can use the old value in order to do up the update. For example, the update expression for the variable counter could be counter + 1 to get incrementing behavior. This functionality can also be used to conditionally update a variable based on other conditions in the update stream.

This release includes the following new samples:

The firstapp sample and its associated Getting Started tutorial in the Help system were revised.

StreamBase Studio is now based on Eclipse 3.3.2. This allows you to install third-party Eclipse plug-ins that support or require Eclipse 3.3, including plug-ins that add version control to Studio.

StreamBase Studio's three perspectives were renamed to SB Authoring, SB Test/Debug, and SB Demos. This improves the perspective name's visibility in the default-sized perspective bar.

StreamBase Studio 6.0 provides one Run and one Debug button in the Studio toolbar, to be used for StreamBase applications and for other Eclipse Run and Debug functions you may need. In the 5.x release series, StreamBase Studio had Run and Debug buttons from the Eclipse framework, plus two separate Run and Debug buttons for StreamBase applications. Studio 6.0 restores the Run button simplicity of the 3.x release series.

Studio now supports the generation of precompiled application files (sbar files) as part of the builder running over your StreamBase projects. This significantly improves the startup time of those applications when launched from Studio. By default, generating precompiled files is disabled, and should remain disabled for modules in active development. You can enable the generation of precompiled application files at the project level, or on a per-file basis.

Studio now participates in Eclipse's refactoring feature for StreamBase application names and CSV files. When you use Refactor → Rename to rename a CSV file, or an sbapp or ssql application file, Studio detects where that resource is referenced in other parts of the current project, and automatically updates those references for you. Studio maintains a refactoring history in each project's Properties dialog. See Refactoring StreamBase Project Resources for more on this feature.

The Properties view for adapters and operators now fully supports operator parameters in the form ${parameter}.

The Copy Schema dialog invoked from a Schema properties page in Studio now allows copying a schema from a stream in the system container. For example, you can copy the error stream schema, or stat stream schema.

To help you document your EventFlow applications, you can now add a description comment for each row of a schema, for the schema as a whole, and for each EventFlow application or module as a whole.

You can now add your own icon overlays to the icons for modules, adapters, and operators in EventFlow diagrams. See Metadata Tab for instructions.

StreamBase Studio now displays a warning about an expired StreamBase license when you try to run or debug an application. In general, warnings and errors from StreamBase Server are more clearly shown: the last few lines output from a failing server process are shown in error dialogs.

StreamBase Studio inherits from Eclipse 3.3 the ability to toggle on and off the highlighting of search terms on the Studio Help pages. After you have searched for a term, look for the highlight button () in the toolbar above the Studio help text window.

The new Reuters RMDS Subscribing Adapter allows a StreamBase application to publish market data to RMDS 6 servers. This adapter complements the existing StreamBase M