Designing Highly Available Applications

This page builds on information in Primer on StreamBase Clustering and HA. Be sure to read that page first, especially the list in StreamBase Components That Support Clustering.

Before beginning your HA application's design, establish which HA problem you are tasked with solving. Have in mind exactly which problem you're seeking to solve with HA techniques.

See What is StreamBase Clustering? for a discussion of the goals of HA solutions.

There are several components used to assemble StreamBase clustering and high availability (HA) solutions, as listed in StreamBase Components That Support Clustering. Each solution uses these components in a different way. Remember that making a highly available application is a complex event processing problem. Take advantage of StreamBase's stream processing fundamentals to find creative ways to implement a highly available application that meets your site's design goals.

In StreamBase, there are common design components for clustering and HA applications. This page describes these HA design features as patterns for your application to build on.

The use of containers is essential in an HA application. Containers allow you to separate your main application's logic from the HA-related logic. You can start and stop the application in a container dynamically, and you can dynamically start and stop enqueuing and dequeuing of individual ports.

This means your HA container can send commands to your application container to enable and disable processing based on events such as a change in leadership status, or missed heartbeat from the other member of a cluster pair.

The following diagram illustrates the separation of application and HA logic into containers. The system container is always present in every sbd instance.

Use the HA Heartbeat adapter in your HA container's application. The adapter is used in pairs, with one member of the pair designated as heartbeat server. The HA Heartbeat adapter in the other StreamBase Server's HA container is designated the client of the heartbeat server, and does the work of connecting to the heartbeat server, and of reporting failure to connect.

See HA Heartbeat Input Adapter for instructions. See the High Availability Sample for an example implementation.

Event tuples are sent on the system.control output stream to announce any change in leadership status of the containing sbd server, and any container startup event. The application in your HA container can monitor the system.control output stream for these events and take appropriate action.

For example, the HA logic in the HA container on the secondary server can monitor for changes in leadership status. When status changes from NON_LEADER to LEADER, the HA logic can send an sbadmin command to the application container to start the application in that container, or, if the application is already running, to start enqueuing and dequeuing.

See Using Control Stream Features.

The External Process operators provide a way for a StreamBase application to invoke a command-line utility at the operating system shell level. You can take advantage of these operators to let the HA logic in the HA container send sbadmin or sbc commands to the application logic in the application container of either server.

See Using the External Process Operator.

To implement a hot-warm template means having the secondary server's application logic running and processing tuples, but not sending output tuples downstream. (See Hot-Warm Server Pair Template for more on the hot-warm template.)

To suspend output tuples, the output streams in the secondary server's application logic can be started with the --dequeue=DROP_TUPLES option. Or to drop messages on input, use the --dequeue=DROP_TUPLES option. On a failover event, send an sbadmin modifyContainer command to re-enable enqueuing or dequeuing with the ENABLED status. See sbadmin in the Reference Guide for more on this feature.

Shared disk access does not mean allowing two separate StreamBase Server instances on separate servers to write to the same disk query table at the same time. Instead, consider putting all application logic that implements disk query table access into its own container in each server.

When a failover event is detected, such as a change in leadership status, or a failure to detect an HA heartbeat, the HA logic on the secondary server can send an sbadmin command to start the container with the disk query table. In this way, you can ensure that only one application is writing to the shared disk query table at the same time.

To minimize startup time when restarting an sbd server, use precompiled applications. See Precompiled Application Archives.

If you're porting an existing StreamBase application to become highly available, take the time to clean up the sbd.sbconf files you're currently using for your application. Starting with release 5.1, most attributes of the high-availabilty element are deprecated, and some new elements were added. See StreamBase Server Configuration XML in the Reference Guide.

You can either:

Most of the design patterns on this page are illustrated in High Availability Sample and High Availability Sample 2: Shared Disk Access. Study the sample applications thoroughly before beginning your own application design.

To ramp up to speed quickly, you can use High Availability Sample as a starting point, substituting your site's stream processing StreamBase application in place of the sample's app.ssql file. Or use High Availability Sample 2: Shared Disk Access as your base, substituing your stream processing application in place of that sample's qtapp.sbapp.