High Availability Sample

HA Sample Overview

This sample shows two StreamBase Server instances running together in an active-active high availability cluster. The sample runs the same simple application, app.ssql, in two servers, one designated primary and the other designated secondary. The servers coordinate their behavior to provide highly available access to the sample application's output.

In each server, the app.ssql application runs in its own container, also named app. Each server has another container named ha that contains the HA-specific applications that coordinate the two servers. haprimary.sbapp is loaded into the primary server and hasecondary.sbapp is loaded into the secondary. Both of these HA applications use two common modules: controlmodule.sbapp and hbeventactions.sbapp.

Dequeuing from the sample application is initially disabled in both servers. Shortly after startup, the HA application in the primary enables dequeuing for the sample application.

Each StreamBase Server has a leadership status that can be set to either LEADER or NON_LEADER. Initially, the primary is designated leader and the secondary is designated non-leader. The HA applications query and set the servers' leadership status as a result of information obtained by the HAHeartbeat adapter, which runs in each server's ha application container and monitors the status of the other server.

Whenever the HA application in the non-leader determines via the HAHeartbeat adapter that the leader is not available, it changes the server's leadership status to LEADER and enables dequeuing from the sample application.

Note

See Comparing the HA and HA2 Samples for a comparison of the HA and HA2 samples.

Importing This Sample into StreamBase Studio

In StreamBase Studio, import this sample with the following steps:

From the top menu, select File → Load StreamBase Sample.
Select the ha sample from the High Availability list.
Click OK.

StreamBase Studio creates a single project containing the sample files. The default project name is sample_ha.

Sample Location

By default, the HA sample files are installed in:

On Windows: C:\Program Files\StreamBase Systems\StreamBase.n.m\sample\ha\
On UNIX: /opt/streambase/sample/ha/

When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace. StreamBase Systems recommends that you use the workspace copy of the sample, especially on UNIX, where you may not have write access to /opt/streambase. In the default installation, the path to this sample in your Studio workspace is:

UNIX:       
  ~/streambase-studio-n.m-workspace/sample_ha
Windows XP:
  C:\Documents and Settings\username\My Documents\
      StreamBase Studio n.m Workspace\sample_ha
Windows Vista:
  C:\Users\username\Documents\StreamBase Studio n.m Workspace\
      sample_ha

This Sample's Files

The application and configuration files delivered with the HA sample are described in the following table.

File	Purpose
`app.sql`	A simple StreamSQL application that you would typically replace with a more complex application of your own design. This application is loaded into a container also named `app`.
`haprimary.sbapp`	The HA EventFlow application used on the primary server. It is loaded into a container named `ha`.
`hasecondary.sbapp`	The HA EventFlow application used on the secondary server, and also loaded into a container named `ha`.
`controlmodule.sbapp`	A module used in both `ha` containers to filter and execute actions when certain HA events occur.
`haeventactions.sbapp`	A module used in both `ha` containers to process events from the HAHeartbeat adapter.
`ha.sbconf`	The server configuration file used by both primary and secondary servers.
`sbd.sbconf`	The server configuration file that provides Studio with the names of environment variables used in the HA application.

The scripts delivered with the HA sample are described in the following table.

Windows script	UNIX script	Purpose
`startprimary.cmd`	`startprimary`	Starts the primary StreamBase Server instance on port 9900.
`startsecondary.cmd`	`startsecondary`	Starts the secondary StreamBase Server instance on port 9901.
`startfeedsim.cmd`	`startfeedsim`	Runs the sbfeedsim command with a comma-separated double URI syntax that addresses both primary and secondary servers at the same time. This starts a feed simulation based on the schema of the `app.sql` application, and begins feeding generated values to both servers.
`startdequeue.cmd`	`startdequeue`	Runs sbc dequeue with the comma-separated double URI syntax, which begins dequeuing from both servers at the same time. However, only the server designated LEADER is processing tuples to be dequeued.
`common.cmd`	`common`	An environment-setting script called by all the files above.

Exploring the HA Sample in StreamBase Studio

StreamBase Studio can run only one server instance at a time, and the HA sample does not make much sense unless both server instances are running. For this reason, you must run the sample in terminal or StreamBase Command Prompt windows as described in the next section.

However, you are encouraged to load each EventFlow application into Studio to see how each is assembled.

Note

This sample makes use of configuration file operator parameters and environment variables. Operator parameters can be used by operators or adapters so that values are not hard-coded in the application. Operator parameters that are referenced within an operator must be defined within the configuration file. In the configuration file, you can set the value of an operator parameter by means of an environment variable. Putting these two features together allows you to define an operator parameter value at runtime.

See the comments in ha.sbconf for information on how environment variables must be set to run this sample.

Running This Sample in Terminal Windows

You can and should open this sample's application files in StreamBase Studio to study how the applications are assembled. However, this sample is designed to be run in UNIX terminal windows or Windows command prompt windows. On Windows, be sure to use the StreamBase Command Prompt from the Start menu as described in the Test/Debug Guide, not the default command prompt.

Open five terminal windows on UNIX, or five StreamBase Command Prompts on Windows. In each window, navigate to the directory where the sample is installed, or to your workspace copy of the sample, as described above.

In the first three windows, you will start a server and let it run. Thus, those windows can be small and remain in the background. The fourth window shows the stream being dequeued from the application cluster, so keep that window visible. The fifth window is where you enter commands to modify the running servers.
In the first terminal window, run the primary server with this command: startprimary
In the second window, run startsecondary
In the third window, run startfeedsim. The generated input stream is now sent to both servers, and is shown in this window.
In the fourth window, run startdequeue. This window now shows a running stream of tuples dequeued from one or both servers, like the following example:
```
out,2555,sb://localhost:9900/
out,3091,sb://localhost:9900/
out,8482,sb://localhost:9900/
out,520,sb://localhost:9900/
out,101,sb://localhost:9900/
out,3538,sb://localhost:9900/
out,867,sb://localhost:9900/
out,8503,sb://localhost:9900/
out,36,sb://localhost:9900/
...
```
Notice the StreamBase URI of the responding server is shown for each line, and only one server is responding. Yet, the command sent by the startdequeue script used a comma-separated list of URIs to address both servers at the same time. After resolving the variables for port number, the command sent by startdequeue resolves to the following:
```
sbc -u sb://localhost:9900/app,sb://localhost:9901/app dequeue
```
So, if the dequeue request went to both servers, why are we seeing a response only from the primary server on port 9900?
In the fifth window, run the following command to determine the status of both servers:
```
sbc -u sb://localhost:9900/app,sb://localhost:9901/app status --verbose
```
This returns a status report line for each server. Notice that the primary server on port 9900 is designated the LEADER. This is why we are seeing dequeue results only from one server, the one designated as LEADER.

You could also obtain the leadership status from each server separately, as follows:
```
sbadmin -p 9900 getLeadershipStatus
sbadmin -p 9901 getLeadershipStatus
```
Now send commands that change the leadership status of both servers at once. The following example is a single command line that may display on more than one line on your screen. This command works as shown for both UNIX and Windows:
```
sbadmin -p 9900 setLeadershipStatus NON_LEADER && sbadmin -p 9901 setLeadershipStatus LEADER
```
Notice that the dequeued output stream in window 4 has switched to showing tuples from the secondary server on port 9901:
```
out,6700,sb://localhost:9901/
out,7156,sb://localhost:9901/
out,1303,sb://localhost:9901/
out,4798,sb://localhost:9901/
out,2305,sb://localhost:9901/
out,9540,sb://localhost:9901/
...
```
Now shut down the secondary server, the one that you have just made the LEADER:
```
sbadmin -p 9901 shutdown
```
Notice the output stream in window 4 immediately switches back to the server on port 9900.
Re-run the status command from step 6. Only one server is running.
Restart the secondary server in window 2 by re-running the startsecondary command.
Experiment with various commands to learn how the servers respond. Make sure you understand the answers to the following questions:
- What output do you see in window 4 if both servers are running and both are designated LEADER? Why?
- What output do you see when both servers are running and both are designated NON_LEADER?
To shut down all servers, run the following commands
- In window 4, press Ctrl-C to stop the dequeuer.
- In window 3, press Ctrl-C to stop the feed simulation.
- sbadmin -p 9900 shutdown
- sbadmin -p 9901 shutdown