StreamBase Documentation
Quick Facts: StreamBase Terms and Components
When you start using a new product, you might ask:
- What is it?
- How will it help me do my job?
- What is the development cycle and environment?
- What are the product's components, and what new terms do I need to understand?
This topic addresses those initial questions. Subsequent topics in this Getting Started Guide walk you through the process of building your first StreamBase application, to help you get acquainted with the product's features.
What is StreamBase?
StreamBase™ is a new type of computing platform, designed specifically to meet the performance requirements of high-volume, real-time streaming applications. At its core, StreamBase implements a Stream Processing Engine™ (SPE) that uses a unique Inbound Processing Architecture. In contrast to the traditional database model where data is first stored and indexed and then subsequently processed by queries, StreamBase processes the inbound data while it is "in flight," as it streams through the server.
The StreamBase Processing Engine, processes inbound tuples in real time, using the business logic that you defined and delivering results as they are produced - typically in milliseconds. Storage of the data records can be done, but it is optional. In many cases, client programs that you write will "consume" the processed results in real time, and take appropriate actions based on your code's direction.
StreamBase can also connect to an external data source, enabling applications to incorporate selected data into the application flow, or to update the external database with processed information. With StreamBase Enterprise Edition an application can connect to a JDBC-compliant database.
The following diagrams illustrate the conceptual differences between the older Outbound Processing model and the new StreamBase Inbound Processing model.
Outbound Processing Model Used by Traditional Database Products:
Inbound Processing Model Used by StreamBase:
In StreamBase applications, data records called "tuples" flow through the query and application processing steps, which may in turn transform this data while it is moving. StreamBase provides a set of data processing units, the "StreamBase operators," that you will define and customize to apply your business logic on the streaming data. With the operators and a supporting cast of "data constructs," you can perform such tasks as:
- Applying aggregating functions to windows of real-time data
- Computing new field values by applying mathematical expressions, adding new fields, or dropping fields, from the data streams
- Filtering data into separate streams for conditional processing
- Performing approximate buffered sorts of the real-time data
- Joining previously split streams of data based on key values
- Within the StreamBase application's process, populating a shared data table so that other portions of the application can lookup data based on key values
StreamBase also features a graphical rapid-development environment, called StreamBase Studio. It lets you design, test, and deploy streaming applications. StreamBase users report that by using StreamBase Studio, their development teams are able to build streaming applications in less time and at lower costs, when compared with their efforts to create or redesign streaming applications.
How Will StreamBase Help Me do my Job?
For developers and administrators, StreamBase provides graphical, text-based, and command-line tools to build streaming applications quickly and efficiently.
In StreamBase Studio, a view called the "Authoring Perspective" lets you design the StreamBase application either graphically, using the text-based StreamSQL language, or with a combination of both. As part of this design step, you define properties for the operators and other components that will apply your business logic on the inbound data. We'll discuss the Authoring perspective's features in the next section.
Whether you are developing the application on Windows or Linux, you can use StreamBase Studio to start a StreamBase Server instance. The StreamBase Server loads the application, processing the inbound data according to your definitions in the operators, and responding to any other requests, such as requests for data from Java, C++, or .NET clients that you write. The StreamBase Client Libraries are documented, and we provide examples of writing "producer" clients that enqueue data to the application, and "consumer" clients that dequeue data from the application.
Once the server is started, StreamBase Studio provides a second perspective, "Test/Debug," that contains tools to enqueue or dequeue data on the running application, or run a debugger to pause the application and step through its processing units, or view performance statistics.
Finally, StreamBase provides client APIs (Java, C++, and on Windows, .NET), that you can use to develop client applications that enqueue to (or dequeue data from) StreamBase applications.
The following diagram illustrates the StreamBase development cycle and platforms.
StreamBase Components and Terms
So far you have learned that StreamBase Studio has a graphical development environment for building StreamBase applications. Let's take a closer look at the product's components and terms.
For starters, what is a "stream"? A stream is a sequence of data records called tuples. A tuple is similar to a row in a database table. In a high-volume streaming application, the streams flow into the application and are processed in real time, before any (optional) on-disk storage occurs. The tuples in a stream have a schema, which defines each field's name, position, data type, and size. A field is simply a named value in a tuple, such as a stock's Symbol in a trade record. A field is similar to a column in a database table. The following diagram illustrates a stream of data containing tuples:
During the course of processing streaming data, you can add, remove, or modify the fields that comprise the tuples, in a highly dynamic fashion. In other words, the initial schema of the stream does not determine its contents during the entire processing execution.
Of course, an important step in building any application is determining which fields will comprise the inbound data, how that data should be organized, named, and typed (examples: string, int, boolean, double, timestamp), and usually which field(s) will contain key values. Often, though, you have already made those metadata decisions for your existing applications, and you want to know how to define the schemas for StreamBase. We will introduce the schema definition steps later in this guide, when you follow the "Your First StreamBase Application" tutorial. For now, notice how in the screen below the schema for a sample "input stream" is defined:
The sample screen above is a peek at just one of many features in StreamBase Studio, our graphical development environment. Let's take a step back, though, and learn how data gets into a StreamBase application.
How Does Data Get into StreamBase?
The entry points for a StreamBase application are called "input streams." Your application may define one or more input streams. The previous sample screen showed the schema definition for a "StockIn" input stream. But how is data sent to a named input stream? The options are:
- Via an adapter program that performs the conversion from an external source (such as a stock market data feed) into the StreamBase protocol. For example, StreamBase provides an adapter that converts TIBCO® Rendezvous™ messages into StreamBase tuples, and vice versa.
- Via a "producer" or "enqueue" client program (Java, C++, or .NET) that submits inbound data to one or more Input Streams defined in your StreamBase application. A client extends the documented StreamBase API to interact with a running StreamBase application.
- From an external database connected to your StreamBase application, using SQL queries to read the data. Currently you can connect to a JDBC database. This method enables your application to work with long-lived ("historical") data along with the streaming data from its input streams.
- And you can use StreamBase commands in terminal windows to perform an even greater level of enqueuing operations with running applications.
- While designing your application, you can use a graphical feature of StreamBase Studio, called the Feed Simulation Editor, to submit randomly generated data or pre-recorded data from an input file such as a comma-separated value (CSV) file. The Feed Simulation Editor includes defaults or customized settings to control the rate of inbound test data.
How Do I Get Data from StreamBase?
You can define Output Streams that serve as named exit points in your StreamBase application. Called a "dequeue" operation, there are several ways to accomplish it:
- Via an adapter program that can return the processed data from StreamBase to the original external source.
- Via a "consumer" or "dequeue" client programs (Java, C++, or .NET) that listens for outbound data on the application's streams. A client extends the documented StreamBase API to interact with the running application.
- Using insert or update queries to an external JDBC database that is connected to your StreamBase application.
- You can also use a graphical feature of StreamBase Studio to view the Application Output.
- And you can use a StreamBase command to dequeue tuples from streams.
The following high-level diagram illustrates the different ways that data can enter and exit a running StreamBase application.
In the preceding diagram, note that the StreamBase adapters are available separately from the StreamBase base kit, and each adapter has its own installation procedure.
StreamBase Studio Development Environment
StreamBase Studio supports both graphical and text-based application development:
- EventFlow applications are are XML files that are edited graphically using StreamBase's Studio's Application Diagram Editor.
- StreamSQL applications are coded using a query language to describe the application and query streams of data. StreamSQL Applications are stored in text files with an
.ssqlextension.
You can also build parts of applications - modules - that can be included as components in other applications. In fact, any EventFlow or StreamSQL application can be used as a module in another application.
Building EventFlow Applications
To create an EventFlow application in StreamBase Studio, you can "drag and drop" the icons for various components from a Palette View to the Application Diagram Editor's drawing canvas. As noted earlier, the components include:
- Input Streams and Output Streams.
- Data processing units called "Operators" that can apply your business logic on streaming data, such as aggregating tuples, or merging tuples, or retrieving data from a table. Each operator performs the work that you specify in its Properties view.
- Data constructs that store information used by an associated StreamBase operator. For example, Query operators are associated with Query Table data constructs.
Later, you can read about these components in the Authoring Guide. And we will use some of these components in the "Your First StreamBase Application" tutorial that appears in this Getting Started Guide.
Here's an illustration that shows the "drag and drop" step, from the left-side Palette to the Application Diagram Editor's drawing canvas.
Once a component is placed on the canvas, you define its runtime behavior by setting parameters and (for most components) defining expressions in the Properties View. In this next sample screen, notice how the MyQueryTable is selected on the lower portion of the canvas, and its Properties view contains tabs that allow us to define it:
Building StreamSQL Applications
If you prefer to work with SQL queries instead of boxes and arrows, you can build applications using StreamSQL. StreamSQL will look familiar to SQL programmers, but there are some important differences. While SQL operates on tabular data that is stored somewhere on disk or in memory, StreamSQL supports operations on streams of continuously changing data.
You can use any text editor to write StreamSQL applications. However, coding in StreamBase Studio offers several advantages:
- The Authoring Perspective includes a StreamSQL editor that facilitates writing StreamSQL queries. The editor's Content Assist feature makes entering keywords easy, and StreamBase checks your syntax as you work.
- StreamBase Studio views make it easy to combine StreamSQL applications and other StreamBase components. For example, you might want to reference a StreamSQL application within a StreamBase EventFlow application. Or, going the other way, you might embed existing StreamBase applications as modules within a StreamSQL query.
The following image shows the StreamSQL Editor in the Authoring Perspective:
Applications As Modules
As we mentioned earlier, an EventFlow or StreamSQL application can be used as a module in another application. For example, you can create an application that performs a common task, and include it in any number of other applications. This reuse can save time in both development and maintenance. Obviously, the functionality in the module must "make sense" for the application that uses it. For example, both must use compatible data.
If you are working on an EventFlow application, StreamBase Studio lets you simply drag an existing application from the Projects View to the canvas of an Application Diagram. In the following figure, an StreamSQL module has been added:
If you are writing a StreamSQL application, you can include an existing application as a module.
It's Time to Start Building an Application...
We have covered some of the key StreamBase components and terms, and the ways in which you can create applications. You can learn much more about the product by following the remaining topics in this guide, where you will have an opportunity to build your first StreamBase application.