Library Articles

Measuring Application Performance

Author: Dr. John Lifter, StreamBase Systems
12-November-2006

Introduction
Measuring Throughput
Measuring Latency
Performance Metrics Example: The Split Adjusted Price Application

Introduction Developers are interested in making performance measurements for several reasons:

1. To measure the throughput and latency of an application under development.
2. To determine the comparative performance of competitive products.
3. To measure the effect of a technical innovation added to an existing product.

This article discusses the basics of making throughput and latency measurements in a StreamBase application and presents an example to illustrate these concepts in practice.

The first rule of measuring performance is to create a test system that allows measurement of the application's performance rather than the capabilities of the test system itself. This objective can be particularly challenging for real time applications such as StreamBase as the rate at which input is delivered from an external source to the application will affect the measurements.

When comparing the performance of multiple products, the cardinal rule is to ensure that the application used to generate the metrics has been built using a commercially available version of the product. Following this rule ensures that the results are not tainted by being derived from applications that have been implemented, or improved, through custom coding that is not supported in the commercial product or by using an optimized, or unreleased, version of the product. You should also monitor whether benchmarking objectives have been dropped from the application, or whether the benchmarking requirements have been changed, to improve the performance of a specific product.

If you are integrating additional functionality into an existing product, you should be certain that you have a test application that truly measures the parameters your innovation is designed to affect. If your innovation has no effect on the parameters being monitored, you will not be able to demonstrate incremental value.

When measuring throughput and latency of an application under development, it is important that small incremental changes to the application be followed by data gathering and analysis. Do not try to change multiple factors simultaneously in an attempt to guess the most effective design or configuration. Take your time to make certain that you understand the impact of each change. Also, be certain you understand the impact that hardware differences can have on your results and analyses.

Measuring Throughput While throughput is simply the rate at which data are processed by the application, where in the application the determination is made can affect the calculated results. In a StreamBase application throughput is expressed as the number of tuples processed per second.

Some applications may simply process a tuple from start to finish, in which case throughput can be measured at any convenient point in the application.

Other applications may rapidly input tuples, which are then internally buffered and processed at a much slower rate. In this situation, it is essential that the performance measurements are taken near the application's output rather than near the application's input. This second situation often arises when the application includes components (such as StreamBase operators or modules) that are running in separate threads; the performance measurements must be taken after these components complete their processing and have returned their output to the main thread.

Finally, the number of tuples emitted from an application may differ from the number of tuples submitted to the application. For example, an application that calculates averages over some period of time will accept a variable number of tuples but emit only one tuple at the end of the time period. In this situation, throughput might be interpreted to be the rate at which tuples are accepted by the application and not have any relationship to the rate at which tuples are emitted by the application.

In order to calculate throughput, a counter must be incorporated into the StreamBase application. With StreamBase's StreamSQL EventFlow approach, a counter may be implemented through a dynamic variable, or an aggregate operator, or by maintaining an entry in a query table. The StreamBase 3.5 StreamSQL text language supports the aggregate operator and query table options; later versions may also support the dynamic variable approach.

Throughput measurements also require that the tuples used in computing the metrics include a timestamp field so that the interval between sequential tuples can be monitored. This timestamp field should not be initialized by the client application; it must be set by the target application. Additionally, the timestamp field should be set as part of the throughput analysis; that is, it should not simply be added when each tuple enters the application since, as noted above, the rate at which tuples enter the application may be greater than the rate at which the application processes the tuples.

The following diagram shows the EventFlow operators that provide the capability to monitor throughput.

The AddTimeStamp map operator adds a timestamp field to each tuple arriving on its input stream. The modified tuple is passed to the downstream operator.

The ThroughputInterval aggregate operator maintains a counter and retains the timestamp values for the first and last tuples collected in its window. For example, this operator might be configured with a window size and advance of one million tuples, in which case it would continue to collect data until the counter totals one million. At this point, the operator would emit a tuple containing three fields: the count; the value of the timestamp field in the first of the one million tuples; and the value of the timestamp field in the one millionth tuple. This emitted tuple is passed to the downstream operator.

The ThroughputCalc map operator uses the counter and timestamp values to compute the throughput and then emits a tuple containing this single field.

throughput = 
  count/(to_seconds(lastTimestamp - firstTimestamp))

The values used for the size and advance window settings are dependent on the actual number of tuples submitted to the application. Typically, the size and advance settings should be 10-25% of the number of tuples submitted. When following this guideline, the application will compute the throughput 4-10 times while processing the input data and it will be possible to judge the consistency and stability of the analysis. Since both the application(s) submitting data and the StreamBase application itself may experience some startup delays, the initial throughput value may be smaller than subsequent values and should not be considered representative of the system's capabilities.

Measuring Latency Latency is the amount of time needed to process a tuple completely through the application. To make this calculation, a timestamp field containing the start time must be added to the tuple immediately after it enters the application and another timestamp field must be added to the tuple immediately before it exits the application. Then the period of time between these two measurements is compared to a predetermined latency objective. For example, if the latency requirement is that each tuple must be processed within 50 milliseconds, the following expression could be used to confirm that each tuple was processed within specification.

(to_seconds(tupleExitTime - tupleArrivalTime)) > 0.050

The following diagram shows the EventFlow operators that provide the capability to monitor latency.

The AddArrivalTimeStamp map operator is positioned immediately following the input stream component and adds a timestamp field to the input tuple.

The dashed arc represents all of the other operators in the StreamBase application.

The AddExitTimeStamp map operator and LatencyCalc filter operator are positioned immediately before an output stream component. The AddExitTimeStamp operator adds another timestamp field to the tuple, while the LatencyCalc operator uses the latency expression to identify tuples that were processed too slowly and only emits a tuple if the latency restriction was not met.

Performance Metrics Example: The Split Adjusted Price Application This application accepts an incoming stream of stock trade data and an incoming stream of stock split data and produces an outgoing stream of stock trade data with split adjusted pricing. Since a given stock may split more than once, it is necessary to maintain a running composite of the total split factor; it is this composite split factor that is used to calculate each split adjusted price.

The application's data processing requirements can be specified through several pseudo-SQL statements. Tuples in the incoming stock trade stream contain four fields:

Tick (
   tradeTime=double,
   symbol=string(5),
   volume=integer,
   price=double
)

Tuples in the incoming stock split stream contain three fields:

Split (
   symbol=string(5),
   splitTime=double,
   splitFactor=double
)

The composite split factor information stored for each stock includes the symbol and the composite split factor.

Storage (
   symbol=string(5),
   compositeSplit=double
)

Each time the application receives a new split factor, the following operation should be executed:

UPDATE Storage
FROM Split AS S
SET (compositeSplit=compositeSplit * splitFactor)
WHERE S.symbol = Storage.symbol

And each time the application receives a stock trade, the following operation should be executed:

SELECT T.symbol, price=T.price * S.compositeSplit,
       T.volume, T.time
FROM Tick AS T, Storage AS S
WHERE S.symbol = T.symbol

A maximum acceptable latency to complete each operation on the stock trade stream, for example, 50 milliseconds, must be enforced.

This application can be implemented as a single StreamBase module to which the additional operators needed to determine latency and throughput are added. The module performs the processing described in the pseudo-SQL statements. It does not include any components related to generating the throughput or latency metrics.

Once the throughput and latency capabilities are added to the application, the module becomes slightly more complex. Notice how the Map operators AddArrivalTimeStamp and AddExitTimeStamp are integrated within the business logic processing while the Aggregate and Map operators used to calculate throughput, and the Filter operator used to determine latency, are added after the business logic processing has completed.

AddArrivalTimeStamp Map Operator

This operator adds a timestamp field, named tupleArrivalTime, to the stream. This timestamp establishes the time that each tuple enters the module.

AddExitTimeStamp Map Operator

This operator adds a timestamp field, named tupleExitTime, to the stream. This timestamp establishes the time that each tuple completes processing within the module.

LatencyCal Filter Operator

This operator calculates the processing latency for each stock trade. The filter includes a single predicate with the following expression, which selects tuples that took longer than 50 milliseconds to process for emission on the operator's first output port.

(to_seconds(tupleExitTime - tupleArrivalTime))>0.050

Tuples that satisfy this predicate expression, which are the tuples not processed within the latency requirement, are sent to a dedicated output port and written to a file by the ProcessingTooSlowWarning file output adapter.

ThroughputCalc Aggregate Operator

This operator is responsible for maintaining the count and timestamp values needed to calculate the throughput. On the Properties view, Dimensions tab, set the size and advance entries to values appropriate to the application being tested. For example, if the input data includes 5 million rows, a suitable entry would be 1000000, so that the throughput calculations are performed 5 times.

This operator emits a tuple containing three fields corresponding to the aggregate expressions: count(), firstval(tupleExitTime), and lastval(tupleExitTime). Note that the throughput calculation requires only the timestamp values generated by the AddExitTimeStamp operator.

ThroughputResults Map Operator

The ThroughputCalc operator can only output results from aggregate expressions. The actual throughput calculation must, therefore, be performed in a following Map operator. The ThroughputResults operator uses the results emitted by the Aggregate operator to calculate the actual throughput using the expression:

throughput = 
  count/(to_seconds(lastTimestamp - firstTimestamp))

The results of this throughput calculation are written to a file by the TimingIntervals file output adapter.