Capacity Planning

This topic outlines some capacity planning considerations for streaming applications. This task is done outside of StreamBase Studio.

Your planning should evaluate both the steady state capacity and peak (spikes) capacity requirements.
Note that a system can handle spikes in the number of input tuples per second by falling behind real-time for a period of time, and then catching up afterwards, as long as system resources are not exhausted during the spike.
Memory resources:
- The size of the page pool is proportional to the number of tuples in the system at a given time. The page pool is dynamic, and will grow at the expense of other memory consumers.
- Page pool is set by the following elements in the sbd.sbconf configuration file:
```
<page-pool>
    
    <param name="max-client-pages" value="2048"/>
</page-pool>
```
  In the skeleton configuration for sbd, the page-size value is commented out. This is the size in bytes per page, and must be multiple of the system page size: 4096 bytes for x86, 8192 bytes for SPARC. The StreamBase Server (sbd) defaults to the appropriate minimum value for the machine's architecture. If you change the page-size value, it must be a multiple of the system page size.
  
  The max-client-pages value sets the maximum number of pages that a dequeue client connection can allocate. The sbd process will disconnect clients that try to allocate more. This parameter is designed to protect sbd from a slow or hung client. Therefore, to understand the resulting number of bytes allowed, multiple the max-client-pages value by the page-size value. A max-client-pages value of 2048 pages results in about 8 megabytes. If you want to enable a dequeue client connection that would allocate unlimited memory use, change the max-client-pages value to 0.
  
  Note: The number of bytes allocated to each client is the product of the max-client-pages and page-size settings. Therefore, increasing page size has the effect of reducing the potential number of clients.
- Operators that have state use memory
- Monitor memory usage to determine headroom
StreamBase Monitor (sbmonitor) displays information that can be useful in understanding the performance of your StreamBase application. This information includes for each node: the size of the page pool, the number of tuples each box processes each second, the time required to process each tuple, and how much CPU time is used by each thread. For details, see Using the StreamBase Monitor.
When you evaluate throughput, test headroom by using the StreamBase Feed Simulator on a representative trace file. Increase the data rate to faster than an estimated real time rate, such as 5x or 10x. For details, start in Using the Feed Simulation Editor.
If memory or throughput headroom is inadequate (that is, if you are already using the maximum processor speed and memory), increase capacity by adding multiple processors. You can also investigate using a cluster of StreamBase Servers using high availability techniques. See Clustering and High Availability for more information.