Using the External Process Operator

This topic explains how to use the External Process operators and the configuration settings you can make on each operator's Properties view.

Introduction

The External Process operators appear in the Global Operators & Adapters drawer of the Palette view in StreamBase Studio, not in the Operators drawer.

The External Process operators are Java operators that provide a way for StreamBase applications to run arbitrary operating system commands as if typed at the shell command prompt for the current operating system. This feature is especially useful in StreamBase HA configurations, where an application in one container might need to send an sbadmin command to an application in another container or on another StreamBase Server.

There are two External Process operators with the same functionality. The two operators differ only as follows:

  • External Process operator — uses a table to specify the command to be run, and all its arguments.

  • External Process Command Line operator — uses a StreamBase expression to specify the command to be run and its arguments. This version of the operator also has an option that allows you to include the resolved command line as run in the tuple output of the operator.

Both versions of the External Process operator are described on this page.

Running External Commands

The External Process operators incur operating system overhead and should not be considered for any part of a StreamBase application that requires fast performance. Neither operator should ever be called repeatedly to perform a task. These operators are designed to be called on rare occasions, such as in a failover event in a highly available StreamBase Server cluster.

You can use these operators to run any command on the PATH of the StreamBase Server instance that is running the operator. You can specify command switches, command arguments, and environment variables for the command you want to run, and you can modify the switches, arguments, and environment variables at runtime while calling the specified command. You can specify variables as part of the command to be run, and have those variables resolved at runtime.

Caution

This operator has the same permissions as the containing StreamBase Server to run any operating system command in the PATH and environment of that server. Use caution in specifying the command for this operator to run.

The specified operating system command must be locatable on the PATH of the containing StreamBase Server, or it can be specified by absolute path or a path relative to the containing StreamBase Server's current working directory.

Ports

By default, the External Process operators have one input port and one output port:

  • Input Port. Both operators take commands from the input tuple. The schema of the input tuple is application-specific, but typically includes string fields containing switches, arguments, or parameters for the command to be run. Each operator runs its specified command on receipt of a tuple on the input port, optionally using fields in the incoming tuple to specify switches or parameters for the command.

  • Output Port. Both operators send a tuple on the output port whenever they run the specified command. By default, the schema of the output tuple is the following:

    Field Name Type Size Description
    stdout blob undefined Contains the standard output of the command that was run.
    stderr blob undefined Contains the standard error of the command that was run.
    exitcode int 4 The exit code value returned from the command that was run.

    The schema of the output tuple can be optionally extended by using the following options, described in Parameters Tab.

    • Include the command line in the tuple output (External Process Command Line operator only)

    • Include input tuple in output

    • Include arguments and environment variable in output

You can add an optional Error Output Port, which outputs a StreamBase error tuple for any error thrown by the operator.

Examples

The HA samples shipped with StreamBase include several uses of the External Process Command Line operator. See High Availability Sample and High Availability Sample 2: Shared Disk Access for instructions on finding and running the samples.

General Tab

An instance of either External Process operator has the following properties on the General tab in its Properties View in StreamBase Studio:

Name: Every application component must have a unique name. Use this field to specify or change the component's name. The name must contain only alphabetic characters, numbers, and underscores, and no hyphens or other special characters. The first character must be alphabetic or an underscore.

Class: A read-only field that shows the fully qualified class name that implements the functionality of this operator. Use this class name when loading the operator in StreamSQL programs with the APPLY JAVA statement.

Start with application: If enabled, this operator instance starts as part of the containing StreamBase Server instance. The default and recommended setting is enabled.

Enable Error Output Port: Check this box to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports and Error Streams to learn about Error Ports.

Description: Optionally, enter a description to briefly describe the component's purpose and function. In the EventFlow canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.

Parameters Tab

An instance of either External Process operator has the properties described in this section on the Parameters tab in its Properties View in StreamBase Studio.

The Parameters tab for the External Process operator begins with the Arguments table, where you specify the command to be run:

Arguments

Specify the command to be run in row 1 of the Arguments section, using a relative or absolute path to the command. In subsequent rows, you can optionally add the exact switches and arguments with which to run the command, split into separate rows on whitespace boundaries. That is, if you want to run the command sbadmin setLeadershipStatus NON_LEADER, specify the three parts of the command on three rows. To run the same command except specifying a StreamBase URI, use five rows as follows: 

sbadmin
-u
sb://remotesbhost:10001/appcontainer
setLeadershipStatus
LEADER

Any row of the Arguments section can contain the name of a field in the incoming input tuple. In this case, the contents of that field are substituted when the command is run. For example, your input tuple might contain a field named serverURI that contains the programmatically determined URI of one member of a StreamBase Server cluster. In this case, specify five rows in the Arguments section, containing: 

sbadmin
-u
serverURI
setleadershipStatus
NON_LEADER

The Parameters tab for the External Process Command Line operator starts with two fields:

Command Line

Specify the full command to be run as a StreamBase expression. You must surround the entire command line with single quotes to ensure the expression is evaluated as an expression, even if your expression contains only literal strings.

Like the table-specified command described above, the command expression can contain the name of one or more fields in the incoming input tuple. In this case, the contents of each field are substituted when the command is run.

Unlike the table-specified command described above, the use of an expression allows you to evaluate variables at runtime, instead of passing the variables to the operator in the input tuple. For example, an External Process Command Line operator in the HA2 sample specifies the following command. This is a single command on one line, broken into two lines for clarity: 

'sbadmin -u\"' + systemproperty("streambase.uri", "URI-not-specified", 100) 
    + '\" setLeadershipStatus LEADER'

This command fills in the argument for the –u option with an expression that evaluates to the StreamBase URI for the currently running StreamBase Server, or to the string URI-not-specified if the expression fails to evaluate.

Include the command line in the tuple output

If you enable this checkbox, the schema sent to the output port is extended with a field containing the exact command line run, with all variables resolved. Use this checkbox for debugging while developing your operator.

The remaining fields in the Parameters tab are the same for either version of the External Process operator:

Include input tuple in output

If you enable this checkbox, the schema sent to the output port is extended with the fields of the tuple received at the input port.

Input tuple field prefix

If you use the checkbox above, the string you specify in this field is prepended to the contents of the input tuple field sent to the output port. The default value is input_.

Include arguments and environment variable in output

If you enable this checkbox, the schema sent to the output port is extended with fields containing the path of the command run, the switches and arguments run, and any environment variables specified in the operator.

Arguments field prefix

If you use the checkbox above, the string you specify in this field is prepended to the contents of any argument fields sent to the output port. The default value is arg_.

Environment variables field prefix

If you use the checkbox above, the string you specify in this field is prepended to the contents of any environment variable fields sent to the output port. The default value is env_.

Environment

Use the Environment section to specify any environment variables needed by the command you want to run, that are not already in the environment of the containing StreamBase Server, or whose values you want to change for the purpose of running the specified command. For each row, specify the variable name, equals sign, and value. For example:

Concurrency Tab

Run this component in a separate thread

This option causes the server to process the component's requests concurrently with other processing in the application. You can distribute the processing of the threads automatically across multiple processors on an SMP machine.

If this is a compute-intensive component and you know that it can run without data dependencies on other components in the StreamBase application, you may be able to improve performance by enabling this option.

Caution

These features are not suitable for every application. For details, see Execution Order, Concurrency, and Parallelism. It includes important guidelines for the use of these features.

Run in parallel threads

If you checked the first option, you can also choose this option, which causes the server to run multiple instances of this component. That is, each instance runs in its own thread. At run time, tuples are dispatched to particular instances based on the Key Expression value (which must evaluate to an int).

Related Topics

Back to Top ^