StreamBase

Trading on Macroeconomic News

Emily — Thu, 29 Sep 2011 02:32:44 +0000

We recently hosted a webinar – “Trading on Macroeconomic News” with Need to Know News (NTKN), a subsidiary of Deutsche Bourse Group Company. With pre-built connectivity to NTKN’s AlphaFlash, we demonstrated how a trading firm could easily incorporate low-latency news algo data feeds into their trading systems. In the 50-minute webinar, we built a basic algo that will trade EUR/USD based on the data of non-farm payroll employment report.

Trading on unstructured or non-traditional data is a topic that is gaining more and more attention. Adam Honoré (@adhonore) at Aite Group reported that 35% of quant firms are now exploring machine-readable news feeds in some way, compared to only 2% in 2008. The development of electronic trading and explosion in data volumes over the past decade play key roles in this phenomenon. Yet, most market participants are still trying to figure out effective and efficient ways of transforming unstructured data into increased profits or prevented losses.

One fund, Derwent Capital has reported success already, announcing a 2% profit from its $40 million dollar fund, with investment strategies based on Twitter mood – derived from a paper published by Johan Bollen and Huina Mao at Indiana University, Bloomington, and Xiao-Jun Zeng at the University of Manchester. In 2009, Zhi Da and Pengjie Gao at Notre Dame and Joseph Engelberg of UNC also proposed correlation between Google Search Volume Index (SIV) and stock prices.

Here at StreamBase, we were one of the first financial technology firms to offer clients a way to connect to unstructured social media content, such as Twitter. Media interest in this area has been high, with coverage from CNBC, the FT, the WSJ, and USA Today; though enabling customers to trading on social media, is really only a small percentage of what we do. The scalability and flexibility of StreamBase CEP makes it a great platform to handle unstructured data in real-time.

Leading economic indicators, such as GDP, consumer confidence index, interest rates and unemployment reports are commonly used for signal generation but a firm needs to make extra efforts to really gain a competitive edge with unstructured data. Simply categorizing news into positive, neutral or negative might not be sufficient enough for effective algorithmic trading. In 2010 we demonstrated how StreamBase can be used with a LingPipe, a natural language processing tool kit that can understand the implications of a news story vs. market expectation.

Although it is rare to see signal generation or alpha seeking strategies solely based on news events, some sophisticated firms incorporate unstructured data and sentiment analysis to develop algos that react to market volatility after a particular news item breaks.

Processing of unstructured data has relevance not only for alpha-generating strategies but also for risk managers and regulators who could use machine-readable news and sentiment analysis to simulate a firms’ exposure or market fairness under extreme events, for example, the recent downgrade of government credit ratings.

Trading on unstructured data will continue to be a popular topic. If you have any new ideas for using other non-traditional data for trading, please let us know. Below are some additional links which might prove interesting.

The Future of FX Trading Technology – Indicators from the 2011 StreamBase FX Survey

Mark Palmer — Tue, 23 Aug 2011 06:01:00 +0000

This year’s third annual StreamBase survey on technology trends in the FX trading market had some interesting findings. The survey was conducted from March to May 2011, drawing 135 FX industry respondents, 55% sell-side and 36% buy side. 43% of the respondents use algorithms for trading, and of those 64% actually use their own in-house developed algorithms (that is, they don’t use algorithms supplied by a vendor or bank). The survey was designed with input from the TABB Group and AITE Group. Here are the three main areas that leapt out to me as the CEO of StreamBase – many of our customers using CEP as part of their FX technology platform.

1) Increased Regulation is the Leading Fear, and Real-Time Risk Management for FX Will Soon Emerge

52% of the respondents said they believe regulation will have the biggest impact on their business, and 37% of buy-side firms expect to see an increased call for market transparency as a result of recent lawsuits regarding FX transaction costs. But what are the implications of this expectation of increasing regulation?

In the past 10 years, the FX markets have lagged the equities market in terms of technology adoption. In many cases, innovation spreads from equities groups to other asset class such as FX – In the past 18 months, pre-trade risk management has become a leading initiative for equities trading groups – but will it soon become common for FX trading? Sang Lee, cofounder and managing partner of Aite Group, thinks so: “it’s critical that firms manage their risk in real-time,” said Sang, in reaction to our survey results.

2) FX Liquidity Aggregation is Becoming Increasingly Complicated

94% of the survey respondents said they are doing some form of FX liquidity aggregation, up from 65% last year. This finding was initially surprising to me; but we see a lot of innovation in FX with CEP, and plenty of firms are adding new aggregation functionality with more connectivity to banks as well as the ECNs. This is undoubtedly leading to more sophisticated real-time aggregation of FX liquidity sources. A skeptic might say, “that sort of result is to be expected from a CEP vendor”, yet less than 10% of the survey respondents were actually StreamBase users, so we believe that this is reflective of an industry trend, not our sample.

What’s more is that 34% of firms are planning to improve or add aggregation functionality – this suggests continued innovation in aggregation applications and platforms – such as demands for new FX sources, direct bank connectivity, and new features to be coming from FX venues. So, keep watching the FX space for more sophisticated aggregation technology and techniques.

3) It’s Better to be (a Little) Slower and Right than Fast and Wrong

The need for speed is always paramount in trading, but this year the survey revealed that speed is no longer job #1 in FX trading. #1 in FX, is the ongoing struggle with the complexity of integrating and normalizing FX data formats, and #2 is accessing low-latency sources. The votes for #1 and #2 were totaled over half of all respondents.

This finding is consistent with what a head of FX trading recently told me, that “it’s better to be a little slower and right than fast and wrong.” The point he was making wasn’t that it’s OK to be slow, only that getting smarter algorithms deployed given the complexity of pricing and trading FX is just as important, if not more important, than being the fastest.

For more details about the 2011 FX Survey, read the press release, download the survey results, watch the online roundtable, and watch for coverage in the media this week.

Real-Time Big Data and the 11 Principles of Modern Surveillance Systems

Mark Palmer — Mon, 11 Jul 2011 06:00:00 +0000

(This article is also posted in the TABB Group’s TABBForum, and generated a broad discussion in the comments there. To participate, visit the original post here.)

In his TabbFORUM article, Dave Tolladay eloquently argues that real-time surveillance is crucial in today’s high-frequency trading landscape, and I agree. Ten years ago, real-time computing began to influence trading; today, HFT is pervasive and existing surveillance techniques, designed to run at the speed of click trading, should be considered to be legacy architectures in need of a radically new, modern approach.

But managing big data in real-time is a new problem of physics, and those physics come center stage when it comes to surveillance. A recent panel on modern surveillance systems, Miranda Mizen from TABB Group, Justin Amos from Redkite Financial Systems and Richard Tibbetts from StreamBase compared modern surveillance systems to “anti-virus software for trading.”

In the course of their conversation, Justin, Miranda and Richard touched on 11 key points about the “next generation” architectural requirements for surveillance, including:

1. Real-time analytics. Surveillance systems must analyze market data at the speed of automated trading. That means real-time computing and complex event processing (CEP) as one of the critical technology enablers of modern surveillance systems.

2. Historical “big data” analytics. Surveillance systems must put activity in context – suspicious activity can be an aberration or a pattern, based on historical actions. The big data movement has spurred a renaissance in the database industry with firms like Netezza (now IBM), Vertica (now HP), MongoDB (open source) as the flag bearers for the NoSQL (Not Only SQL) database revolution that aims to supplant legacy RDBMS systems to deal with the rising deluge of data described in the Economist piece. In surveillance, “big data analytics” allow on-line access to trade-by-trade history for investigation, trending, and discovery to be mashed up with real-time data to provide a real-time and historical context to behavior.

3. Inclusion of new types of data sources. Justin suggested that the inclusion of real-time news – and even social media data sources – can be a benefit in correlating, for example, suspicious trading on news.

4. Data fusion. Real-time, historical and new data sources must be fused for a complete picture of intra-day trading behavior, and a 360 degree view of market data, activity and sentiment.

5. Multi-asset class support. Since HFT is becoming increasingly cross-asset class, so must modern surveillance systems.

6. Pre-built algorithms. Todayʼs surveillance systems must support standard algorithms like front running, spiking, tailgating, and ramping that can mistakenly or purposefully occur in computer-based HFT systems.

7. Adaptive algorithms. The ability to adapt algorithms on the fly to reduce false positives is important, due to the increasingly dynamic capability of modern HFT systems to adapt their behavior.

8. Algorithm extensibility. The ability to quickly add new algorithms to the system as market dynamics evolve is also a key modern requirement for systems – they must keep pace with the rate of change of modern trading algorithms.

9. Flexible workflow management and case management. Surveillance systems generally donʼt make automated decisions to change trading behavior or stop trading on their own; instead, they tend to guide a series of workflow steps to alert compliance officers, portfolio managers or traders and manage the process of alert analysis, resolution and tracking.

10. Accessibility for the business analyst. With regulation changing so quickly, more firms want their own control over surveillance behavior, so some provide tools that allow compliance personnel to discover and implement surveillance algorithms on their own.

11. Cloud-based or on-premise deployment. Surveillance is not typically the core expertise of trading groups, so the ability to deploy systems either in the cloud or on premise allows for more rapid and global rollout of surveillance tools.

Real-time big data is a change in computing physics on Wall Street. New requirements motivate new solutions, and the new solutions and innovation coming from the surveillance community continues to prove that innovation never sleeps on Wall Street.

StreamBase News at SIFMA

Mark Palmer — Tue, 14 Jun 2011 09:15:04 +0000

The SIFMA show in New York is always a big week for news for firms that do business in the capital markets. This year, we had so much new news that we issued a series of announcements over the last several weeks. In case you missed any of them, here’s a summary of what we’re talking about at SIFMA this year.

1. Thomson Reuters Selects StreamBase CEP. This is a huge endorsement from a leader in the capital markets. Thomson Reuters will utilize StreamBase in their open, global, ultra-high speed and resilient network for the financial industry, Elektron, as an integrated element in their Enterprise Platform for Velocity Analytics, as an embedded CEP engine in Thomson Reuters applications, and in their Thomson Reuters Dealing Aggregator FX Trading Solution. We’re very honored to be selected by Thomson Reuters!

2. BMF&BOVESPA Selects StreamBase CEP. The largest exchange operator in South America selected StreamBase as their enterprise-wide deployment platform for event-based applications – yet another huge endorsement and evidence of the increased use of CEP for enterprise-wide deployments.

3. StreamBase in Asia: Complex Event Processing in Trading with SunGard. Our partner SunGard recently held a online panel discussion about algorithmic trading, market data management, risk management, and other other applications in Hong Kong with Benjamin Becar, algorithmic trading product management for SunGard in Hong Kong, and Richard Tibbetts, StreamBase CTO. You can watch the replay of the session here.

4. New product release: StreamBase 7. Fantastic new usability, and performance improvements in four different dimensions.

5. StreamBase University Program. Our new program to provide StreamBase software and certification resources to universities to help students learn about the growing field of CEP in school. The StreamBase University Program is integrated with our StreamBase Developer Certification Program and StreamBase Component Exchange community for sharing StreamBase integrations, code examples, frameworks, and adapters.

6. Erjang / Erlang integration and the growing StreamBase Component Exchange Community. The latest contribution from Darach Ennis (@darachennis) has been a big hit in the Erlang community and on SlideShare. Watch it here: Complex Erjang / Erlang Processing with @StreamBase.

7. CEP in Risk Management and Surveillance. CEP isn’t just for algo trading anymore! Broad adoption by firms like Thomson Reuters and BMF&BOVESPA, and increasing use in applications such as risk management and surveillance illustrate that CEP is becoming a strategic platform for firms on Wall Street, and for third party developed applications with “CEP Inside.”

8. “Faster than a Flash” – MIT Technology Review on CEP for market data risk management. MIT Technology Review published a story on how proprietary trading firm PhaseCapital used CEP to have ZERO bad trades on the day of the flash crash. While other pundits promote fear of a looming “splash crash” – bigger and worse than the “flash crash,” we’re simply trying to promote firms that are actually doing something about it with innovative technology approaches to make trading safer.

I’ll be at the show in New York this week and addressing questions on Twitter about these developments and others at @mrkwpalmer.

5 Trends Revealed by BM&FBOVESPA’s Selection of CEP

Mark Palmer — Tue, 07 Jun 2011 03:00:00 +0000

Today StreamBase announced that BM&FBOVESPA, the largest exchange operator in Latin America, has selected CEP (for an introduction to complex event processing watch this short video) as their enterprise-wide deployment platform for event-based applications.

Big, public declarations like this one beg the simple question: "what's going on here?" I think BM&FBOVESPA's decision to use CEP points to 5 key trends:

1. Brazil continues their development as an economic power and an innovation power. The Brazilian economy has been surging in past years, and BM&FBOVESPA's use of CEP further reflects the innovation in this increasingly important market. According to analyst AITE group, high frequency trading is up 52% since last November (read their report on the Brazilian market Brazil: The Best of the BRIC?). CEP will be used to build innovative applications that will help the markets operate more efficiently, and more safely. While the U.S. struggles with increased regulation, the conditions in Brazil continue to become more attractive in terms of liquidity and fairness for worldwide investors.

2. Visual CEP enables "Smart Jobs." NPR and Wired Magazine fascinating study this month on areas where the economy is doing well found that "Smart Jobs" are they key to current and future growth and will continue to grow in importance for years to come. The Smart Jobs on Wall Street are jobs where traditional IT AND business analysts work together to innovate by building smarter trading infrastructure. Marcio Castro, Chief Technology Officer of BM&FBOVESPA, said it this way: “The development of capital markets in Latin America is experiencing the fastest growing phase in decades. Innovation is the key for us to develop a liquid, low-latency and fair marketplace for our worldwide investors. [StreamBase was selected to help enable this innovation].” To learn more about how the StreamBase visual programming environment, StreamBase Studio, enables Smart Jobs, watch "Build a CEP App in 46 Seconds."

3. CEP is an increasingly strategic piece of the enterprise IT landscape. BM&FBOVESPA's commitment to CEP isn't for an algo trading app or two – that was the first generation of CEP use in Brazil. Bovespa shows that CEP is becoming an essential platform for applications of all types, from front to back office. CEP is growing up and increasingly critical in the most critical IT environments.

4. IT infrastructure continues to become more event-based. Traditional IT architecture utilize a static computing model; they store a bunch of data, and allow humans to query that data to make decisions. Modern IT architecture is event-based: repetitive decisions are automated – decisions don't have to be left to the end of the day, or end of the month. In trading, tasks that used to be performed at the end of the day, like risk management, are now performed during the day – in real-time. BM&FBovespa's move shows that the static platforms of the past are being augmented by the event-based platforms of the future.

5. StreamBase is taking Brazil by storm. OK, I have to brag just a little! It doesn't get much bigger than an endorsement by a global leader like BM&FBOVESPA. Other CEP companies moved first in Brazil, but when customers look for the leader in CEP, they turn to StreamBase.

Fascinating things are going on in Brazil – now, with the largest exchange in Latin America featuring CEP to power innovation, new innovation lies ahead – stay tuned!

Fast is Good, But Fast and Dumb is Dangerous

Mark Palmer — Mon, 23 May 2011 13:38:35 +0000

Last week I was on a panel at the Bloomberg Enterprise Technology Summit (@BBGlink) with Steve Goldman from CME Group (@CMEGroup), the CTO of NYSE Technologies, Bloomberg TV Markets Reporter Dominic Chu (@The_Domino) and we talked about how speed is important, but that the real innovation going on today is about making applications smarter, not just faster. The conversation was constructive and productive and was re-tweeted all over the internet.

Then, this weekend, Barron's article "Next Danger: 'Splash Crash'" featured an interview with Progress Software's CTO Dr. John Bates. Barron's cites his view that high frequency trading poses an "extreme risk" of creating a "splash crash," his own term, which he describes as a worse than the flash crash: "trading systems could clog up, limited bandwidth could choke orders, exchanges could freeze up — splashing across all of the affected asset classes. Pandemonium."

Bates "frets that, absent more oversight, terrorists wielding smart machines could attack the markets in an attempt to cripple our economy," and has indeed been fretting publicly for a while now in interviews blog posts, such as:

"The trouble with algorithms: wild children or reckless parents?"
"Beware the splash crash"
"Red flags in morning, firms take warning"
"When does a rogue trader become a scoundrel?"

It's unhelpful to focus on the fear; we should focus on solutions. Every automated application – from kiosk-based airport check-in to security in a retail store – needs surveillance. Trading is no different. This is not new. What's new is that trading occurs at speeds never before seen, and when things go wrong, it's easy to blame the bankers of nefarious activity.

So today's article in MIT Technology Review, "Faster than a Flash," was a refreshing break from provocative predictions of doom and gloom. MIT features Boston-based proprietary trading firm PhaseCapital (disclosure: PhaseCapital is a StreamBase customer). CEO Eric Pritchett described to MIT reporter Erica Naone that his algorithmically-driven trading firm made NO erroneous trades on the day of the flash crash.

How did Phase Capital use algorithms and provide a safe trading harbor to their investors on the day of the flash crash? They use the same technology to analyze their market data in real-time that they use to trade in real-time: complex event processing, or CEP (intro here).

Corwin Yu, director of electronic trading for PhaseCapital, explained their innovation this way:

Typically, PhaseCapital processes 30,000 to 40,000 'ticks,' or pieces of market data, per second. During the flash crash, that number jumped to more than 289,000 ticks per second— much of it representing stocks swinging wildly. "A lot of scrubs kicked in and realized that this data didn't make sense," Yu says. For example, some large publicly held companies, such as Accenture, traded for less than $1 during the flash crash. PhaseCapital continued trading, but filtered data that appeared suspicious. Because it didn't act on that data, it was spared erroneous trades.

Yu concluded that speed is essential in the world of trading today, but that the trick is to be "fast and smart." Read about PhaseCapital's innovation in MIT Technology Review.

The solution for a safer market isn't heavy-handed regulation of "wild, reckless algorithms" in order to catch the "scoundrels" of Wall Street. Fast does good on Wall Street; but fast and dumb is dangerous. So we should tune out the fear mongers and tune in to the innovators that can help make Wall Street be fast - and smart.

Thomson Reuters Selects StreamBase CEP for Elektron High-Speed Cloud, Quant Platform, Apps, and FX

Mark Palmer — Wed, 18 May 2011 07:41:32 +0000

Today we're excited to report that Thomson Reuters has selected StreamBase as its complex event processing (CEP) platform for a broad set of initiatives. CEP "flips the database on its head" by providing the ability to monitor, analyze, and act on events happening in real-time, rather than looking in the rear-view-mirror of what happened yesterday, last week, or last month. CEP powers real-time, big data applications on Wall Street including high frequency trading, pre-trade risk management, surveillance, market data management, smart order routing, pricing, settlement, real-time P&L, monitoring – applications from front to back office on Wall Street.

Why did Thomson Reuters partner with StreamBase for CEP? Initially, they announced four reasons:

StreamBase in Elektron Brings CEP to Thomson Reuters' High-Speed Cloud

The StreamBase platform lies at the heart of a new generation of real-time, automated applications on Wall Street. As the use of CEP continues to grow, our customers increasingly ask to deploy some of their applications in the cloud. They want their automated processing to happen as close to the exchange as possible for the lowest possible latency, and they want secure and reliable access to their market data.

Thomson Reuters announced that they will provide StreamBase CEP in Elektron, their open, global, ultra-high speed and resilient network for the financial industry. By adding StreamBase CEP to Elektron, Thomson Reuters is creating the industry’s only platform that includes comprehensive real-time and historical data, high capacity network connectivity, managed hosting services, and CEP for rapid application development and strategy deployment.

The First End-to-End Quant Platform for Back Testing, Algorithmic Development and Deployment

Although StreamBase is all about acting on what's happening now, the knowledge of what happened yesterday is essential for any automated system. History is essential to discover trends and patterns in event streams. History is essential to back test automated strategies. History is essential to gain perspective on real-time actions at execution time. Over 90% of applications that use StreamBase also use some form of historical event data management.

Thomson Reuters will integrate StreamBase with the Enterprise Platform for Velocity Analytics, which integrates all Thomson Reuters content, including tick history, reference data, corporate actions, condition code processing and cancellation and correction messages. It also brings together many disparate third party applications including data management solutions, EMS, OMS, price discovery, back testing, trade surveillance and low latency execution capabilities across the quantitative research and trading workflow.

By integrating Velocity Analytics with StreamBase CEP, Thomson Reuters has created the first end-to-end quant platform for back testing, algorithmic development and deployment.

Thomson Reuters Applications "Powered By" StreamBase

Thomson Reuters will deliver a suite of applications that are “powered by” StreamBase CEP to address low-latency workflow. CEP inside will provide Thomson Reuters clients the benefit of time to market usually associated with turnkey solutions, while at the same time allowing full customizability of the underlying application. One application Thomson Reuters already provides is the real-time surveillance application from Redkite financial which is based on the StreamBase CEP platform.

StreamBase and Thomson Reuters FX Solutions

The FX markets have rapidly emerged as an important market for the capabilities of CEP. Thomson Reuters announced that StreamBase CEP will complement the existing Thomson Reuters Dealing Aggregator FX trading solution, available exclusively to Thomson Reuters Dealing customers, by providing CEP capability in tandem with the low latency algo trading platform.

Why did Thomson Reuters choose StreamBase? Mike Powell, Global Head of Enterprise Real-Time Solutions for Thomson Reuters, said it this way: “The financial market landscape continues to evolve rapidly, meaning that nimbleness and time to market have become essential for market participants. Elektron empowers our customers to innovate and achieve high performance through a seamless infrastructure that brings together global market participants, delivers world-class content and creates a suite of new added-value services. The combination of StreamBase’s market leading CEP engine together with Elektron will provide a platform for us to deliver flexible and sophisticated solutions to our customers faster and more efficiently.”

For StreamBase, we’re excited about the vast resources and capabilities of Thomson Reuters can provide our customers. We’re honored to be selected as Thomson Reuters’ CEP platform of choice – their selection validates our focused approach to CEP, the ever increasing importance of the cloud and real-time big data, the disruption of simplicity, and the continued need for low latency and scalability on Wall Street.

Can Innovative Surveillance Be a New Source of Competitive Edge?

Mark Palmer — Thu, 12 May 2011 11:01:30 +0000

In sports, great defense can be a catalyst for great offense. Can the same idea hold true for surveillance – the defense – in the capital markets? This was a question raised by today’s panel of surveillance experts at today’s StreamBase webinar (watch the replay).

First, TABB Group‘s Miranda Mizen assessed the current surveillance market. She described how surveillance is morphing from a defensive obligation to an offensive differentiator, because the markets are changing so dramatically. Mizan argues that the quantity of data, speed of trading, and complexity of instruments have made traditional surveillance techniques insufficient. And since regulation (e.g., MIFID II) is changing so quickly, she argues that firms must re-think their surveillance strategies and technology choices.

Justin Amos joined Miranda. Justin is a former global markets sales and trading executive at JPMorgan, and the MD of Redkite Financial Systems, a UK-based firm whose system called Redeye has been quickly becoming the standard for modern, real-time surveillance systems. Justin described the requirements of today’s surveillance systems by describing the motivation for Redeye, which he likened to “anti-virus software for trading.”

Together, Justin and Miranda touched on 12 “next generation” surveillance requirements, including:

Real-time analytics. Surveillance systems must analyze market data at the speed of automated trading.
Historical “big data” analytics. On-line access to trade-by-trade history for investigation, trending, and discovery.
Data fusion. Mash-ups of increasingly complex data, including unstructured data.
Inclusion of new data sources. The ability to integrate live data, static data, and social media data.
Multi-asset class support. Modern surveillance must work across asset classes.
Pre-built surveillance algorithms. Standard algorithms like front running, spiking, tailgating, and ramping.
Adaptive algorithm modification. The ability to adapt algorithms on the fly to reduce false positives.
Alert algorithm extensibility. The ability to quickly add new algorithms to the system as market dynamics evolve
Customizable surveillance algorithms. The ability for a firm to tweak an algorithm for their own models.
Flexible workflow management and case management. Control over the process of alert analysis, resolution, and tracking.
Accessibility for business analyst. With regulation changing so quickly, more firms want their own control over surveillance behavior.
Cloud-based or on-premise deployment. The cloud allows more rapid rollout, and may ease global scalability challenges

The panel argued that existing surveillance systems can’t handle these requirements. Taken together, the challenge makes what was once traditional “defense,” or a compliance function, very new and challenging – a source of differentiation.

Miranda went deeper, with examples of how brokers and the buy side are using modernized surveillance systems: agency brokers are beginning to use surveillance tools to monitor their trading algorithms and optimize their client service and order routing for best execution; the buy side are using surveillance to gain greater control to know that their trading is safe, its orders get to the right places, and that dark pools are working correctly in relation to the rest of the market; sell side institutions are using surveillance tools to institute process and control across the organization, globally, and integrated with automated systems.

Richard Tibbetts, CTO of StreamBase, gave a brief introduction of StreamBase CEP, and why complex event processing is increasingly being used to power surveillance applications; indeed, CEP provides the engine to address many of the 12 modern surveillance requirements set out by the panel.

Simon Townsend, CIO of Redkite, gave a “whistle stop tour” around Redkites’ application. He hit on most of the 12 requirements above, and even showed didn’t just claim that their algorithms are extensible and dynamic (requirements 6, 8 and 9) – he showed it in StreamBase Studio, and showed how business analysts can extend the Redeye surveillance algorithms.

One of the most exciting times is when a market is transformed by new ideas. In the capital markets, perhaps that innovation is upon us in surveillance, where tools that meet modern trading requirements can turn a defensive obligation into a tool for innovation.

Twitter and Trading: A Ripple in the Big Data Wave

Mark Palmer — Mon, 09 May 2011 15:28:46 +0000

In 2009, StreamBase began talking publicly about using CEP with Twitter. The topic sparked a flurry of mainstream media coverage about Complex Event Processing (CEP) on Wall Street on outlets such as CNBC, the Wall Street Journal, and the Financial Times. Even Popular Mechanics named CEP as one of 10 Tech Concepts to watch, along with Fracking and Flywheel Energy Storage! (watch this short video for an introduction to CEP)

Last week, USA Today published an article about CEP and trading on the front page of its Money section: Wall Street traders mine tweets to gain competitive edge. The report covered developments such as:

The opening of Derwent Capital, a firm that specializes in news-driven trading strategies, including Twitter
A report from Adam Honore from AITE group whose two recent reports (on trading strategies and on unstructured data) revealed that interest in unstructured data has jumped from 2% in 2008 to 35% in 2011
Rich Brown, who talked about how the Thomson Reuters news sentiment engine is more effective than the "quirky and abbreviated language of Twitter"
Johan Bollen, a professor of informatics at Indiana University, whose research claims an 87% accuracy rate in using Twitter mood measurements to predict Dow stock prices three to four days later
MarketPsych, a firm that used to run sentiment-based strategies, switched from being a hedge fund to a software provider

The article shows that Twitter continues to be a controversial topic: some tweets are clearly important – for example, the CME Group tweets news about the commodities market and has over 780,000 followers. On the other hand, computer-measured sentiment is just that – one more barometer of what the masses are thinking. Good investment decisions are powered by experience and knowledge, and sentiment is one input of many.

The noise over use of Twitter on Wall Street drowns out the real story, which is why I was quoted in the article that Twitter is "a ripple, not a wave" on Wall Street (and Main Street as well).

The big wave is big data, and Twitter generates just a few buckets of sand in world of big data today.

In "New Rules for Big Data," the Economist reported that Google processes a petabyte of data an hour. Here's a way to put Twitter in context*: think of a bucket of sand. If every grain of sand in the bucket was 1 byte of data, then:

The entire work of Shakespeare fills just one bucket of sand (about 5MB)
A fast financial market data feed (OPRA) fills a beach of sand in 24 hours (about 5TB)
Google processes all the sand in the world every week (about 100PB)
We generate 60% more sand every year

In these terms, all of Twitter generates only a sand castle of quality data a day. While some of that data is very valuable, such as tweets from CME Group (@CMEGroup), most of it simply tells us about mass sentiment.

On the other hand, the debate about Twitter does reveal the opportunity and challenge of big data. The Economist used a quote from Oscar Wilde in 1894 to put that challenge in context – Wilde said: “It is a very sad thing that nowadays there is so little useless information."

So the big ruckus about Twitter on Wall Street reveals the real big story – today's big data challenge.

* Thanks to Rob Doherty at Cloudswitch for sharing Chad Sakac's (EMC) buckets of sand concept, the Economist New Rules for Big Data for sizing data from Google, and @RJMetrics on big data (TED Philly talk) for conceptual description of big data.

StreamBase Loves Erlang

Mark Palmer — Thu, 21 Apr 2011 12:51:55 +0000

The philosophy of Erlang and StreamBase align quite well. Quoting Mike Williams, one of the three inventors of Erlang:

Find the right methods—Design by Prototyping
It is not good enough to have ideas, you must also be able to implement them and know they work
Make mistakes on a small scale, not in a production project

These philosophies of rapid development also apply to the way we've designed StreamBase; additionally, our customers also demand that much less technical users participate in the rapid implementation of ideas. StreamBase has a visual development approach that allows less technical, more "business" users – with mathematics, analytics, and business domain expertise – use the same tools as professional software developers.

What if we could add Erlang developers to that party? Event-driven, visual development, coupled with the power and rapid development of Erlang?

Well you can. Darach Ennis (@darachennis) shows us how in his recent presentation at Erlang UG in London.

StreamBase has a graphical, compiler-oriented programming language, targeted at serious developers, that can also be used by the less technical. Erlang programmers, and the Erjang folks with whom we share a virtual machine, do their thing in Erlang, and that code becomes a graphical "operator" on the StreamBase Studio pallet. In Darach's example he's used Erlang to implement a MACD analytic, and integrate that into a StreamBase-based trading system so it operates in the event-driven CEP application.

The combination of StreamBase and Erlang combines the best of both worlds: StreamBase for rapid, graphical development to develop low latency applications that process 100's of thousands of events a second. Combining "how," "why," and "under what conditions," real time streams are matched and processed. StreamBase allows business analysts, actuaries, quantitative contributors and developers to share a common language. An event-driven integration whiteboard, if you will, that compiles to efficient optimized native binary code. Add Erlang for development of analytics like MACD and you have a powerful combination.

Erlang support will be contributed soon to the StreamBase Component Exchange (SBX), our public "app exchange" for CEP integrations, applications, frameworks, and sample code. You'll already find support for Java, C++, R, Matlab, python, .NET. If you want to plug in some cool analytics written in C++, just do it. You want to exploit Basho's Riak? Sure, plug it in.

As a result, StreamBase allows developers control over how they exploit the facilities at their disposal. Use our graphical DSL for flow oriented algorithms and real time low latency high frequency data management. It's high performance from a human productivity and systemic perspective.

Learn more by watching Darach's presentation here, and follow him on Twitter at @darachennis.