So many events, so little time: can technology help decide which ones matter?

World War I was a complex event.

That’s the example I once heard Dr. David Luckham, the father of complex event processing, provide as a working definition of CEP. That is, there were a lot of inputs and interrelated events that resulted in a lot of changes to a lot of people, organizations, and governments.

Fortunately, most businesses don’t need to attempt to analyze the impact of global conflicts on their market prospects. But every business can benefit from awareness of what’s going on inside and around — to be able to sense and respond and things happen. But which events really matter?

A typical business has hundreds of thousands, and even millions of events each day — from weather to lost shipments to equipment breakdowns to employee absenteeism to unusually large transactions. But only five or six events — out of a million — may have a material impact on the business going forward. Which are relevant? Which will dent sales?

I recently had the opportunity to join David Olson, director of CEP product marketing for Progress Software’s Apama unit, along with Brenda Michelson, principal with Elemental Links, in a rousing roundtable discussion over at ebizQ on the impacts of event processing in what has been a very eventful year for businesses. (Listen to the full Event Processing Roundtable here. Read the full transcript here.)

I’ve always felt that “complex event processing” is a term that scares people away. Perhaps there needs to be a softer way to describe what this thing is — perhaps just plain old “event processing” will do? Or perhaps “sense and respond, times 100,000″?

The good news is you don’t need to be a state-of-the-art, completely online organization to move to event processing. David Olson said from a technical standpoint, the fundamental system requirement for event processing is having a server to capture and correlate events, apply appropriate rules to an event stream, and send the information to a dashboard or portal.

But the success of an event processing implementation is a management challenge, he says.  “One of the more complicated endeavors that people have to figure out is what events make sense to them, and what types of patterns do they need to look for?”

Brenda pointed out that millions of events occur across enterprises every day — from stock prices bouncing to soda cans shifting on a pallet in a warehouse. The challenge is being able to capture and process the events that have the greatest impact on the business. “Event processing is what discerns is if that thing that happened is notable,” she explained:

“Is it important to me? Is it important to my business? Do I need to act on it?  And is that notability by itself, or perhaps its notable because of a couple of other events that are going on? Is it just my business that’s bad, or is it the entire credit market that’s tanked? And then once you discern that notability.. you decide what am I going to do with that event next?  Am I going to forward it along into an event channel? am I going to trigger some kind of downstream action?”

David elaborated on that theme and pointed out that “its not sufficient just to be able to capture the events and perhaps store them for future use, since the volume and the velocity can be quite great in certain circumstances.”

Now, the $800 billion question: could complex event processing have steered financial institutions away from the subprime meltdown of 2007-2008? If these companies had more advanced capabilities to process and analyze key events, they might have been able to head off many of the slips and slides that led to the financial crisis of 2008, David and Brenda agreed. Still, there are many lessons being gleaned from the whole experience that will benefit all industries for years to come.

“I think [Complex Event Processing] could have been used in some situations that got us into this financial crisis,” says Olson.”Perhaps we should have employed CEP a few more years ago in order to help stem the tide of what has happened. But there’s certainly a lot of learning that’s been going on as people go back and research what has happened, to figure out what kind of rules they should put in place in the future to make it happen again.”

There is great potential being demonstrated in financial services in terms of unearthing fraud, David points out. “One of the exciting areas in CEP, especially in capital markets is the whole notion of surveillance, where there are a fair amount of rules that look for abnormal patterns of activity,” he says.

“For example, one of our customers uses it to monitor actions that could indicate insider trading, or joint trading opportunities where two traders trade large volumes of a symbol, but not large enough that it tips anybody’s radar to abnormal activity. But when these joint trading activities occur either ahead or behind a news feed, that could indicate to us there’s unusual activity going on here, and we should do something with this particular symbol that’s being traded. In the past, trying to catch those types of activities could take months of auditing in order to figure out that something bad happened. And in the capital markets space, minutes of something bad happening could be billions of dollars.”

It’s uncertain whether CEP awareness could have headed off the recent financial storm. But developing some complex event processing capabilities is smart because it can help organizations sense and respond to both potential threats or opportunities — from surfacing unusual or suspicious events to seeing new markets emerging.