Login | Register

• Print this |
• Email this
• Digg
• De.licio.us
• Slashdot
• Technorati

Current with Events

CEP can be used to track shipments, other event-driven systems

Where could complex-event processing work better than standard database analysis? During the May JavaOne conference in San Francisco, Thomas Bernhardt offered one answer: tracking shipments of valuable items.

Bernhardt is head of a project to develop complex-event processing software called Esper. He said the software could easily keep track of warehouses of items bearing radio frequency identification tags.

Say, you are in the military overseeing warehouses full of ammunition. How do you keep track of all the items, especially as they move from building to building? How could you set up an alert to tell you if a box “fell off the truck” or was “lost in transit,” in the parlance of yore? RFID readers would detect a box of ammunition leaving or entering a building. But what back-end software could interpret and coordinate these messages? In a busy environment, their sheer numbers would quickly overwhelm a traditional database trying to log them.

Nor would standard database analytic software necessarily be able to coordinate all these messages into something meaningful. If a box of ammunition were moved from one building to another, how could the manager tell it had made the journey successfully, short of manually following the progress of each RFID number? Online analytical processing may be good for parsing multidimensional datasets, but it doesn’t do well with making sense of data elements in terms of when each element was logged in.

“It is hard to deal with temporal streams of data,”Bernhardt said.

Bernhardt characterized systems such as an RFID-equipped warehouse as having —or needing — an event-driven architecture. These systems are built around detection of and reaction to events. Timing and aggregation are essential — and difficult. “It [is] very hard to construct an event-driven application manually,” he said.

In the industry, the basic processing for this sort of environment is called event stream processing (ESP). The idea is to pull the right events out of a constant stream of data points and aggregate them in a meaningful way. For instance, someone might want to calculate the average price of a stock over the past 30 ticks of trading, which involves constantly recalculating the average of numbers that are being constantly updated.

But even ESP isn't strong enough to handle more-complex jobs, such as watching RFID tags as they move about. Here is where CEP comes in. CEP is the advanced analytical detection of patterns in a series of events. A CEP function specifically looks for complex behaviors, such as two successive gains in stock price of one company followed immediately by a drop in the price of another company's stock.

Bernhardt’s open-source Esper software can do both ESP and CEP. Under benchmarks, Esper has been able to process and make sense of about 110,000 events per second. The Esper container works in either a Java Enterprise Edition application server or the Microsoft .Net framework.

What Esper does is run all the messages it receives through a set of queries defined by a user. Instead of committing these messages to a database and then analyzing them, ESP analyzes the data as it comes across the wire. As soon as one of the query engines detects a set of messages that meets some predefined set of conditions, Esper alerts the user or another program.

Users describe the events they are looking for in Esper’s Event Query Language, an SQL-like language with a number of unique characteristics for identifying data on the fly. EQL can do many things SQL can’t. For instance, EQL can define a range of parameters as a unique set. You can define a warehouse based on the geographic coordinates within that warehouse, so you know any items identified at one of those coordinates are in the warehouse. Although SQL lets you specify a certain point in time, EQL lets you specify a period, or window, of time. So you can easily compile a list of items that have entered a warehouse within the last 30 seconds, 30 minutes or some other window. And, like SQL, it can bring in additional sets of data for comparison through the join function. You could join data from two database tables to watch how materials are flowing from one location to another.

Using this function, Bernhardt said, you could write a query that could watch when a large number of items moves from one warehouse to the next and send you an alert when one or more of the items that left one location fail to reach their destination. And so CEP solves a problem with an ease other tools can hardly match.

To see Bernhardt’s JavaOne presentation, go to GCN Quickfind 806. To download Esper, go to esper.codehaus.org.

— Joab Jackson