Real-time business intelligence (RTBI) is an approach to business intelligence that reduces the delay between events and the information used in decisions. This is different from conventional business intelligence (BI), where reporting is usually from historical data. By making data available closer to the time that the activity occurs, RTBI can support monitoring and operational response. RTBI systems use refreshed analytical stores, event streams, event logs, complex event processing, business activity monitoring, or related architectures. Their output may appear through dashboards, alerts, process monitoring tools, analytical views, or responses. “Real time" depends on the context. Some systems process events within seconds, while others are better described as near-real-time. RTBI therefore involves trade-offs between speed, data freshness, cost, complexity, and the reliability of the information available at the time of analysis.
Latency and data freshness Traditional BI has often focused on improving the reporting of historical data. RTBI, however, places greater emphasis on reducing delays from data capture through to the response. All real-time business intelligence systems have some latency, but their aim is to reduce the delay between the use of information from when the event happened. Analyst Richard Hackathorn describes three types of latency:
Data latency: the time taken to collect and store the data; Analysis latency: the time taken to analyse the data and turn it into actionable information; Action latency: the time taken to react to the information and take action. Recent work on real-time analytics describes both speed and freshness as important features of real-time analytical systems. Data freshness is not the same as latency. A system could quickly process a query, but with data that is already out of date. Conversely, operational data collected soon after an event, could still require further processing before it can support a decision. What "real time" means depends on the context. Some systems process events within seconds, while others are described as near-real-time, because data is made available as soon as possible to support a timely decision. Hackathorn notes "right-time" as an alternative to "real-time", but argues the more important issue is whether the timing of information created will create business value.
Architectures RTBI architectures vary according to how data is collected, processed, stored, and presented. Conventional business intelligence systems include operational sources, data integration processes, analytical storage, query processing, and tools for presenting it. In RTBI, those functions remain important, but the architecture is designed to reduce the delay between a business event and the use of information from it.
Event processing and streams In event-driven architectures, changes in state are represented as events and processed as they happen. Event stream processing and complex event processing systems process information continuously from multiple sources. These systems may filter, aggregate, correlate, or detect patterns in event streams before all data has been loaded into a conventional database. For RTBI, event processing supports dashboards and alerts, when current business conditions meet defined rules. Streaming systems also need to handle data that does not arrive in sequence. Records may also arrive late and some cases incomplete. Techniques such as continuous queries and time windows are used to calculate results over set periods of an otherwise continuous stream.
Event logs and event sourcing Some RTBI architectures use event logs to pass changes to other systems. An event log is an append-only record that can be consumed by other systems to update dashboards, analytical stores, alerting systems, search indexes, and materialised views. Related approaches include change data capture and event sourcing. Change data capture is one way of producing streams from existing databases. It records changes made to database tables and makes those changes available to other systems without reloading the full data set. In RTBI, this can help keep analytical stores or event-processing systems closer to current operational data. Event sourcing is an architectural pattern where changes to an application's state are stored as a sequence of events, rather than just a current state record. The current state of an entity can be reconstructed by replaying all the stored events in order. In RTBI, an event-sourced system can provide an event history that supports reporting and analysis through derived read models. These views can then be used by dashboards or other analytical tools. It can support auditability, reconstruction of earlier states, and the creation of multiple views from the same event history. It is often used with Command Query Responsibility Segregation (CQRS), because querying directly over an event store can be inefficient and separate read models built for reporting or analysis. Event sourcing may introduce design issues, including event schema changes, replay, storage growth, and eventual consistency between the event store and derived views.
Near-real-time analytical stores Another approach is to refresh a data warehouse or analytical store more often. This keeps the historical comparison and reporting features associated with conventional BI, while reducing the delay between activity and the availability of the data. Some cloud data warehouses and hybrid transactional or analytical processing systems are described as supporting near-real-time analytics. In practice, this depends on how current the available data is, the complexity of the queries being run, and how the source and analytical systems are designed. Event-driven systems and analytical stores can be used together. For example, selected events may be sent directly to dashboards or alerts, while the same operational data is also kept in an analytical store for later review.
… excerpt ends here. Continue reading the full article.
