Real-Time Event Recognition and Forecasting under Uncertainty
Development of real-time event recognition and forecasting technology operating on Big Data. The proposed technology will be:
- supported by machine learning techniques that allow for automated construction of event definitions;
- resilient to the inherent uncertainty of Big Data.
The aim is to develop event recognition and forecasting technology operating on Big Data. Attention will be restricted to techniques with formal (probabilistic) semantics such as the Event Calculus. In the context of SPEEDD, the input of the module developed will be credit card transactions and events detected by fixed and mobile sensors on motorways. To achieve the aim of this work package the following tasks are defined.
- Machine Learning Algorithms for Constructing Event Definitions
- Real-Time Event Recognition and Revision under Uncertainty
- Event Forecasting under Uncertainty
Real-time event-based decision-making under uncertainty
The aim is to provide a body of proactive event-driven decision-making tools for use in fully autonomous or semi-autonomous Big Data applications. These tools will be designed to maximally exploit the forecast techniques described above. In particular, the tools developed will consider both the nature and degree of uncertainty in model forecasts when making decisions, and will provide a range of optimization options (e.g. probabilistic constraint satisfaction or worst-case guarantees) to allow users to select the most appropriate decision method for their particular application. These methods will then be applied to the SPEEDD use cases using a novel event-driven decision-making strategy to enable varying degrees of automation. Three types of decision-making methods will be explored: Worst case real-time decision-making (based on robust optimization), stochastic real-time decision-making (based on stochastic programming) and randomized, scenario based decision-making (based on randomized optimization). All three methods will be developed in an event-driven context, where decision-making/optimization actions are carried out not at specific times, but triggered by the occurrence of predicted events.
The key features of the decision-making methods developed as part of this work package will be that they enable:
- real-time decisions to be selected, i.e. the optimization methods developed will be designed to provide sufficiently good decisions or operator recommendations at a rate fast enough to keep pace with the Big Data application;
- uncertainty in forecast models operating on Big Data to be considered directly as part of the decision-making process, and in particular consideration of the value of future information and the possibility of recourse decisions;
- the option of making proactive decisions in an event-driven mode, i.e. the decision-making and decision-support tools can be triggered selectively in
- response to application-specific user-defined events; and
- a wide range of decision support options for human operators, ranging from the simple evaluation of operators’ planned actions to fully automated real-time decision-making without operator intervention.
Real-Time Visual Analytics for Proactive Decision Support
The primary objective is to explore the impact of real-time proactive decision computation on human decision-making in Big Data applications. The SPEEDD use cases have practitioners who have developed strategies based on their experiences, primed by the available features under different situations. This process, characterized as recognition-primed decision-making, involves responding to specific collections of features. An automated system could potentially obscure or remove these collections of features (simply because it has processed some of the features into its solution). One challenge, therefore, is to allow the human experts to access in real-time the information that they would normally use to apply their strategies.
Finally, this work package explores alternative designs for Visual Analytics which will help human decision-makers with their initial situation identification and expert response, as well as enabling learning by both the human and automated system. The interaction of human decision-makers with the SPEEDD prototype will lead to the following objectives for this work package:
- understanding the decisions involved in each SPEEDD use case and the information sources on which these decisions are based, in order to understand how decision-makers assimilate information from multiple sources in a socio-technical system;
- describing human decision-making as a rational, objective response to goals and changing context, thus understanding human decision-making in the context of SPEEDD and the manner in which this process can be learned and refined by both human and automated system through their interactions;
- developing novel, real-time visualization techniques for Big Data that support human decision-makers in their situation awareness, sense-making, decision-making and appreciation of recommendations made by the SPEEDD prototype.
- Develop a highly scalable event processing infrastructure supporting real-time event delivery and communication minimization.
- Integrate the SPEEDD components into a prototype for proactive event-based decision support.
To achieve the objectives of this work package, the tasks below are defined:
- Computation Scalable Algorithms. The algorithms for event recognition and forecasting under uncertainty will be scaled by distributing the computational load among multiple nodes. In order to achieve this goal, efficient strategies for scaling the underlying mechanisms employed in WP3, such as Probabilistic Logic Programming and Markov Logic Networks, will be developed. Various partitioning methods (e.g. vertical, horizontal) and pipelining techniques will be employed. Furthermore, the algorithms will be adaptive to changing query requirements and event distributions. Particular emphasis will be placed on distributing inference tasks over probabilistic graphical models. The developed algorithms will exploit the continuous nature of the recognition and forecasting tasks by incrementally modifying the inference as Big Data arrive. Inference techniques, such as Belief Propagation and Markov Chain Monte-Carlo sampling will be customized to, and extended for event recognition and forecasting under uncertainty.
- Communication Scalable Algorithms. The increasing number of distributed Big Data-generating sources requires that inherently-limited network resources be employed efficiently. In this task, we will develop communication-efficient algorithms for event recognition and forecasting under uncertainty, that support the full range of functionality required by Big Data applications