Event-based systems (EBS) are increasingly used as underlying technology in many mission critical areas and large-scale environments, such as environmental monitoring and location-based services. Moreover, novel event-based applications are typically highly distributed and data intensive with stringent requirements for performance and scalability. Common approaches to address these requirements are benchmarking and performance modeling. However, there was a lack of general performance modeling methodologies for EBS as well as test harnesses and benchmarks using representative workloads for EBS. Therefore, this thesis focused on approaches to benchmark EBS as well as the development of a performance modeling methodology. In this context, novel extensions for queueing Petri nets (QPNs) were proposed. The motivation was to support the development and maintenance of EBS that meet certain Quality-of-Service (QoS) requirements.
To address the lack of representative workloads we developed the first industry standard benchmark for EBS jointly with the Standard Performance Evaluation Corporation (SPEC) in whose development and specification the author was involved as a chief benchmark architect and lead developer. Our efforts resulted in the SPECjms2007 standard benchmark. Its main contributions were twofold: based on the feedback of industrial partners, we specified a comprehensive standardized workload with different scaling options and implemented the benchmark using a newly developed complex and flexible framework. Using the SPECjms2007 benchmark we introduced a methodology for performance evaluation of message-oriented middleware platforms and showed how the workload can be tailored to evaluate selected performance aspects. The standardized workload can be applied to other EBS. E.g., we developed an innovative research benchmark for publish/subscribe-based communication named jms2009-PS based on the SPECjms2007 workload. The proposed benchmarks are now the de facto standard benchmarks for evaluating messaging platforms and have already been used successfully by several industrial and research organizations as a basis for further research on performance analysis of EBS.
To describe workload properties and routing behavior we introduced a novel formal definition of EBS and their performance aspects. Furthermore, we proposed an innovative approach to characterize the workload and to model the performance aspects of EBS. We used operational analysis techniques to describe the system traffic and derived an approximation for the mean event delivery latency. We showed how more detailed performance models based on QPNs could be built and used to provide more accurate performance prediction. It is the first general performance modeling methodology for EBS and can be used for an in-depth performance analysis as well as to identify potential bottlenecks. A further contribution is a novel terminology for performance modeling patterns targeting common aspects of event-based applications using QPNs.
To improve the modeling power of QPNs, we defined several extensions of the standard QPNs. They allow us to build models in a more flexible and general way and address several limitations of QPNs. By introducing an additional level of abstraction, it is possible to distinguish between logical and physical layers in models. This enables to flexibly map logical to physical resources and thus makes it easy to customize the model to a specific deployment. Furthermore, we addressed two limiting aspects of standard QPNs: constant cardinalities and lack of transition priorities.
Finally, we validated our methodology to model EBS in two case studies and predicted system behavior and performance under load successfully. As part of the first case study we extended SIENA, a well-known distributed EBS, with a runtime measurement framework and predicted the runtime behavior including delivery latency for a basic workload. In the second case study, we developed a comprehensive model of the complete SPECjms2007 workload. To model the workload we applied our performance modeling patterns as well as our QPN extensions. We considered a number of different scenarios with varying workload intensity (up to 4,500 transaction / 30,000 messages per second) and compared the model predictions against measurements. The results demonstrated the effectiveness and practicality of the proposed modeling and prediction methodology in the context of a real-world scenario.
This thesis opens up new avenues of frontier research in the area of event-based systems. Our performance modeling methodology can be used to build self-adaptive EBS using automatic model extraction techniques. Such systems could dynamically adjust their configuration to ensure that QoS requirements are continuously met.