A Comparison of Soundness Results Obtained by

A Comparison of Soundness Results Obtained by Different Approaches Volker Gruhn and Ralf Laue Chair of Applied Telematics / e-Business⋆ Computer Science Faculty, University of Leipzig, Germany {gruhn,laue}@ebus.informatik.uni-leipzig.de

Abstract. Business processes are often modelled using a language for which no semantics is standardized in a formal way. Examples for such languages are BPMN or Event-Driven Process Chains (EPC). The common way for reasoning about the soundness of such models is to define a formal semantics first by translating the model into a well-founded formalism (for example Petri-nets). Afterwards, formal reasoning methods can be applied on the obtained formal model. In the past years, several such semantics that give a formal meaning to BPMN or EPC models have been published. In this paper, we used a repository of almost 1,000 real-world EPC models and computed their soundness using three different tools. Those tools build on different semantics definitions: Kindler’s fixed-point semantics, Mendling’s state/context semantics and the YAWL semantics. While the soundness results for the majority of models were the same for all three tools, we identified a few interesting cases where the results differ. The results of our comparative study can lead to a better understanding of the differences between the semantics.

1

Introduction

Business processes are often modelled using a language for which no semantics is standardized in a formal way, for example BPMN or Event-Driven Process Chains (EPC). The first challenge on the way to a tool that verifies the correctness of such a model is to define a formal semantics of the model. The usual way is to map the model to a semantically well-founded formalism. In particular, Petri nets, Pi calculus or formal automata have been used for this purpose. Different semantics definitions for modelling languages like BPMN or EPC have been suggested. An overview is given in Sect. 2. The aim of this paper is to analyse a large number of real-world models using different semantics in order to gain insight into the differences between the approaches. Sect. 3 describes the origin of the models used in our experiment. We selected three tools that build on different semantics defintions and used them to validate the soundness ⋆

The Chair of Applied Telematics / e-Business is endowed by Deutsche Telekom AG

2

property for the models from our repository. An overview of the tools used is given in Sect. 4. The results are presented and discussed in Sect. 5. Finally, Sect. 6 summarizes and discusses the findings from our comparative study.

2

Semantics Definitions from the Literature

For the basic modelling constructs used in languages like BPMN, the mapping to a well-founded formalism like Petri-nets is rather straightforward. However, for some modelling constructs such a mapping is difficult. In particular, it has been shown in [1] that it is impossible to define a semantics for the OR-join in a satisfactory manner. The OR-join is used to model the “Synchronizing Merge” workflow pattern. This pattern is described in [2] as “a point in the workflow process where multiple paths converge into one single thread. If more than one path is taken, synchronization of the active threads needs to take place. If only one path is taken, the alternative branches should reconverge without synchronization”. Informally, an OR-join has to wait until all previously started control flows that can arrive on its incoming arcs have been completed. This means that before processing an OR-join, it has to be decided whether the flow of control can reach one more of its incoming arcs. In [1], some counterexamples have been given that show the impossibility to define a formal semantics of an OR-join which corresponds to the concept of a synchronizing merge. Problems can occur in models with multiple OR-joins in a loop where the possibility to process one OR-join depends on the possibility to process another one. From a theoretical point of view, the OR-join problem has been solved by the work of Kindler [3]. In [3], the concept of a fixed-point semantics has been used for developing an algorithm that decides whether a satisfying semantics can be defined for a model. Models that have such a semantics are called clean and others (like the counterexamples from [1]) unclean. An algorithm has been presented that computes efficiently the fixed-point semantics (if it exists) or shows that there is no fixed-point semantics [4, 5]. Nevertheless, the calculation of a fixpoint-semantics has also been criticized as too difficult and time-consuming by some authors. For this reason, several other semantics definitions have been proposed that claim to have certain advantages over the computation of a fixed-point semantics[6–10]. Some work has been done on comparing different semantical approaches. In [11], some of the earlier approaches to define a semantics for event-driven process chains are discussed in comparision. All those approaches impose some additional well-formedness requirements on the models. Wehler [12] also compared two such semantics - the one defined by van der Aalst [13] for models without loops and the semantics proposed by Langner, Schneider and Wehler [14] that is based on Boolean Petri nets and restricts the modeller to use loops in a certain wellstructured way only. Mendling [15, 9] discusses five semantics definitions and identifies disadvantages of those semantics. Some semantics can be used for a certain class of

3

“structured” models only. For others, a “well-strucutred refinement” (inserting a well-structuring construct into a model) can change the semantics of OR-joins in the original model. Some differences among the definitions for OR-join semantics have been discussed in [7]. The authors of [7] propose a new semantics which does not impose any restrictions on the structure of the model and requires a lower computational complexity than other known approaches. All papers mentioned above used mainly a small number of constructed examples for comparing the different semantics definitions. To our best knowledge, there has been no attempt so far to use a large number of real-world models for identifying differences among different semantics and the validation tools based on them. We used the “soundness validation” analysis that is included in three different tools for analysing 984 models. By comparing the results from those tools, some advantages and disadvantages of those tools and its underlying approaches can be found.

3

Models used for our Comparative Study

For the purpose of this comparative study, we collected a repository of 984 business process models which have been modelled as Event-Driven Process Chains (EPC). This modelling language has been introduced in 1992 as an informal business process modelling language. EPCs consist of functions (activities which need to be executed, depicted as rounded boxes), events (pre- and postconditions before / after a function is executed, depicted as hexagons) and connectors (which can split or join the flow of control between the elements). Arcs between these elements represent the control flow. The trigger for starting the execution of an EPC is that certain start events (i.e. events without incoming arc) happen.

AND-join

XOR-join

OR-join

AND-split

XOR-split

BPMN

YAWL EPC Table 1. Connectors in different modelling languages

OR-split

4

Table 1 shows the different connectors that can be found in EPC models and the corresponding symbols for the connectors in YAWL and BPMN1 in order to assist those readers who prefer another language. The models in our repository have been collected from 130 sources. These sources can be categorized as follows: – – – – – – – – – –

531 models from the SAP R/3 reference model 112 models from 31 bachelor and diploma thesises 25 models from 7 PhD thesises 13 models from 2 technical manuals 82 models from 48 published scientific papers 12 models from 6 university lecture notes 4 models from sample solutions to university examination questions 88 models from 11 real-world projects 88 models from 7 textbooks 29 models from 14 other sources

Among the models in our repository, there is a great variation in size of the models, purpose of modelling, business domain and experience of the modellers. For this reason, we think that the models represent a reasonable good sample of real-world models.

4

Tools used for our Comparative Study

For analysing the models, we used three open source tools that are freely available: EPCTools, the ProM plugin for EPC soundness analysis and the YAWL Editor. All of them offer a feature “analyse soundness”. What makes the comparision of the analysis results interesting is the fact that the tools use different definitions of semantics. All the tools run as a Java program. We executed the tools on an Intel Core2 Duo CPU running at a speed of 3 GHz. By starting the Java virtual machine with the option -Xmx1536m, we allowed a heap size of 1.5 GB to be used. EPCTools[4, 5] calculates a fixed-point semantics for a model. If such a fixedpoint semantics exists, a temporal model-checker is used by EPCTools for deciding about the soundness property. For the majority of models from our repository, an analysis result was given in a few seconds. There was only one model for which the analysis took more than 10 minutes. This model was validated in 63 minutes. For 28 models, EPCTools had to stop the computation because of an Out of Memory error. We applied soundness-preserving reduction [16, 9] on these models. For 6 of the reduced models EPCTools still ran into a Out of Memory error, the others have been analysed. The ProM plugin for EPC soundness analysis [17] uses the semantics defined by Mendling [9] for constructing a transition system for the model. Mendling’s semantics combines the concept of a state (represented by tokens attached to 1

In BPMN, the connectors are called gateways.

5

arcs of the model) with the concept of a context (represented by additional binary tokens attached to the arcs). The context can be either “wait” (i.e. the arc still has to wait for more tokens to arrive) or “dead” (no tokens are expected to arrive). Both kinds of tokens are propagated in a four-staged process. For 31 models, ProM failed to deliver a result because of an Out of Memory error. For 5 models, the computation took more than 10 minutes, the longest computation time was 26 minutes. The third tool, YAWL Editor [18, 19], originally has been constructed for analysing YAWL models. Our EPC models had to be translated into YAWL in order to analyse them. While the mapping of EPC modelling elements to YAWL is straightforward, there is an important difference between EPC and YAWL: YAWL does not support process models with more than one start event. In order to avoid the problems that arise from the different instantiation semantics for EPC and YAWL models[20], we ran the YAWL soundness validation only for those 737 models for which the EPC model has exactly one start event after applying soundness-preserving reduction rules as described in [16, 9]. YAWL Editor has a built-in restriction that stops the execution of the analysis if the calculated state-space exceeds 10,000 states. This is necessary, because the YAWL semantics allows an infinite state space[18]. This restriction was enforced for 22 models, meaning that no analysis result was available for them. The computation was very fast for the majority of the models. 628 have been analysed in less than 1 second, the longest computation took 23 sec. However, it took much longer to detect that a model cannot not be analysed because the state space exceeds 10,000 states. For only one model this fact could be realized in less than 5 minutes. The longest computation took more than 5 hours before the program terminated with the information that the state space exceeds 10,000 states.

5

Soundness Analysis Results

Soundness is an important and widely used correctness criterion for business process models. It has been originally introduced by van der Aalst for workflow nets[21, 22] and later adapted to the EPC notation[13, 9]. The formal definition of soundness can be found in the mentioned literature. Informally, a business process model is sound, if: 1. In every state that is reachable from a start state, there must be the possibility to reach a final state (option to complete). 2. If a state has no subsequent state, then it must be a final state (proper completion). 3. There is no element of the model that is never processed in any execution of the model (no needless elements). We used the tools described in Sect. 4 for testing the soundness property of the models from our repository. 712 models could be analysed by all three tools. The results of their analysis is shown in Tab. 2.

6 ProM plugin for EPC soundness EPCTools YAWL Editor Models found analysis unsound unsound unsound 32 unsound unsound sound 0 unsound sound unsound 1 unsound sound sound 12 sound unsound unsound (4) sound unsound sound 0 sound sound unsound 0 sound sound sound 663 Table 2. Result of the soundness analysis of those models which could be analyed by all three tools

The 4 cases in the 5th line of Tab. 2 (sound/unsound/unsound) could clearly be traced to a bug in the ProM plugin which is of no interest for our comparision of the theory behind the tools. The interesting cases are the ones where the results from the tools differ, i.e. one case “unsound/sound/unsound” and the 12 cases “unsound/sound/sound”, highlighted by gray background. We will discuss the differences shown in the gray lines of Tab. 2 and other remarkable results for the different tools in the next sections. 5.1

EPCTools (using Fixed-Point Semantics)

EPCTools tries to compute a fixed-point semantics for each model. From the 984 models in our repository, exactly 3 did not have such a semantics, i.e. EPCTools identified them as not being clean. As all tools used in our survey, EPCTools defines the state of an EPC by placing tokens on modelling elements. An initial state is one for which only start events carry a token. EPCTools than computes soundness based on the following definition: An EPC is sound in a given initial state if, from all reachable states, a proper terminal state can be reached; where a proper terminal state is a state in which only end events carry a token2 . This means that it is assumed that the initial state (i.e. combination of start events that can trigger the execution of the model) is known. As this information is in most cases not part of the model, we considered all possible combinations of start events and let EPCTools check for which of these combinations the EPC is sound. Afterwards, we classified an EPC as sound iff there is an initial state for which it is sound. This is the same approach that has been used in the EPC soundness definition by Mendling [9] which is implemented by the ProM plugin. However, by comparing EPCTools’ soundness results defined this way with the ones computed by the ProM plugin, we found 46 models that were reported 2

This definition has been slightly simplified, in reality the tokens are placed on the arcs instead of the modelling elements.

7 E1

E2

END

Fig. 1. Regarded as sound by EPCTools when execution is triggered by E1

as being sound by EPCTools while ProM identified them as unsound. The reason for this difference lies in the fact that EPCTools did not take the third property of the soundness definition (no needless elements) into account. For example, the model in Fig. 1 has a proper execution when triggered by start event E1. However, if triggered by start event E2, a deadlock at the AND-join will occur. Hence, the model should be regarded as unsound, because E2 never contributes to a proper execution of the model. The conclusion is that for models with more than one start event, EPCTools fails to detect problems that result from needless elements. 5.2

ProM Plugin (using Mendling’s Semantics based on State and Context)

The ProM plugin for EPC soundness analysis is based on the state/context semantics defined by Mendling [9]. The plugin uses the soundness definition by Mendling [9]. In short, for an EPC to be sound, it is required that 1. There is at least one initial state (i.e. a combination of start events which are marked at the beginning of the execution) that leads to an execution ending in a state where only events without outgoing arcs are marked. 2. Every start event belongs to such an initial state. 3. From the selected initial states, it is not possible to reach a state other than an end state (where only events without outgoing arcs are marked) that does not have a successing state. ProM found that all three EPCs that have computed to be unclean by EPCTools are not sound under Mendling’s semantics. For the majority of EPCs, the soundness results from the ProM plugin coincided with the soundness results of EPCTools. However, we have identified one class of models where a model which is sound according to the fixed-point semantics runs into a deadlock under Mendling’s semantics. All those models for which the result from EPCTools differs from the result by the ProM plugin contain a pattern where an OR join is the entry into a loop. Fig. 2 (a) shows the most basic variant of such a pattern. For this model, Mendling’s semantics would lead to a deadlock at the OR-join. In our opinion,

8 (a)

(b)

(c)

Fig. 2. Models that have a deadlock under Mendling’s semantics

this is an undesirable property of this semantics. The statement “an OR-join can always replace any XOR-join or AND-join” which holds for the behaviour of models under other semantics becomes wrong for Mendling’s semantics. Fig. 2 (b) shows a variant of this pattern. This model would also deadlock at the OR-join. The interesting point here is that the reduction rules published in [9] (which are assumed to be soundness-preserving) would completely reduce the model in Fig. 2 (b) which would lead to the wrong result that the model is sound. As a consequence, the reduction rule that removes a control block starting with an XOR-split and ending with an OR-join should be removed from the set of reduction rules in [9], because it does not necessarily preserve soundness. Finally, the EPC in Fig. 2 (c) is a third variant of the same pattern. Here the OR-join at which the deadlock occurs would have to be replaced by a combination of an OR-join (which ends the block started by the OR-split) and another XOR-join (as a loop end point) in order to make the model sound. 5.3

YAWL Editor (using YAWL semantics)

In order to avoid problems of OR-joins depending on each other, the YAWL semantics computes the ability to forward tokens for each OR-join separately. Other OR-joins are assumed to act like XOR-joins with a non-local semantics, i.e. they forward every incoming token. The computation whether an OR-join can forward tokens is performed by computing the predecessor markings of the current marking (see [18] for details). For the models from our repository, the soundness results delivered by YAWL were almost identical to the soundness results computed by EPCTools. The differences will be discussed below. Other that the semantics definitions used by EPCTools and ProM, the semantics definition of YAWL allows elements marked with more than one token. Such a definition allows the state space of a model to become infinite. However, the analysis presented in [18] works for models with a finite state space only, and

9

Fig. 3. Models with an infinite state space in the YAWL analysis

no algorithm is given that can decide whether the state space will become infinite. The YAWL editor stops the computation when a threshold of 10,000 states is reached. Such a situation most likely indicates that the state space becomes infinite and the model is not sound. From the 737 EPCs that have been validated by YAWL, the mentioned restriction for 10,000 states was enforced for 22 models. From the 3 models which are unclean according to EPCTools, two could be reduced to models with a single start event and hence analysed by YAWL. For both, the state space exceeded 10,000 states. All other models for which YAWL stopped after the threshold of 10,000 markings was reached were unsound under the fixed-point semantics used by EPCTools. This result supports the expectation that an increase above 10,000 markings indicates an error in the model. However, a drawback of the YAWL analysis is that even in simply-looking cases like the ones shown in Fig. 3 the error cannot be found. Combining the YAWL approach with techniques like invariants [23] or reduction rules with error cases [9] could help to improve the results. Another interesting case is shown in Fig. 4. This model contains two ORjoins in a feedback loop. EPCTools computes a fixed-point semantics (where both OR-joins forward a token without having to wait). However, the YAWL semantics concludes that both OR-joins block while waiting for another token to arrive. Hence, this model is sound according to EPCTools but unsound according to YAWL3 . In our opinion, in this case the fixed-point semantics meets the expectation of the modeller better than YAWL semantics.

Fig. 4. Two OR-joins depending on each other

3

Because of an error in the implementation of the reduction rules, YAWL has to be started without applying YAWL reduction rules for coming to this result. Note that removing the loop is not soundness preserving.

10

(a)

(b)

Fig. 5. Two unclean models

6

Findings

In this section, we want to summarize the findings of our analysis: 6.1

Unclean Models

Three models that do not have a fixed-point semantics have been found among our real-world examples. Although they are rare (3 out of 984), such models exist, i.e. the discussion about their semantics is not just an academic pastetime. To our surprise, we even found an instance of a model that was almost identical to the original vicious circle published in [1]. Other examples for unclean models are the “partial redo” pattern (see Fig. 5 (a), discussed in [24] and [7]) and the pattern shown in Fig. 5 (b). 6.2

Soundness Definitions

The soundness definition used by EPCTools differs from van der Aalst’s definition [21, 22]: It does not take into accout the requirement that there are no elements in the model which do not contribute to a proper completion of the model. This way, some models are classified as being sound (for a certain initial state) even if such useless elements are present. We recommend to prefer Mendling’s definition [9] (as used in the ProM plugin) which pays attention to the “no needless elements” requirement. 6.3

Mendling’s Semantics

While Mendling’s semantics based on state and context performes well for most models, we have identified a class of models for which it leads to unexpected results. OR-joins that are a loop entry will lead to a deadlock. While Mendling’s

11

semantics has several desirable properties (as discussed in [15, 9]), it does not have the property that any AND- or XOR-join can be replaced by an OR-join without affecting the semantics of the model. We have shown an example for which the reduction rules given in [9] are not soundness-preserving as assumed. However, by simply removing one rule from the rule set, the problem disappears. 6.4

YAWL Semantics

For most models, the analysis using YAWL semantics lead to the same results as the EPCTools analysis using fixed-point semantics. We found exactly one model for which YAWL computes the soundness property differently from EPCTools (which uses fixed-point semantics). A drawback is that for models with a infinite state-space some kinds of errors cannot be located.

7

Conclusion and Directions of Future Research

In our comparative study of soundness results computed by three different tools, we found some differences that can lead to interesting insights on advantages and disadvantages of semantics definitions. A by-product of our analysis of a large number of models was that some bugs in YAWL (in particular in the reduction rules) could be detected. We would like to thank the YAWL community and in particular Arthur ter Hofstede, Michael Adams and H.M.W. Verbeek for the fruitful discussion and for fixing the bugs very quickly. From the fact that we found several bugs in YAWL and one in the ProM plugin, we have learned the lesson that testing with a large repository of real-world models is very useful for assuring a high quality of tools that validate business process models. As a next step, we will look into the question for which category of models the execution of the soundness analysis by the tools takes unusual long time. We hope that this can help to improve the algorithms used for the validation.

References 1. van der Aalst, W.M., Desel, J., Kindler, E.: On the semantics of EPCs: A vicious circle. In: EPK 2004, Gesch¨ aftsprozessmanagement mit Ereignisgesteuerten Prozessketten. (2002) 71–79 2. van der Aalst, W.M., ter Hofstede, A.H.M., Kiepuszewski, B., Barros, A.: Workflow patterns. Distributed and Parallel Databases 14 (2003) 3. Kindler, E.: On the Semantics of EPCs: A Framework for Resolving the Vicious Circle. In: Business Process Management. (2004) 82–97 4. Cuntz, N., Kindler, E.: On the semantics of EPCs: Efficient calculation and simulation. In: EPK 2004: Gesch¨ aftsprozessmanagement mit Ereignisgesteuerten Prozessketten, Proceedings. (2004) 7–26 5. Cuntz, N., Freiheit, J., Kindler, E.: On the Semantics of EPCs: Faster calculation for EPCs with small state spaces. In: EPK 2005, Gesch¨ aftsprozessmanagement mit Ereignisgesteuerten Prozessketten. (2005) 7–23

12 6. Wynn, M.T., Edmond, D., van der Aalst, W.M., ter Hofstede, A.H.M.: Achieving a General, Formal and Decidable Approach to the OR-Join in Workflow Using Reset Nets. In: ICATPN. (2005) 423–443 7. Dumas, M., Grosskopf, A., Hettel, T., , Wynn, M.: Semantics of BPMN process models with or-joins. Technical Report Preprint 7261, Queensland University of Technology, Brisbane (2007) 8. van Hee, K.M., Oanea, O., Serebrenik, A., Sidorova, N., Voorhoeve, M.: Historybased joins: Semantics, soundness and implementation. In: Business Process Management. (2006) 225–240 9. Mendling, J.: Detection and Prediction of Errors in EPC Business Process Models. PhD thesis, Vienna University of Economics and Business Administration (2007) 10. B¨ orger, E., S¨ orensen, O., Thalheim, B.: On defining the behavior of OR-joins in business process models. J. Universal Computer Science 14 (2008) 1–22 11. Rittgen, P.: Quo vadis EPK in ARIS? Wirtschaftsinformatik 42 (2000) 27–35 12. Wehler, J.: Boolean and free-choice semantics of event-driven process chains. In: EPK 2007, Gesch¨ aftsprozessmanagement mit Ereignisgesteuerten Prozessketten. (2007) 77–96 13. van der Aalst, W.M.: Formalization and verification of event-driven process chains. Information & Software Technology 41 (1999) 639–650 14. Langner, P., Schneider, C., Wehler, J.: Relating event-driven process chains to Boolean Petri nets. Report (1997) 15. Mendling, J., van der Aalst, W.M.: Formalization and verification of EPCs with OR-joins based on state and context. In: Proc. of the the 19th International Conference on Advanced Information Systems Engineering (CAiSE 2007). (2007) 16. van Dongen, B.F., van der Aalst, W.M., Verbeek, H.M.W.: Verification of EPCs: Using reduction rules and Petri nets. In: CAiSE. (2005) 372–386 17. Barborka, P., Helm, L., K¨ oldorfer, G., Mendling, J., Neumann, G., van Dongen, B.F., Verbeek, E., van der Aalst, W.M.: Integration of EPC-related tools with ProM. In N¨ uttgens, M., Rump, F.J., Mendling, J., eds.: EPK. Volume 224 of CEUR Workshop Proceedings., CEUR-WS.org (2006) 105–120 18. Wynn, M.T.: Semantics, Verification, and Implementation of Workflows with Cancellation Regions and OR-joins. PhD thesis, Queensland University of Technology Brisbane, Australia (2006) 19. Wynn, M.T., Verbeek, H., van der Aalst, W.M., Edmond, D.: Business process verification - finally a reality! Business Process Management Journal (15) 74–92 20. Decker, G., Mendling, J.: Instantiation semantics for process models. In: Proceedings of the 6th International Conference on Business Process Management, Milan, Italy. (2008) 21. van der Aalst, W.M.: Verification of workflow nets. In Az´ema, P., Balbo, G., eds.: Application and Theory of Petri Nets 1997, 18th International Conference, ICATPN ’97, Toulouse, France, June 23-27, 1997, Proceedings. (1997) 407–426 22. van der Aalst, W.M.: Structural characterizations of sound workflow nets. Computing Science Reports/23 (1996) 23. Verbeek, H.M.W., van der Aalst, W.M.P., ter Hofstede, A.H.M.: Verifying workflows with cancellation regions and or-joins: An approach based on relaxed soundness and invariants. Comput. J. 50 (2007) 294–314 24. Gruhn, V., Laue, R.: Good and bad excuses for unstructured business process models. In: Proceedings of 12th European Conference on Pattern Languages of Programs (EuroPLoP 2007). (2007)