1
Complete and Interpretable Conformance Checking of Business Processes ˜ Luciano Garc´ıa-Banuelos, Nick R.T.P. van Beest, Marlon Dumas, Marcello La Rosa and Willem Mertens
F
A PPENDIX S URVEY INSTRUMENT This experiment is part of a joint research project by Data61 (Australia), University of Tartu (Estonia) and Queensland University of Technology (Australia) on the use of two different business process conformance checking approaches. Given an event log recording the actual execution of a business process, and a process model capturing its expected or normative execution, the goal of conformance checking is to pinpoint and describe any differences between the process behavior observed in the event log and that captured in the process model. Any such difference could indicate a conformance problem, e.g. lack of compliance. Using this experiment, we aim to assess the relative merits and weaknesses of two different business process conformance checking approaches, herewith called “Alignment” and “Verbalization”. Therefore, we will present a process model and the differences between this model and an event log, as obtained by using the two different approaches. Subsequently, we will inquire about your perceived usefulness, ease of use and intention to use for the two alternative approaches. Altogether, this questionnaire contains four parts: • • •
•
• • •
Part 1 gathers information on your process modeling background and your experience with Petri net modeling. Part 2 checks your level of theoretical process modeling knowledge. Part 3 presents the process model and provides for each of the two approaches the output showing the differences between this model and the log, along with a short introduction on how to interpret the output. Part 4 inquiries about your preference for either of the two approaches.
L. Garc´ıa-Banuelos ˜ and M. Dumas are with the University of Tartu, Estonia. E-mail: {luciano.garcia, marlon.dumas}@ut.ee N.R.T.P. van Beest is with Data61, CSIRO, Brisbane, Australia. E-mail:
[email protected] M. La Rosa and W. Mertens are with the Queensland University of Technology, Australia. E-mail: {m.larosa, w.mertens}@qut.edu.au
The total time required for this survey is approximately 10-15 minutes. When answering the questions, please bear in mind the following: • • •
Completing this questionnaire is voluntary. This is an anonymous questionnaire. Please do not try to guess any answer. If you are not reasonably sure about the answer to a given question, please simply answer “I don’t know”. In this way, the results will be more accurate.
Thank you for your participation! If you have any questions about the survey, you can send your query to
[email protected].
Part 1: Background questions 1) Which description matches best your current status?
Academic
Professional 2) How many years ago did you start modeling business processes? 3) How many process models have you analyzed or read within the last 12 months? (A year has about 250 work days. In case you read one model per day, this would sum up to 250 models per year) 4) How many process models have you created or edited within the last 12 months? 5) How many activities did these models have on average? 6) How many work days of training on process modeling have you received within the last 12 months? (This includes e.g. university lectures, certification courses, training courses. 10 weeks of a 120-minute university lecture is roughly 3 work days) 7) How many work days of self education in process modeling have you made within the last 12 months? (This includes e.g. learning-by-doing, self-study of textbooks or specifications)
2
Please rate your agreement with the following statements. Please answer each question open and honestly. 8) Overall, I am very familiar with Petri nets. Strongly disagree
Disagree
Somewhat disagree
Neutral
Somewhat agree
Agree
Strongly agree
9) I feel very confident in understanding process models created with Petri nets. Strongly disagree
Disagree
Somewhat disagree
Neutral
Somewhat agree
Agree
Strongly agree
Part 3: Learning the Alignment approach and the Verbalization approach Figure 1 shows the Petri net of a given business process, say a process for handling insurance claims. Let us assume we have an event log recording 53 executions of this claims handling process. Naturally, these executions may slightly differ from the intended process behavior as represented in the model. These differences can be represented using two different approaches: Alignment and Verbalization.
10) I feel very competent in using Petri nets for process modeling. D
Strongly disagree
Disagree
Somewhat disagree
Neutral
Somewhat agree
Agree
Strongly agree
I
J
K G
H A
B
C
E
Part 2: Your theoretical knowledge of business process modeling
F
Figure 1. Business process model under investigation.
Please answer each question open and honestly. Avoid guessing. 11) After an exclusive choice, at most one alternative path is executed.
True
False
I don’t know 12) Exclusive choices can be used to model repetition.
True
False
I don’t know 13) Synchronization means that two activities are executed at the same time.
True
False
I don’t know
Next, we show the output of each of these approaches along with a short explanation on how to interpret this output. Note that it is not the intention of this survey to fully understand the process itself. We are interested in whether you find the representation of model-log differences provided by each approach easy to use and useful. Approach 1: Alignment The Alignment approach consists of two distinct visualizations. The first one shows a Petri net representation of the process model, where the differences in frequencies and occurrences w.r.t. the log are indicated with different colors. Figure 2 shows the model projection of the changes, along with the legend on how to interpret the different colors and arrows.
D (27/0)
14) An AND-split can activate concurrent paths.
True
False
I don’t know
I (87/0)
J (87/0)
H
A (58/0)
1
B (41/15)
K (87/0)
G (18/0)
C (87/15)
E (9/0)
15) If two activities are concurrent, they have to be executed at the same time.
True
False
I don’t know
Low
High
Transition label
F (12/0)
(Freq mutual occurrence / Freq model occurrence only) Occurrence in model and log (ratio)
Occurrence in model only (ratio)
- Transition color shows frequency - Red border: the transition occurs in the model - Grey transition: invisible Marked place for log only Other place marked when a log only transition occurred States where no transition occurred in the log
16) If an activity is modeled to be part of a cycle, it has to be executed at least once.
True
False
I don’t know
- Place size shows freq. of occurrence in the log
Figure 2. Model representation of the Alignment approach. The second visualization of the Alignment approach shows the different traces in the event log including the individual differences with the process model. Figure 3 shows the alignments of all different traces along with a legend indicating how to interpret the color coding.
3 ABC
HE I J K
CD I J K
C
E
ABC
HE I J K
CD I J K
C
ABC
HE I J K
C
ABC
HE I J K
CD I J K
G
ABC
HE I J K
CD I J K
ABC
HE I J K
C
ABC
HE I J K
C
HE I J K
HE I J K
A BCD I J K
C
A BCD I J K
H I JK
G
ABC
HE I J K
HI JK
CF
A BCD I J K
C
E
E
ABC
HE I J K
HI JK
CF
ABC
HI JK
CF
CF
A BCD I J K
C
HE I J K
ABC
HI JK
CF
ABC
HE I J K
CD I J K
ABC
HI JK
G
G
ABC
HE I J K
CD I J K
ABC
HE I J K
HE I J K
ABC
HE I J K
HI JK
ABC
E
ABC
HE I J K
HI JK
A
E
I JK
ABC
HE I J K
C
E
G
A
E
G
ABC
HE I J K
C
E
G
HE I J K
C
G
E
G
ABC
HE I J K
C
HI JK
ABC
HE I J K
C
E
ABC
HE I J K
C
HI JK
ABC
HE I J K
C
E
ABC
HI JK
HE I J K
ABC
HI JK
G
G
20) What is the most useful approach for identifying the differences between a process model and an event log? Strongly Somewhat Somewhat Strongly Prefer Prefer prefer prefer Neutral prefer prefer alignment verbalization alignment alignment verbalization verbalization
21) Which approach would you likely use for checking the conformance of an event log to a process model? Strongly Somewhat Somewhat Strongly Prefer Prefer prefer prefer Neutral prefer prefer alignment verbalization alignment alignment verbalization verbalization
Legend Activity occurs both in the model and the log
C
C
E
G
Activity occurs in the model only and is not observed anywhere in the log Activity occurs in the model only, but occurs in the log in another trace Activity occurs in the log only, but occurs in the model in another path
Figure 3. Trace alignments provided by the Alignment approach. Approach 2: Verbalization The verbalization approach presents the differences between the process model and the event log in natural language by showing a set of statements, each indicating a specific difference. Differences: • In the event log, after the occurrence of A, B is substituted by E • In the event log, after the occurrence of C, H is substituted by E • In the event log, D can be skipped, while in the model it cannot • In the event log, G occurs after E, while in the model it does not • In the event log, I occurs after E, while in the model it does not • In the model, C can be skipped, while in the event log it cannot • In the model, H occurs after K, while in the event log it does not • In the model, E is repeated multiple times, while in the event log it is not • In the model, G is repeated multiple times, while in the event log it is not Part 4: Assessment of the approaches 17) What is the easiest approach for checking the conformance of an event log to a process model? Strongly Somewhat Somewhat Strongly Prefer Prefer prefer prefer Neutral prefer prefer alignment verbalization alignment alignment verbalization verbalization
18) What is the most useful approach for checking the conformance of an event log to a process model? Strongly Somewhat Somewhat Strongly Prefer Prefer prefer prefer Neutral prefer prefer alignment verbalization alignment alignment verbalization verbalization
19) Which approach would you likely use for identifying the differences between a process model and an event log? Strongly Somewhat Somewhat Strongly Prefer Prefer prefer prefer Neutral prefer prefer alignment verbalization alignment alignment verbalization verbalization
22) What is the easiest approach for identifying the differences between a process model and an event log? Strongly Somewhat Somewhat Strongly Prefer Prefer prefer prefer Neutral prefer prefer alignment verbalization alignment alignment verbalization verbalization
