Checking Temporal Integrity of Interactive Multimedia Documents I. MIRBEL1, B. PERNICI2, T. SELLIS3, S. TSERKEZOGLOU4, M. VAZIRGIANNIS4
1
Laboratoire I3S, CNRS-UNSA, Sophia Antipolis, 06560, France.
[email protected]
2
Politecnico di Milano, Milano, Italy.
[email protected]
3
Dept. of El. & Comp. Engineering, National Technical University of Athens, Hellas.
[email protected] 4
Dept of Informatics, Athens University of Economics & Business, Hellas.
[email protected]
Abstract. When authoring multimedia scenarios, and in particular scenarios with user interaction, where the sequence and time of occurrence of interactions is not predefined, it is difficult to guarantee the consistency of the resulting scenarios. As a consequence, the execution of the scenario may result in unexpected behavior or inconsistent use of media. The present paper proposes a methodology for checking the temporal integrity of Interactive Multimedia Document (IMD) scenarios at authoring time at various levels. The IMD flow is mainly defined by the events occurring during the IMD session. Integrity checking consists of a set of discrete steps, during which we transform the scenario into temporal constraint networks representing the constraints linking the different possible events in the scenario. Temporal constraint verification techniques are applied to verify the integrity of the scenario, deriving a minimal network, showing possible temporal relationships between events given a set of constraints.
Keywords. Multimedia presentation, Constraint networks, Temporal integrity
1
1.
Introduction
An Interactive Multimedia Document (IMD) involves a variety of individual multimedia objects presented according to the presentation sequence specified in an IMD scenario. During the presentation, multimedia objects that participate in the IMD are transformed, either spatially or temporally, in order to be presented according to author's requirements [Vazirgiannis 96] specified in the interactive multimedia document description. At authoring time, the author defines the spatial and temporal order of objects within the application context and the relationships among them. In addition, the way the user will interact with the application as well as the way the application will treat application or system events have to be defined. The related application domains are quite challenging and demanding, such as interactive TV, synthetic movies, animation etc. In such interactive documents we consider the following constituents: •
the Objects that participate in the document along with their spatial and temporal transformations (i.e. cropping, scaling) to align with the authors requirements.
•
the Events, considered as the fundamental carriers of interaction in the context of an IMD, raised by user actions, by objects participating in the IMD or by the system. They may be simple (i.e. not decomposable in the IMD context) or complex and have attached a spatiotemporal signature (i.e. the space and the time they occur).
•
Spatiotemporal composition: represents the spatial and temporal ordering of the participating objects in the IMD, i.e. when, and where each object is presented.
•
Scenario: the integrated behavioral contents of the IMD, i.e. which events will trigger the spatiotemporal compositions defined above.
Real world large IMDs can be very complex with respect to (i) the number of involved objects, (ii) transformations of the objects in the scope of an application, and (iii) relationships among them. It is obvious that in a complex IMD it may be very difficult, at authoring time, to keep track of all paths the application may follow, i.e. to decide if the document is temporally consistent. Interaction, with its inherent temporally indeterminate elements, may complicate the process for 2
temporal integrity checking for verifying the temporal consistency of the scenario. As temporally consistent we define a document that under any allowed interaction will not lead to runtime errors and moreover eventually lead to a state where all the participating objects end their presentation.
In the present work, we propose a methodological approach for IMD temporal integrity checking focusing on temporal properties of the presentation. An interesting issue is the temporal consistency of the presentation, that provides the possibility to decide in advance, i.e. before the scenario execution, if the order of presentation actions is consistent according to a set of constraints. These constraints may be either generic (i.e., “an idle object cannot be stopped”) or application specific (i.e., “video clip A should not overlap with sound B” or “two sounds should not be active at the same time”). The scenario integrity issue is crucial when designing a complex scenario. Moreover the multitude of events and the related interactions that may occur in an IMD session may create a lot of potential IMD evolution paths which are difficult to follow at authoring time, considering all potential implications. Therefore there is the necessity to provide a formal and sound background for building tools to support authors in designing consistent scenarios and to verify the integrity of existing ones.
The approach we elaborate on in the following sections is based on previous work of some of the authors for modeling interactive multimedia scenarios [Vazirgiannis and Boll B; Vazirgiannis et al. 98] which is briefly illustrated in section 2. In this approach, the scenario consists of a set of self-standing functional units (scenario tuples) that include: triggering events (start and stop events), presentation actions (in terms of spatiotemporal compositions of actions on multimedia objects) to be carried out in the context of the scenario tuple, and related synchronization events. In the rest of the paper we will interchangeably use the terms integrity and consistency referring to the correctness of the temporal aspects of an IMD scenario. The methodology for checking interactive multimedia document consistency presented in the paper has the following main characteristics: The methodology is based on the transformation of scenario specifications into networks representing the temporal constraints between multimedia objects and 3
those concerning their evolution. During the transformation, the effect of possible user interactions during the scenario execution has to be taken into account. The method is based on a set of rules for deriving constraint networks. Techniques from temporal reasoning [Brusoni et al. 97] are applied in this context to analyze the resulting networks and examine their consistency and potential problems.
The structure of the paper is as follows. In Section 2, we present the scenario model that serves as the basis of our integrity checking methodology. In Section 3, we discuss properties of a consistent scenario with respect to temporal aspects. The integrity checking (Section 4) consists of a set of discrete steps which transform scenarios into temporal networks that correspond to the fundamental functional units of the scenario model mentioned above. In Section 5, we present a way to use the approach in order to evaluate queries and additional constraints that may be posed to the scenario structure. In Section 6, we discuss the methodology in terms of time complexity and potential optimizations. In Section 7, we present related work and compare it to our approach.
2.
The Scenario Model
In this section we present briefly the scenario model that serves as the basis of the temporal constraint checking scheme. A multimedia scenario is defined as the high level definition of a multimedia application functionality that covers the spatial and temporal synchronization of participating media objects and the handling of interaction between the user, the system and the application. In the present paper, we are going to exploit an existing model for IMDs [Vazirgiannis and Boll 97; Vazirgiannis et al. 98]. According to that model an IMD consists of the spatial and temporal composition of the participating media objects, the definition of interaction between the user and the IMD (external interaction), the interaction between internal IMD entities (internal interaction) and the specification of media synchronization in the temporal and spatial domains.
The term scenario in the context of IMDs stands for the integrated behavioral contents of the IMD, i.e. what kind of events the IMD will consume and what actions will be triggered as a result. The scenario consists of a set of autonomous 4
functional units (scenario tuples) that include the triggering events (for starting and stopping the scenario tuple), the presentation actions to be carried out in the context for the scenario tuple, related synchronization events and possible constraints. More specifically, a scenario tuple has the form: (Start_event, Stop_event, Action_list, Synch_events) where: •
Start_event represents the event expression that triggers the execution of the actions described in the Action_list.
•
Stop_event represents the event expression that terminates the execution of this tuple (i.e. it terminated the execution of the actions described in the Action_list before its expected termination).
•
Action_list represents the list of synchronized media-presentation actions that takes place when this scenario tuple becomes activated. The expressions included in this attribute are in terms of compositions as described in the following and as in [Vazirgiannis et al. 98].
•
Synch_events is a pair of events generated at the beginning and at the end of the current tuple execution. These events may be used for synchronization purposes.
Temporal Relationships Hereafter we present a set of Temporal Access Control (TAC) operators [Little and Ghafoor 93] that represent the temporal composition of media objects, together with the causality of temporal relationships among presentation intervals. For more details refer to [Vazirgiannis et al. 98]. These operators correspond to the well-known TAC actions: start (>), stop(!), pause (||), resume (|>). Therefore: TAC_operation :== “>” | “!” | “||” | “|>” For instance, assume a multimedia object A, then “A>” represents the start action of the multimedia instance, “A!” the stop, “A||” the pause and “A|>” the resume actions. We have to illustrate the events arising from the temporal state changes of a multimedia object. i.e. when the object A starts its presentations then the “A>” temporal event is raised. Special attention should be paid to the event generated when the object finishes its execution (“” | “” | “||” 5
Following, we define a temporal composition scheme based on the one presented in [Vazirgiannis et al. 98], to represent the temporal composition of media objects in the context of an IMD scenario. Let time be the length of a temporal interval between two events or TAC operations. Therefore: temporal_composition := , | $ | (object (t_event | TAC_operation) [{time object TAC_operation}] where , is the spatiotemporal origin of an IMD session. Additionally the symbol $ represents the start of the tuple. For instance: the expression: “A> 4 B! 0 C>” conveys “Start A, after 4 time units, stop B, and just after this start C”. Finally, we define the natural duration
dA
of a multimedia object
A
as the temporal interval
between the temporal events “A>” and “A” is applied to, it falls into the active state. All the allowed transitions between states and the corresponding actions appear in Fig. 1. It has to be stressed that the object’s initial and final state is always the idle one. ! Idle Active
< >
||
!
|> Suspended
Figure 1: State transition diagram for time-dependent media object
In a multimedia scenario there may be a set of actions that are applied to a media object due to scenario expressions. The execution of these actions must obey to some constraints. For instance, it is not possible to stop an object which is not active (i.e. in idle state). In the following table, we list the runtime constraints that emanate from the above diagram (i.e. which conditions must be fulfilled in order to be able to apply an operator):
OPERATOR/EVENT
NAME
CONSTRAINT
>
Start
The object must be in IDLE state
!
Stop
The object must be in ACTIVE or SUSPENDED state
||
Pause
The object must be in ACTIVE state
|>
Resume
The object must be in SUSPENDED state
, !, ||, |>, and t the time elapsed between the two operations. The first general constraint is that : “t should be such that the time of execution of the second action is not before the start of the scenario execution ”. Moreover it is important to stress the fact that some combinations are not feasible (“illegal”) and, among the rest of the combinations, some are feasible under some constraints. An exhaustive combination of TAC operations and their features, at authoring time, appears in Table 2. In this table the sequence of actions and the time t have to respect the constraints imposed by state transition diagram of Fig. 1. An important clarification here refers to the fact that the consecutive actions appearing in each line of Table 2 refer to the same instance (i.e. presentation) of each object. This means that a second presentation of the same object is considered as another instance. To start again an object which is already active, a second instance of the object has to be started. This is possible in our model. The sequence “start-start” would lead to a cycle, which is not allowed in the proposed methodology. Moreover, we consider time as linear. A second start action on a specific object in the current execution plan would result in potential shift back in time. Therefore, concurrent play outs of the same object are allowed but considered as different objects. We also assume that t can also be a negative value.
ACTION
ACTION
Legal
Constraint
1
>
>
No
2
>
!
Yes
0
Yes
6
!
!
No
-d
!
Yes
0
No
t>0
d: duration of object presentation r: remaining time of object presentation Table 2: The constraints on pairs of consecutive TAC actions on the same object
Summarizing the above table, the sequences of action in lines: 1,4, 6, 11, 13 and 16 violate the state transition diagram of Fig. 1 and thus are considered as illegal. One could argue that, for instance, in line 1 a second > operator would not harm. This is true but it would imply that a second instance of the same object should be started, whereas in our approach consecutive application of operators are applied to the same instance. Thus we do not allow TAC actions that cannot change the temporal state of the object starting from the current one.
3.2. Scenarios represented as Networks of Temporal Constraints As a basis for our methodology for consistency checking and scenario analysis support, we adopt an internal representation of scenarios based on constraint networks. The derivation of constraint networks is a non trivial problem, in particular when considering different possible user interactions during the scenario execution. The methodology for deriving temporal constraint networks from scenario descriptions is the core of the present work and is described in the following sections. In the present section, we introduce the basic assumptions on the type of networks we use for our analysis.
10
We assume to derive from scenario descriptions a set of temporal constraint networks which represent the constraints between the different actions performed within a particular scenario execution, and representing also multimedia objects internal properties, such as for instance the maximum allowed duration for an object execution.
Constraints derived from scenario tuples are represented internally as conjunctions of bounds of differences between variables representing events [Dechter et al. 91; Brusoni et al. 97]. Such differences represent the minimum and maximum possible temporal distances between events, as derived from the specification of actions in tuples and from possible concatenations of tuple executions. For instance, if event a is at least 10 units of time before event b, but b occurs no later than 50 time units after a, we write 10 ≤ b-a ≤ 50. We build networks (or network fragments), where nodes represent events in tuples (internal and external events), and arcs express the minimum and maximum possible distances between two nodes n1, n2 in the form b1≤ n2- n1 ≤ b2. In such a network, a variable name can appear only once.
For each network (built according to the rules illustrated in next section), the consistency of the conjunction of bounds of differences can be checked using temporal reasoning techniques such as the STP framework [Dechter et al. 91] in O(n3) time, where n is the number of nodes in the network. The algorithm checking the consistency produces also the minimal network, representing all possible solutions, i.e., possible assignments to the variables satisfying the constraints. The minimal network contains, for each pair (ni, nj) of variables, the maximal range [c,d] that includes all and only the values for nj- ni satisfying all the constraints in the network [Dechter et al. 91]. According to this approach a network is inconsistent if there are negative cycles, i.e. a cyclic path from a node to itself which results in negative bounds of differences. In the following, we adopt a graphical representation of the bounds of differences, using a graph where the nodes correspond to the network nodes, and arcs are labeled with the minimum and maximum distances between the events represented by the nodes.
11
In the following sections, we show how to build constraint networks, how to verify local properties of portions of the scenarios using reasoning on network fragments, and how to pose queries to the minimal networks to examine scenario properties during authoring. The issue of considering the complex problem of examining all possible scenarios executions is also considered in the methodology and solutions provided according to a defined set of hypothesis on the scenario being authored.
4.
IMD Scenario Integrity Checking
The temporal integrity of an IMD scenario may be considered at two levels:
At scenario tuple level, we check if the sequence of events, as they are generated by the action list of the scenario tuple, is temporally consistent according to the criteria defined in previous sections. At scenario level, we examine the various execution paths that the IMD may follow according to the events generated. For each path, we verify its consistency. Thus we find the consistent execution paths. The consistency at scenario level is checked using the STP framework Dechter & al. 91].
The consistency checking methodology may be summarized in the following. The IMD scenario as defined in the previous section consists of a set of scenario tuples. Each of them is transformed into a constraint network fragment (cnf). A cnf represents the temporal constraints between the start/stop events of the tuple and the TAC actions/events in the action part of the scenario tuple. Due to the interactive nature of the IMD scenarios there are a lot of alternative sequences of events that will start/stop tuples. Each of these sequences (paths) is an execution plan for the IMD scenario. Thus after each event we have a different set of tuples that are active (i.e. their action part is being executed). Each such set of active tuples defines a state for the IMD scenario. The transition between states is caused by occurring events and defines an execution plan. There are roughly as many execution plans as many as paths. In the case of pre-orchestrated scenarios (i.e. no interaction is assumed), only one execution plan is possible, while in interactive scenarios several alternatives for execution have to be considered. Each execution
12
plan defines a sequence of tuples, which is mapped in a composition of the corresponding cnfs. The result is a composite constraint network (cn).
The temporal consistency of the IMD scenario is carried out in the following consecutive stages: •
Qualitative checks, in which we verify if the sequences of TAC actions are legal according to the state diagram in Table 1.
•
Quantitative checks, in which we verify whether the temporal intervals between legal sequences of TAC actions are not creating inconsistencies, according to the criteria set in Table 2.
•
Temporal consistency check, that verifies the global temporal consistency of the scenario. This verification is carried out exploiting the framework presented in [Dechter et al. 91].
Through the stages of the methodology, we distinguish warnings from errors. An error is detected when a certain runtime error occurs as a result of an action expression. Then the scenario cannot be further executed. For instance, the stop of an object which is not anymore running lead to an execution error. As regards warnings we distinguish two cases. In the first one, there may be cases in which there is an action expression that locally causes an error that may be fixed when combining it with other actions from other tuples in the process of constraint networks building. These cases raise warnings. Also, paths leading to the undefined state generate to potential errors, and should be signaled by a warning.. The second case in which warning is raised is when the scenario may not run properly or may lead to an execution error. For instance, a scenario tuple which never ends is considered only as a warning because it may be required by the designer of the scenario. The methodology consists of four steps (see Fig. 2): Step 1: Network Fragment Design: this step aims at translating each tuple of the scenario in one or several constraint network fragments (cnfs), expressing constraints on the tuple events (Start Event, Stop Event, Synch Event) and the TAC events (generated by the TAC actions : >, ).
13
Step 1 IMD scenario
Constraint Network Fragments
Transformation
Preliminary consistency h k
Step 2
Preliminary consistency check
Transformation ∀ path
Step 3 Composition of cnfs
Execution plans Preliminary Constraint networks
( Step 4
) ∀ cn
consistency h k
Consistency check
Valid paths
Figure 2: The architecture of the IMD scenario consistency checking methodology
Step 2: Alternative Execution Plans: this task is carried out independently of step 1. The aim here is to analyze the event sequences in the scenario, in order to define the different possible execution plans (also called paths in the following). The global states that can be reached in the scenario are identified, and corresponding transitions between states are derived from the tuple execution. The result of this step is the execution plans diagram. Step 3: Composition of Network Fragments: in this step the aim is the composition of cnfs built in step 1 with regards to the different paths elaborated in the second step to allow one to check the consistency in each of the possible execution plans. The composition is carried out by building constraint networks (cns) representing the temporal constraints of a scenario session following a path. Step 4: Temporal Consistency Check: this last step aims at checking the consistency of the cns derived in Step 3 for each execution plan. Then, the results of the consistency check allow the author (i) to verify the correctness of the execution plans and (ii) to submit queries related to the scenario to verify further properties. 14
In the following sections, we elaborate on the steps of the methodology and we illustrate them through a running example.
4.1 Step 1: Network Fragment Design This first step of the methodology aims at translating the tuples of the scenario into temporal networks, called constraint network fragments (cnfs). These cnfs represent the temporal constraints of each tuple through bounds of differences in order to check the temporal consistency of the scenario. A preliminary check of consistency will also be discussed at the end of this step.
4.1.1 Building the cnfs The cnfs represent the constraints among events (tuple events and TAC events) in the tuple. As the check is based on conjunctions of bounds of differences, if there are alternatives in the tuple (for instance complex start/stop events, such as e1 OR e2),
different cnfs have to be created, in order to check the consistency on each
possible alternative. Let NT= be the cnf, where: •
N
is the set of the nodes of the network. A node represents a start/stop event or
a TAC event. TAC events are taken from the Action list. •
A
is the set of labeled arcs between the nodes of the network. An arc
represents a temporal constraint between two events. It can, for example, link the TAC start event and the TAC stop event of a time dependent object (for instance a video clip). On each arc there is a label of the form (x, y) where x,y are the minimum and the maximum temporal duration values between the connected nodes. For instance, the duration associated with an arc linking the start and stop events of a time dependent object is 0-ObjectDuration.
Thus for each tuple we create a cnf starting with the start event and continuing with the TAC events. After that we have to connect the stop events to the actions that have to be stopped. As mentioned before the start/stop events may be composite [Vazirgiannis and Boll 97]. In this case they have to be translated into cnfs according to the following mappings: •
OR(e1, ..., en): In this case, n cnfs have to be created, each of them representing one alternative of the disjunction.
15
•
ANY(k, e1, ..., en): Each possible subset of k events from the n given in the clause have to be represented in different cnfs. There are
n k
different cnfs resulting. Durations on arcs are unknown, so labels are of the form (0-). •
SEQ(e1, ..., en): The sequence of events has to be represented by a path from e1 to en, with unknown durations (0-) on their arcs.
•
TIMES(n, e1): In this case, n occurrences of e1 (with different names for each occurrence to distinguish them) have to be defined, all linked to a same node representing the composite event. Durations on the arcs are unknown (0-).
•
SIMULTANEOUSLY(e1,e2): In this case, e1 and e2 are starting points for arcs ending on a common node representing the composed event. Arcs have 00
durations.
The mapping of the complex events to cnfs are summarized in Fig. 3, where the tuple is represented as (Start event, Stop event, Actions List, Synchronization events) and different composite starting events are considered. Tuple 1, in which action a is executed, is triggered by event e4 or e5; Tuple 2 is triggered by two of the events e1, e2, e3; Tuple 3 is triggered by the event sequence e1, e2; Tuple 4 is triggered by three consecutive occurrences of e1; and Tuple 5 is triggered by simultaneous occurrence of e1 and e2.
Previously, we introduced the symbols “θ” and “$”. Their mapping into cnfs is performed as follows: θ: This symbol refers to the temporal origin of the scenario. Usually, there is a StartApp event to indicate the start of the scenario. Therefore this symbol θ has to be replaced by an arc starting from the StartApp event. If the tuple contains the StartApp event, the arc can be drawn during this step of the process, otherwise it will be done later during the composition step. $: This symbol refers to the start of the tuple. Therefore it has to be replaced by an arc starting from the Start Event of the tuple.
16
Tuple 1 e4 e5
Tuple 3
(OR(e1,e2), -, a, -) 0-0 a Fragment 1 Fragment 2 0-0 a
Tuple 4
(SEQ(e1, e2), -, a, -) 00-0 e1 e2
a
Fragment 1 (TIMES(3,e1), -, a, -) e11 00-0 a e12 0e13 0-
Tuple 2 (ANY(2, e1, e2, e3), -, a, 0e1 0-0 a 0e2 Fragment 1 Fragment 2 Fragment 1 0e1 5 Tuple a 00-0 e3 (SIMULTANEOUSLY(e1, e2), -, a, -) 0e2 Fragment 3 0-0 e1 0-0 a 0a 0-0 0-0 e3 e2 Fragment 1 Figure 3: Mapping of the composite events to cnfs
Appendix A presents a sample IMD scenario. The scenario represents a simple application related to ancient Olympic Games. We use this example to illustrate the four steps of the methodology we propose.
An example of one of the constraint networks fragments corresponding to the Stage1
tuple of the “Olympic Games” example is given in Fig. 4. The tuple
follows: (StartApp, ANY(1; _DoubleClick; _KeyEsc; _IntroStop), TITLE> 0 INTRO> 3 IMG1>). As three alternatives are provided as stop events (_IntroStop,
_DoubleClick or _KeyEsc) three cnfs have to be created. In Fig. 4 we present the cnf built in the case that _IntroStop stops the execution of Stage1.
StartApp
0-0
TITLE> 0-0 INTRO> 3-3 IMG1>
0-
TITLE< 0-0
0-47
0-
INTRO< 0-0 IMG1
) and INTRO end 17
(INTRO and INTRO< nodes in Fig. 4 will be translated into the following set of inequalities: 0 ≤ (INTRO) ≤ 47 where 47 is the duration of INTRO. On the other hand, the duration on the arc between TAC TITLE> and event TITLE< nodes has no maximum bound, as it is a text object and thus it is not time dependent. This arc is translated into the following set of inequalities: 0 ≤ (TITLE) ≤ ∞
4.1.3 Preliminary Consistency Check It is already possible to do a preliminary consistency check on cnfs for two reasons: •
to detect as soon as possible inconsistencies: some are already present in the cnf and it is useless to compose them together following a path (the global network will be inconsistent).
•
to better locate the inconsistencies: it is easier to find an inconsistency and to correct it at the cnf level than on a whole network representing a whole scenario session (therefore bigger and more complex).
This preliminary check verifies the constraints presented in Table 1 and Table 2. We distinguish two cases: qualitative constraints: the consistency of the action sequences is checked (Table1). For instance sequences of the form : A!tA! may cause an error, if another event from another tuple (such as A>) does not rectify this error sequence. Therefore, if this situation occurs, it is detected as a warning while building the cnf. Another situation that may cause errors is when the same event appears in the start event of the tuple as well as in its action list. Such a tuple is: (A>,_, A>5A!,_). 18
The result is that there will be an endless loop since the action A> starts the tuple recursively: a warning is raised. quantitative constraints: legal sequences of TAC actions may cause errors if the temporal intervals between them exceed specific limits. This set of constraints on bounds of differences (bods) has to be checked for each of the objects involved in the tuple. If they are not satisfied, a warning is raised. The constraints appear in Table 2 and are translated as follows: If ∃ bod = x ≤ (A! - A> ) ≤ y, then x≥0, y≤d (lines 2 & 5 of Table 2) If ∃ bod= x ≤ (A|| - A>) ≤ y, then x≥0, y≤d (lines 3 & 9 of Table 2) If ∃ bod= x ≤ (A! - A|| ) ≤ y, then x≥0 (lines 7 & 10 of Table 2) If ∃ bod= x ≤ (A! - A|>) ≤ y, then x≥0, y≤r (lines 14 & 8 of Table 2) If ∃ bod= x ≤ (A|> - A|| ) ≤ y, then x≥0 (lines 12 & 15 of Table 2) Such kinds of inconsistencies are raised as warnings, because they may not appear anymore when the tuple under consideration is combined with other tuples in the scenario (for a complete check including these constraints see Section 5).
This check is performed on the minimal network obtained through temporal reasoning, from the constraint networks (i.e. their corresponding sets of bounds of differences). It shows the possible temporal relationships between represented events which satisfy the constraints. For instance, the minimal network corresponding to the fragment shown in Fig. 4 constraints the duration of TITLE actor to be the same as INTRO video clip.
At the end of this step, each tuple of the scenario is translated into one or more cnfs, and thus a preliminary consistency check has been carried out on each of them
4.2 Step 2: Alternative Execution Plans The second step of the methodology, which is independent (i.e. it does not need to follow) of the first one, has the goal of identifying possible event sequences resulting in different orders in the tuple execution. The sequence of execution of 19
the different tuples of the scenario is driven by events. Depending on the sequence of occurring events, the different tuples of the scenario start and by this way define global states in the scenario. The goal of this step is to identify these different global states, and the events (transitions) leading from one state to another. The events belong to the ones appearing in the scenario. The result of this step is the set of all the possible execution plans (paths) of the scenario. For this purpose, an automaton is created, representing the different states of the scenario while the events allow transitions between states. Then, with regards to this automaton, we can already carry out some preliminary checks (that result in potential errors, indicated as warnings), which are discussed after the presentation of the way to build the automaton.
4.2.1 Automaton Building A state is defined as a set of executing (active) scenario tuples. A state change occurs when a new tuple is started. All the start events (Start event, Synch events) have to appear on at least one transition. A transition is caused by at least one of the start events. The resulting state-transition diagram must at least contain the Idle state (no tuples are running), from which all the paths start, through the StartApp event. If there are stop events that do not lead to a new state, then they have to be written on a transition ending in the pre-defined state called undefined state. In Fig. 5, the execution plans diagram for the example presented in Appendix A is depicted.
Each path starting from the Idle state ending to the Idle state represents an execution plan of the scenario. From Fig. 5 , we deduce two alternative paths :
P1: StartApp - _DoubleClick/_KeyEsc. In this path the presentation starts with the occurrence of StartApp event, which leads the scenario to State 1. The next event is _DoubleClick or _KeyEsc, which leads the scenario to Undefined state. P2: StartApp - _IntroStop - _Timer10 - e1/_AppTimerEvent. StartApp event is raised first, leading the application to State 1, followed by _IntroStop, leading the scenario to State 2. Then _Timer10 event starts Stage3. When user presses EXITBTN, _ExitEvent occurs stopping Stage2 and triggering e1 synchronization
event. This leads to State4. Alternatively, if the user does not press EXITBTN until 20
time 01 minute and 30 seconds, _AppTimerEvent occurs also leading anyway the application to State 4. Then the ExitApplication event is raised and the application goes back to the Idle state.
ExitApplication
State 1: Stage1 running ApplTimer running
Idle StartApp
State 1
State 2: Stage2 running ApplTimer running
_DoubleClick\_KeyEsc
State 3: Stage2 running Stage3 running ApplTimer running
_IntroStop Undefined State 2
State 4: Stage3 running ExitTuple running ApplTimer running
_Timer10
State 3
e1/_AppTimerEvent State 4
Figure 5: Sample execution plans diagram
4.2.2 Preliminary Consistency Check At this step of the process, consistency checks can already be done. The forthcoming step of the methodology consists in composing the sets of bounds of differences defined in step 1, with regards to the different paths elaborated through the second step of the methodology, to know if such paths are consistent or not.
4.3 Step 3: Composition of cnfs The goal of this step is to group the different cnfs with regard to each path, to check its consistency. Composition is done in three steps:
21
•
Merging the sets of bounds of differences corresponding to the different fragments under consideration,
•
Adding additional information if necessary,
•
Adding constraints to express the order in the occurrence of events.
The composition of different cnfs to obtain a cn is done by following a path (an execution plan) of the scenario. Therefore, for each path deduced from step 2, we compose the cnfs representing the tuples involved in the path, to get the cn corresponding to a whole scenario execution plan (path).
4.3.1 Building cnfs When there are alternative cnfs corresponding to a tuple, each of the alternatives has to be composed with the cnfs representing the other connected tuples. Each possibility of composition between the alternative cnfs of each tuple of the scenario under consideration leads to a constraint network (cn): we must take into account the various instantiations of the (event_in, event_out) pairs, where event_in is an event leading to the state in which this tuple is active, and event_out is an event leading out of that state.
In the sequel, we present the composition of the cnfs corresponding to path
P2
of
the sample scenario. All the tuples are involved in the path. We must find which cnfs should be involved in the cn representing path P2. There is only one cnf corresponding to each of the tuples Stage2 and Stage3, according to the rules presented in section 4.1.1. On the other hand, there are three different cnfs corresponding to tuple Stage1, one for each alternative stop event. From the definition of path P2, Stage1 is stopped by the event _IntroStop. This means that the cnf having _IntroStop as stop event (depicted in Fig. 4) is involved in the composition. The ExitTuple is mapped into two cnfs, one for each of the two different start events (e1 and _AppTimerEvent). Both of them are involved in the composition. Hence, two alternative cns represent the composition of the cnfs involved in path P2. Fig. 6 depicts the first cn, in which we included the cnf having e1 as start event for tuple ExitTuple.
22
The tuples have been linked together in the following way: Tuple Stage1 is stopped by the _IntroStop event, which also starts tuple Stage2. Event _Timer10 starts tuple Stage3. The sequence of _Timer10 and _ExitEvent events stops tuple Stage2 and generates synchronization e1 event which starts the last tuple of the path, ExitTuple.
Then, according to step ii, arcs have to be added, for each cn, between events generated by TAC actions dealing with the same object. If the object is timedependent, the arc duration has to fulfill the rules presented in Table 2. If the object is not time-dependent, the maximal bound of the arc duration has to be unbound. In Fig. 7, three additional arcs have been drawn (shown with slashed bold lines). The three arcs connect start and stop actions performed on YMNOS1, YMNOS2 and KAVSOUND media objects. The duration on the first two arcs
(labeled as arc 1 and arc 2) has unknown maximum bound, while the arc representing the constraint between start and stop actions performed on KAVSOUND actor (arc 3) is labeled by duration 0-30.
Then, the events starting the different tuples have to be linked together through arcs indicating their sequence of occurrence (following the path under consideration). Since the occurrence time of interactions is not predefined, the duration on these arcs have unknown maximum bound (0-). In Fig. 7, two additional arcs have been drawn (shown with dotted bold lines). The first arc (labeled as arc 4) connects start action on actor EXITBTN with _ExitEvent. The second arc (arc 5) connects start action on timer _TIMER1 with event _Timer10. The duration on the first arc has unknown maximum bound while the second arc is labeled by duration 10-10, indicating that the event _Timer10 will occur 10 seconds after the initiation of timer TIMER1.
23
Stage1 StartApp
0-0
0-
TITLE> 0-0
TITLE< 0-0
0-47
INTRO> 3-3
0-
IMG1>
INTRO< 0-0 IMG1
0-20 0-0 YMNOS1> 0-0 TIMER1> 6-6 EXITBTN>
KAVALAR< 0-0
0-
YMNOS1
0-
EXITBTN
INTRO< 0-0 IMG1
0-20 0-0
KAVALAR< 0-0
0-
YMNOS1> 0-0
YMNOS1
6-6
TIMER1
0-
EXITBTN
2 0-0
e1 0-0 0-30 KAVSOUND! A r c 3 0-0
0-
K A V S O U N D > 0-30 0-0
EXITBTN” is not present: one cannot pause or resume an object which has not been started. Similarly, “A|>” can not appear without “A||”. These constraints are fulfilled through a set of constraints for each object involved in the cn. Some of these are: x, y such that x ≤ (A! – A>) ≤ y x, y such that x ≤ (A|| – A>) ≤ y x, y such that x ≤ (A|> – A>)≤ y x, y such that x ≤ (A|| – A|>) ≤ y
where x, y are positive values for bod as defined above. These checks have to respect also the state transition diagram of the object temporal state (Fig. 1). If such constraints are not fulfilled, an error is raised.
Although these constraints have been checked at the tuple level (cnf), they have to be evaluated again at the scenario level (cn), since it is not possible to check them fully at the tuple level, as a given object can be, for example, started in a tuple and stopped in another one. Therefore, such inconsistencies were detected as warnings at tuple level and are detected as errors at scenario (path) level. It is important to note that the satisfaction of the previous constraints is not sufficient but necessary to insure the consistency of the scenario.
At this point of the methodology, we have different sets of bounds of differences, each of them corresponding to an alternative in a possible path of the scenario. Preliminary consistency checks have been performed at the cnf and cn level. The global consistency check can now be performed. 26
4.4 Step 4: Temporal Consistency Check The consistency check is performed according to the STP framework [Dechter et al. 91] for each of the cns derived by the cnf composition step described above, which takes place for each alternative path. During the consistency check, for each cn, the corresponding minimal network [Dechter et al. 91] is derived. From the consistency check, we can find paths considered as consistent (without errors) or inconsistent. Inconsistent paths have to be corrected to avoid their occurrence at runtime.
The consistency check provides also constraints on unbound variables (unknown durations on arcs). It can for example derive the maximum duration necessary between two events to assume the consistency of the scenario for a given path. Therefore, the scenario can then be enriched with this set of new constraints in order to improve its robustness.
In the example under consideration, we built four cns. We presented the two alternative cns corresponding to path P2 in the previous section. The remaining two cns represent the temporal constraints for path P1, which leads to the undefined state as shown in section 4.2.
After performing the consistency check to the first of the two cns for path P2 (presented in Fig. 7) we conclude that the network has no negative cycles (a cycle composed of nodes with arcs adding up to negative bounds). This means that the path is consistent [Dechter & al. 91]. Table 3 presents a portion of the minimal network that step 4 produces for the cn presented in Fig. 7. For each pair of events ei and ej, the cell MN[i, j] in Table 3 contains the minimum and the maximum time distance between the occurrences of the two events. For example event _ExitEvent must occur more than 2 seconds and less than 30 seconds after the
occurrence of the event _Timer10 (the cell MN[4, 2] in Table 3 contains the value [2, 30]). As we can see there are no negative cycles: Every diagonal cell (MN[i, i]) contains the value [0, 0].
27
Timer1>
Timer10
_e1
_ExitEvent
KAVSOUND!
YMNOS2!
Timer1>
[0, 0]
[10, 10]
[12, 40]
[12, 40]
[12, 40]
[12, 40]
_Timer10
[-10, -10]
[0, 0]
[2, 30]
[2, 30]
[2, 30]
[2, 30]
e1
[-40, 12]
[-30, -2]
[0, 0]
[0, 0]
[0, 0]
[0, 0]
_ExitEvent
[-40, -12]
[-30, -2]
[0, 0]
[0, 0]
[0, 0]
[0, 0]
KAVSOUND!
[-40, -12]
[-30, -2]
[0, 0]
[0, 0]
[0, 0]
[0, 0]
YMNOS2!
[-40, -12]
[-30, -2]
[0, 0]
[0, 0]
[0, 0]
[0, 0]
Table 3: A portion of a consistent Minimal Network (MN)
The second constraint network (in which ExitTuple starts with _AppTimerEvent) corresponding to path P2 produces a network with negative cycles. For instance, the diagonal cell for the KAVSOUND! event has the value [-3, -3], which implies that the sounds stops before it starts, thus the corresponding constraint network is found to be inconsistent [Dechter et al. 91].
4.5. Advanced features In this section, we provide some advanced features of the methodology presented previously. First, we address the issue of partial execution of paths in scenarios and we discuss the problem of operators appearing several times in the tuples of the scenario. Then, we explain how to take into account complex events using the composition operators S_CON, IN, NOT while checking the consistency of scenarios.
4.5.1 Partial execution of paths In an IMD session, we distinguish events which occur definitely (for instance raised by time-dependent media), from events that occur conditionally (e.g. raised by user-interaction). If a path includes events from the latter category, the path has also to be checked for consistency in the case this kind of event does not occur.
In addition, when a stop event is used in a tuple and when this stop event is raised by user interaction, its occurring time is unknown. It may be wrong to directly link it to a TAC stop event, especially if the stopped object is a time-dependent 28
one (it has to be stopped only if it is still in active state). Depending on the occurrence time the stop event, objects could be idle or active and therefore could be stopped or not. Therefore, an additional arc and an additional node between the TAC stop event and the arc starting from the stop event have to be provided, with a variable duration depending on the time the stop event is raised. As the network is different with regards to the time of occurrence of the stop event, the duration between the start of the tuple and the stop event (the duration of the tuple) is also represented through a variable called "d". In Fig. 8, depicting the second cnf for Stage1, an additional node has been created in-between the stop event of the tuple (_KeyEsc) and the TAC end event of INTRO audio clip (INTRO 0-0 INTRO> 3-3 IMG1>
0-
TITLE< 0-0
0-47
INTRO
3 IMG1>. It produces two alternatives depending on the value of d. In the former the value of d is between 0 and 3 seconds (i.e. IMG1 is not yet shown), while in the latter situation d is greater than 3 seconds. In both situations INTRO audio clip is still running when _KeyEsc event is raised. We have already presented the cnf for the second alternative (Fig. 8). Fig. 9 shows the cnf when d is varying between 0 and 3 seconds. As we can see, IMG1 actor has not yet been presented and therefore should not be stopped.
29
(d) StartApp
0-0
TITLE> 0-0 INTRO>
0-
TITLE< 0-0
0-47
INTRO
