Integrity Constraints for Interactive Multimedia

A119

Integrity Constraints for Interactive Multimedia Scenarios I. Mirbel, B. Pernici

M. Vazirgiannis

Politecnico di Milano, Piazza L. Da Vinci, 32 I-20133 Milano ITALY

Dept of Informatics, Athens University of Economics & Bussiness, Patision 76, 10434, Athens, HELLAS

Abstract Checking the integrity of multimedia scenarios needs to take into account both the characteristics of preorchestrated scenarios and scenarios with interactions from users, the sequence and time of occurrence of which is not defined when the scenario is designed. The paper proposes a methodology for checking the integrity of Interactive Multimedia Document (IMD) scenarios at authoring time at various levels. The IMD flow is mainly defined by the events occurring during the IMD session. Integrity checking is based temporal reasoning techniques and consists of a set of discrete steps, during which we transform the scenario into temporal constraint networks representing the constraints linking the different possible events in the scenario. Temporal constraint verification techniques are applied to verify the integrity of the scenario, deriving a minimal network showing possible temporal relationships between events given a set of constraints. A set of query categories is also defined to examine the properties of a scenario according to the possible temporal relationships: such queries can allow the refinement of a scenario, the verification of given properties, and support scenario revisions.

1. Introduction An Interactive Multimedia Document (IMD) involves a variety of individual multimedia objects presented according to the IMD scenario. The multimedia objects that participate in the IMD are transformed, either spatially or temporally, in order to be presented according to author’s requirements [Vaz96]. Moreover, the author has to define the spatial and temporal order of objects within the application context and to define the relationships among them. Finally, the way that the user will interact with the application as well as the way that the application will treat application or system events have to be defined. The related application domains are quite challenging and demanding. Among others, these can be interactive TV, digital movies, and virtual reality applications. In the framework of IMDs we consider the following as cornerstone entities: • Events: they are the fundamental entities of interaction in the context of the IMD, raised by user actions, or by objects participating in the IMD or by the system. They may be simple (i.e. not decomposable in the IMD context) or complex and have attached their spatiotemporal signature (i.e. the space and the time they occurred). For more details refer to [VB97]. • Spatiotemporal Composition: it is an essential part of an IMD and represents the spatial and temporal ordering of media objects in the corresponding domain. The issue of spatial and temporal relationships among the objects is critical [VTS98]. • Scenario: it stands for the integrated behavioral contents of the IMD, i.e. which events the IMD will consume and what presentation actions will be triggered as a result. In our approach [VTS98], the scenario consists of a set of self-standing functional units (scenario tuples) that include: triggering events (for start and stop), presentation actions (in terms of spatio temporal compositions) to be carried out in the context of the scenario tuple, and related synchronization events. Scenario Rendering is the process of defining an execution plan for the IMD scenario, i.e. when, where, for how long and under what transformations each media object will be presented. This task is rather straightforward in “pre-orchestrated” applications [KT96], while in the case of IMDs with rich interaction it becomes more complicated [VTSM98]. Real world large IMDs can be very complex with respect to (i) the number of involved objects, (ii) transformations of the objects in the scope of an application, and (iii) relationships among them. We consider a IMD as a container that includes objects that are transformed and interrelated for the purpose of the application. It is obvious that in a complex IMD it may be very difficult to describe all possible functionality and paths the user or the application may follow. Therefore, one can think of a IMD as an event-based environment in which there is a rich set of events that may occur and define its flow. For instance, the end of a video clip, the spatial coincidence of two objects in the application window, the occurrence of a pattern in a media object are events that may be exploited to trigger other actions in an application. An interesting issue is the temporal consistency of the presentation, that is to be able to decide in advance, i.e. before the scenario execution, if the order of presentation actions is consistent according a set of constraints. These constraints may be

generic (i.e. “an idle object cannot be stopped”) or application specific (i.e. “video clip A should not overlap with sound B”). The topic of scenario integrity is crucial when designing a complex scenario. Moreover the multitude of events and the related interactions that may occur in an IMD session may create a lot of potential IMD evolution paths. Therefore there is the necessity to provide the IMD authors with tools to aid them in authoring consistent scenarios and also to verify the integrity of existing ones. In this research effort we elaborate on a methodology for checking the integrity of an IMD scenario at authoring time combining an existing model for such scenarios [VTS98] (section 2.1) and techniques from temporal reasoning [BCTP97] (section 2.2). The integrity checking (section 3) consists of a set of discrete steps we transform them into temporal network fragments that correspond to the fundamental functional units of the scenario model mentioned above. The IMD flow is mainly defined by the events occurring during the IMD session. Thus there are many alternative execution paths that an IMD session might follow. According to the scenario we find all the possible paths (step 1) and for each of them we create an integrated temporal network resulting from merging the corresponding fragments (steps 2 and 3). Then exploiting temporal constraint verification techniques we check the integrity of the scenario and we derive the minimal constraint network (step 4). Also, in section 4, we propose a set of query categories that may process using the same methodology. In section 5, we demonstrate the potential of our approach by applying the methodology to a sample interactive scenario. In section 6, we present related work and compare it to our approach. In section 7 and last, we conclude by summarizing and presenting directions for further work.

2. Background In this section we present briefly the scenario model that we adopt to serve as the basis of the temporal constraint checking scheme we aim at. 2.1. The scenario model A multimedia scenario is defined as the high level definition of a multimedia application functionality that covers the spatial and temporal synchronization of participating media objects and the handling of interaction between the user, the system and the application. We are going to exploit an existing model for Interactive Multimedia Documents [VTS98]. According to that model an IMD consists of the spatial and temporal composition of the participating media, the definition of interaction between the user and the IMD (external interaction), the interaction between internal IMD entities (internal interaction) and the specification of media synchronization in the temporal and spatial domains. The fundamental utility of multimedia objects (actors) is to be presented according to the authors’ requirements. The primitive actions for presentation of time dependent media objects are defined as Temporal Access Control (TAC) Actions in [LG93]. Temporal & Spatial Relationships Hereafter we present a set of TAC operators that represent the temporal composition of media objects, together with the causality of temporal relationships among presentation intervals. For more details refer to [VTS98]. These operators correspond to the well known TAC actions: start (>), stop(!), pause (||), resume (|>). Therefore: TAC_operation :== “>” | “!” | “||” | “|>” For instance, assume A is a multimedia object, then A> represents the start of the multimedia instance, A! the stop, A|| the pause and A|> the resume actions. We have to illustrate the events arising from the temporal state changes of a multimedia object. i.e. when the object A starts its presentations then the “A>” temporal event is raised. Special attention should be paid to the event generated when the object finishes its execution (“” | “” | “||” Now we define a temporal operator to be used for temporal composition representation. Let A, B two multimedia objects, then the expression: A t_event t_instance TAC_action B, represents all the temporal relationships between the two multimedia instances, where t_instance corresponds to the length of a temporal interval. Therefore: 2

temporal_composition :== ( | object) [{temp_rel object}] temp_rel:==t_event t_instance TAC_action For instance: the expression: “A >4! B C” conveys “Start A, after 4 time units stop B, and just after start C”. Finally, we define the duration dA of a multimedia object A as the temporal interval between the temporal events A> and A” is applied to, it falls into the active state. All the allowed transitions between states and the corresponding actions appear in Figure 2.

! Idle < |> >

!

Active > |>

Suspended

||

Figure 2: State transition diagram for a time dependent media object In a multimedia scenario there may be a set of actions that are applied by the scenario expressions to a multimedia object as regards it state. The execution of these actions must obey to some constraints. For instance it is not possible to stop an object which is not active. In the following table we list the runtime constraints that emanate from the above diagram are depicted (i.e. which condition must be held in order to be able to apply an operator): OPERATOR > ! || |>

NAME Start Stop Pause Resume

CONSTRAINT None The object must be in ACTIVE or SUSPENDED state The object must be in ACTIVE state The object must be in SUSPENDED state

Table 1. The TAC operators and the related constraints It is important to stress that these constraints are applicable both at authoring and at runtime. 3.2 Constraints on one time-dependent object In a multimedia scenario a TAC operator may be applied to a time dependent object resulting in a change of state (e.g. from IDLE to ACTIVE). As the scenario evolves another TAC operator may be applied to the same object resulting in another change of its state. Not all the sequences of TAC actions on media objects are legal: we can accept only the ones that are represented in the state transition diagram in Figure 2. Lets assume the expression: A op t A op Where A is an actor, op is one of the operators: >, , and t the time elapsed between the two operations.

6

The first general constraint is that “t should be such that the time of execution of the second action is not before the start of the application”. It is important to stress the fact that some combinations are not feasible (“illegal”) and, among the rest of the combinations, some are feasible under some constraints. An exhaustive combination of TAC operations and their features appear in Table 2. Another assumption is that the constraints that are described in the previous sections apply also here, so for instance in the case of the expression (A>tA||) the fact that A should be active is checked by that kind of constraint. We make the assumption that t can also be less than 0. We assume that the expression is closed, i.e. no other action is applied to the actor in the evolution of the expression ACTION ACTION Legal > > Yes > ! Yes > || Yes > |> No ! > Yes ! ! No ! || Yes ! |> Yes || > Yes || ! Yes || || No || |> Yes |> > Yes |> ! Yes |> || Yes |> |> No d: duration of object presentation tr: remaining presentation time for an object active

Constraint 0

di: “Intro” duration dr: “Runs" duration

Intro


0-di

0-0 0-0 0-

2-2 IMG2-2>

IMG2-1< 2-2

0-

0-0 IMG2-2
1-1 Runs>

di: “Intro” duration dr: “Runs” duration

Intro


0-db

A

B

B
, -, -) composed with (OR(e4,e5), -, a>, -, -) e1

00-0 0-

0-

e1

0-0 a>

0-0 a>

e4

0-

e2

e5

0-0

e3

e1

Network 1 Network 2

00-0 0-

0-

e2

a>

e5

a>

0-

Network 3

0a> 0-0

e2

e4

00-

0-0

e3

e3

e4

0-0

e3

0-

0-0

0-0

e2 e1

Network 4 Network 5

a>

0-0 Network 6 e5

0-0

Figure 8. Composition of fragments into network. 3.3.4 Consistency check Consistency check is performed according to the STP framework [DMP91] for each of the networks derived by the network composition step described above, which takes place for each alternative path. During the consistency check, for each network the corresponding minimal network is derived, representing all possible instantiations of the scenario. Moreover, the resulting minimal networks are evaluated with regards to the object temporal state constraints presented in section 3.1 and section 3.2. From the consistency check, we can find paths considered as consistent or inconsistent. Inconsistent paths have to be corrected to avoid their occurrence at runtime. The check of the consistency also provides constraints on unbound variables (unknown durations on arcs). It can for example derive the maximum duration necessary between 2 events to assume the consistency of the scenario. Therefore, the scenario can then be enriched with regards to this set of new constraints in order to improve the robustness of the scenario: unbound durations on arcs can be replaced by the values found through the consistency check. Then, the possible execution plans (or the consistent paths) are proposed to the designer.

4. Query After the consistency check, the IMD author designer can query the set of constraints of the scenario. This can be helpful in order to: • Inform the user about the underlying temporal constraints of his/her scenario, • Modify the scenario and, in this way, create, correct or improve the scenario in an incremental way. In this last case, the author depending on the answer of the query can do the modification on the scenario, currently under creation. It is evident that each of the modifications of the scenario have to be followed by a new consistency check, in order to have a correct view of the scenario under design at any moment. The temporal constraint network constructed in the previous phase may be exploited to answer efficiently queries that authors might submit. The queries may be classified into the following categories: Temporal, which deal with temporal features of the presentation and they be further classified in • Point queries, where the author is interested to find a relationship between two events. For instance: “Does A start before B?”, “ Does end A after the start of B?” etc. • Relationship, in this category we are interested whether the temporal interval corresponding to the presentations of objects fulfill a required relationship [All83]. Such a query would be: “Are the presentations of objects A and B overlapping?”, “Is A presented before B?” etc. Conditional ones In this category we are trying to capture the higher lever consistency of the scenario according to author defined constraints and conditions that relate to temporal properties of the presented objects and their relationships. 13

•

Constraint: In this case the author must have defined a set of constraints related to temporal features of the presented objects. Such constraints could be: “video A must start before/simultaneously/after B”, “video B must be temporally shorter that music C”. Then he/she may check a temporal network corresponding to a scenario path against these constraints.

•

Event based: here we are interested in queries that are based on the assumption that a specific set of events in a specific order has occurred. This set of events would result in a path and thus in an integrated temporal network. Then the authors could submit queries from the aforementioned categories, which would give them information on properties of the IMD document given the events occurred. Such a query could be: “Will the presentation reach video A before video B given the sequence of events e1,e2,e3?”. Such queries are handled in the following way: given the event sequence (i.e. e1, e2, e3), the corresponding network is created and then it is evaluated against the requested condition (e.g. “startA before start B”).

Layout: in this kind of queries results in graphical representation of the queries of an author mainly depicting the temporal relationships among presented objects. This is a necessity that is recognized among the authoring community [VTS98]. Such a query would be: “Show the temporal layout of the IMD between the 2nd and 10th second of the presentation” or “Show the temporal layout of the IMD between the events e1 and e2”

5. Example In this section, the full example of the “Olympic Games” is presented, following the methodology illustrated in section 3. 5.1 Finding the possible paths in the scenario The automaton presenting the different states of the scenario has already been presented in Figure 3. Here is the list of the corresponding paths. P1: StartApp - DoubleClick/KeyEsc P2: StartApp - _IntroStop - _ACDSoundStop - NextBtnClick , e1 - Exit/AppTimer P3: StartApp - _IntroStop - _ACDSoundStop - Exit/AppTimer P4: StartApp - _IntroStop - AppTinmer P5: StartApp - _IntroStop - _TIMEINST – Exit/AppTimer P6: StartApp - AppTimer 5.2 Network-fragment design The fragment network corresponding to tuple1 is presented in Figure 4 and Figure 5. In Figure 9, the network fragment corresponding to the tuple2A is presented.

14

_IntroStop

0-0

IMG1_3>

0-

IMG1_3


0-10961

0-0 ACDDM


0-xxx

TIMER1


0-

INFOTXT


0-

NEXTBTN
EXITBTN< 0-0 e1

Figure 11: The second constraint network fragment corresponding to “Stage2B” tuple In Figure 12 and Figure 13, the two constraint-network fragments corresponding to the “stage3” tuple are presented.

15

_TIMEINST e1

0-

0-0

0FLUTE>

INFOTXT< 0-0 0-8734

FLUTE


0-

YMNOS


IMG1_1


0-

3-3 EXITBTN>

IMG3_1< 0-

EXITBTN


IMG3_1< 0-

EXITBTN
0-0

0Intro> 3-3 0IMG2-1> 2-2 IMG2-2> 2-2

0-

Intro< 0-0 0-0 IMG2-1< 2-2

IMG1-2> 01-1 Runs> Stage 2B

Stage1




0-

0-

IMG1_3
0NEXTBTN< _NextBtnClic 0- 0-0 0-0 k 0-0 0EXITBTN EXITBTN 0-0 0> < 0e1

INFOTXT< Stage 3

0-8734 FLUTE> FLUTE< 0-0 0YMNOS> YMNOS< 2-2 0IMG1_1> IMG1_1< 0-0 0IMG3_1> 3-3 EXITBTN> 00-0 ExitEvent

Exit stage

TIMER1