Using a Formal Method to Verify the Temporal

Using a Formal Method to Verify the Temporal Semantics of SMIL Documents P.N.M. Sampaio, C.A.S. Santos, J.P. Courtiat LAAS – CNRS 7 Av. du Colonel Roche 31400 Toulouse – France Tel: (33) 5.61.33.62.44 {psampaio, saibel, courtiat}@laas.fr Abstract Several works have been dedicated to the authoring and presentation of interactive multimedia applications to be distributed over the web, but few of them address semantic verification issues of these documents. This paper presents a formal approach for the design and semantic verification of SMIL documents which relies on the classical reachability analysis performed on RT-LOTOS specifications and on the utilization of a simple scheduling graph. Furthermore, some erroneous semantic interpretations of SMIL documents which are not conformant with the reference behavior expressed by the scheduling graph are illustrated using some currently available SMIL players.

Keywords: Formal Methods, LOTOS, RT-LOTOS, Interactive Multimedia Documents, SMIL 1

INTRODUCTION

Several publications have addressed the development of Interactive Multimedia Documents (IMDs) based on the W3C standard Synchronized Multimedia Integration Language (SMIL) [1]. Many of these works address the authoring of SMIL documents [2, 3, 4] and the implementation of SMIL players [5, 6, 7, 8]. These works focus on the specification of authoring requirements and synchronization constraints, and few of them address semantic verification issues. This paper presents the continuation of the approach previously introduced in [9, 10]. It proposes a methodology for the design of IMDs based on the formal description technique RT-LOTOS [11], a temporal extension of the LOTOS standard [12]. This methodology has been implemented for the verification of consistency properties of the Nested Context Model (NCM) [13], and can easily be applied to other high level authoring models. This work addresses the application of this

methodology for the XML-based DTD SMIL. An XML parser provides the correct syntactical verification of the author’s documents. However, the semantic correctness of its presentation is not always ensured. That means that the author’s synchronization requirements for the presentation of his document may not be always completely satisfied. The application of the above methodology provides a complete semantic verification framework for IMDs relying on the concept of temporal consistency.

2

USING RT-LOTOS FOR THE FORMAL VERIFICATION OF INTERACTIVE MULTIMEDIA DOCUMENTS

The proposed methodology aims to provide a framework for the design (specification, verification and presentation) of complex Interactive Multimedia Documents which relies on the Formal Description Technique RT-LOTOS and its associated verification/simulation tool RTL, developed at LAAS-CNRS [11,14]. Using this formal approach for the design of IMD's has three main advantages. First, it provides a formal semantics to the high-level authoring model, describing without any ambiguity the behavior of the document during its presentation. Second, it enables us to check consistency properties on the formal specification derived from the authoring model, using standard verification techniques. Finally, it provides a consistent and operational representation for the scheduling of the document. The methodology applied for the formal design of IMD's is illustrated in Figure 1. This methodology provides a high flexibility for the author during the edition of the IMD since he is able to describe his document using the authoring model of his preference, such as SMIL [1], NCM [13], etc. Then, the logical and temporal structure of the document can be automatically translated into an RT-LOTOS specification which describes completely the semantics of the document. It is

important to emphasize that the RT-LOTOS specification is, then, kept totally hidden to the author during the specification and verification phases.

High-Level Document Authoring (SMIL, NCM Model, etc.)

The utilization of this methodology is illustrated in the next sections and, furthermore, it is also applied to the semantic verification of SMIL documents.

3

Automatic Translation into an RT-LOTOS Formal Specification no

Derivation of the Minimal Reachability Graph

Is the document Consistent?

Analysis of Consistency Properties

yes Scheduling Graph

Figure 1 -

occurrence of non-controllable events, such as user interactions.

Formal design methodology of IMDs

Once the RT-LOTOS specification for the respective IMD is available, a minimal reachability graph can be obtained as a result to some verification techniques developed and implemented for RT-LOTOS within the RTL software tool. Later on, based on the reachability graph, we are able to verify the temporal consistency of the document. Thus, some reachability properties can be determined, such as internal and extrinsic consistency [9], in order to ensure that all the temporal constraints associated with the components of the document are fulfilled during its presentation. Furthermore, aggregation techniques can also be applied in order to avoid the state space explosion problem that may come up with the utilization of labeled transition systems [15]. If all the temporal constraints of the document can be fulfilled (if the document is consistent), then we are able to perform the scheduling of its presentation. The scheduling is, then, accomplished based on an appropriate representation (scheduling graph) which is obtained from the reachability graph. The scheduling graph is simple and operational enough and still provides the controllability of the document during its presentation. In opposite, if the document is still inconsistent after the reachability analysis, its highlevel description must be revisited. For this purpose, the reachability analysis provides a feedback for the author proposing valid solutions for the presentation of the document, in particular, with respect to the

CONSISTENCY INTERACTIVE DOCUMENTS

ANALYSIS OF MULTIMEDIA

The formal verification of consistency of IMD’s using RT-LOTOS has been previously addressed in [9, 10, 16]. In [16], a document was considered as consistent if the action characterizing the start of the document presentation is necessarily followed (some time later) by an action characterizing the end of the presentation. This definition was revisited in [10] in order to make a clear distinction between two kinds of events that may lead to temporal inconsistencies, namely: Internal non-deterministic events which are related to the flexibility of media presentation duration (themselves related to admissible QoS adjustments for the media), as well as to incomplete timing constraints, and; External non-deterministic events which are related to the occurrence of external events, such as user interactions on anchors, network delays and processing results from database queries, scientific simulations, and so on. Temporal inconsistencies may be the consequence of either internal or external non-determinism, or even both. Basically, the temporal consistency of a document can be determined by the identification of the inconsistency sources of a temporal scenario and, then, checking whether they can be handled by a temporal formatter. A [0,+∞]

B [10,20]

start

end

C [0,40]

Figure 2 -

Multimedia Scenario

To illustrate the utilization of this methodology, let us consider Figure 2 which depicts a multimedia scenario that describes the simultaneous presentation of a sequence of two media objects (A and B) with the interactive media object C. The presentation duration of media objects A, B and C are, respectively, [0,+∞], [10,20] and [0,40] seconds. As synchronization constraints, the presentation of media objects A and C must start together, and presentation of media objects B and C must terminate together. It is important to note that the

The latter concept makes the previous scenario inconsistent since there are some inconsistent branches on its reachability graph that are generated by the occurrence of an external non-deterministic event.

presentation of this scenario can be interrupted at any moment if there is a user interaction on object C. In this example, observe that a valid temporal interval must be determined for the occurrence of either the user interaction and the duration of media object A so that the synchronization constraints of the scenario are satisfied during its presentation.

The reachability graph is representative enough for the verification of consistency properties, as presented in [17]. Although, for scheduling purposes, an operational and simple scheduling graph can be obtained from a consistent reachability graph (all the branches lead to the occurrence of the action end). For this reason, when a potential inconsistent branch is generated by the occurrence an internal non-deterministic event, this branch must be cut out of this graph (since this event is controllable). Similarly, when a potential inconsistent branch is generated by the occurrence an external non-deterministic event, this branch can also be cut out of this graph in order to determine a valid temporal interval for the occurrence of this event. The resultant consistent reachability graph is illustrated in Figure 4. In this sense, a scheduling graph is obtained from the consistent reachability graph, called a Time Labeled Automaton (TLA in short) and has been formalized in [18].

Once the temporal and logical behavior of this document is translated into RT-LOTOS, we are able to perform the reachability analysis and, then, to verify its temporal consistency. The reachability graph for the previous scenario is illustrated in Figure 3. Note that, on this reachability graph all the branches that lead to the configuration 7-(), also lead to occurrence of the action end of the document’s presentation. Although, potential inconsistent branches (those branches that do not lead to the action end) still occur in this reachability graph. These branches are associated either with the occurrence of internal (e.g., time progression), and/or external (e.g., user interaction) nondeterministic events, since they occur outside the valid temporal limit that satisfies the synchronization constraints of the scenario.

The TLA turns straightforward the semantic verification of an IMD’s presentation since it describes the correct reference behavior for the scheduling of this document. In this sense, a document’s presentation is semantically consistent if its resultant behavior is in conformance with its associated scheduling graph.

If a potential inconsistent branch is generated by the occurrence of an internal non-deterministic event, this inconsistency can be handled by the presentation system. However, if this branch is generated by the occurrence of an external nondeterministic event, it can not be ignored by the system to avoid an inconsistency situation (since the occurrence of this event is not controllable) [10].

0-() i(start) 1-() i(sAC) C={1} 2-(0 0 0) t 2-(20 20 20) t i(eA_sB) C={1} theta={(2,2),(3,3)}

2-(20.5 20.5 20.5)

i(eA_sB) C={1} theta={(2,2),(3,3)}

user C={2} theta={(1,1)}

t

i(eA_sB) C={1} theta={(2,2),(3,3)}

user C={2} theta={(1,1)}

2-(30 30 30)

user C={2} theta={(1,1)}

t

3-(0 20 20)

3-(0 20.5 20.5)

t

user C={2} theta={(1,1)}

i(eBC) theta={(3,1)}

user C={2} theta={(1,1)} i(eBC) theta={(3,1)}

t

3-(20 40 40)

t

t

8-(40)

3-(10 40 40)

9-(0)

i(end)

t

3-(20 20 20)

3-(0 30.5 30.5)

i(eA_sB) C={1} theta={(2,2),(3,3)}

3-(20.5 61 61)

user C={2} theta={(1,1)} user C={2} theta={(1,1)}

i(eBC)

2-(1000 1000 1000)

user C={2} theta={(1,1)}

5-(10 0)

5-(20 0)

4-(0 0)

t

user C={2} theta={(1,1)}

t

i(eA_sB) C={1} theta={(2,2),(3,3)}

i(eA_sB) C={1} theta={(2,2)}

user C={2} theta={(1,1)}

t

6-()

t

4-(1000 1000)

i(eA_sB) C={1} theta={(2,2)}

5-(0.5 0.5)

t 5-(20.5 20.5)

Figure 3 -

4-(0.5 0.5)

t

5-(0 0)

i(eBC)

i(end) 7-()

2-(40.5 40.5 40.5)

user C={2} theta={(1,1)}

t

user C={2} theta={(1,1)}

user C={2} theta={(1,1)}

t

i(eA_sB) C={1} theta={(2,2),(3,3)}

3-(0 40.5 40.5)

t

i(end)

3-(10 10 10)

user C={2} theta={(1,1)}

t

3-(20.5 20.5 20.5)

user C={1}

3-(0 30 30)

t

user C={2} theta={(1,1)}

2-(30.5 30.5 30.5)

t

3-(10 30.5 30.5)

t

user C={2} theta={(1,1)}

3-(0 0 0)

t

3-(10 30 30)

user C={2} theta={(1,1)}

i(eA_sB) C={1} theta={(2,2),(3,3)}

Reachability graph for the previous scenario

i(eA_sB) C={1} theta={(2,2)}

presentation of the scenario takes place at t=40 seconds. For this reason, the firing window (condition W) for the occurrence of eBC is compensated (t3=40-t2) considering the time elapsed for the occurrence of eA_sB, that is, the enabling condition 20≤t2≤30. Finally, both actions user and eBC lead to the occurrence of the action end of presentation. As we can note, the TLA provides the control of the occurrence of user interaction within a valid temporal interval so that it can respect the synchronization constraints of the scenario.

0-() i(start) 1-() i(sAC) C={1}

2-(0 0 0) t 2-(20 20 20)

i(eA_sB) C={1} theta={(2,2),(3,3)}

t

i(eA_sB) C={1} theta={(2,2),(3,3)}

2-(20.5 20.5 20.5)

i(eA_sB) C={1} theta={(2,2),(3,3)}

t

2-(30 30 30)

3-(0 20.5 20.5)

i(eA_sB) C={1} theta={(2,2),(3,3)}

t

3-(0 30 30)

t

3-(10 40 40)

t

3-(10 30.5 30.5)

0

t user C={2} theta={(1,1)}

3-(20 20 20)

user C={2} theta={(1,1)}

8-(40)

user C={2} theta={(1,1)}

t0=0 i(start)

user C={2} theta={(1,1)}

1

user C={2} theta={(1,1)}

5-(20 0)

user C={1}

i(end)

3-(10 10 10)

t

3-(20 40 40)

user C={2} theta={(1,1)}

i(eBC) theta={(3,1)}

3-(0 0 0)

t

3-(10 30 30)

t

i(eBC) theta={(3,1)}

3-(0 20 20)

i(eBC)

t1=0 i(sAC)

5-(10 0)

i(eBC)

2

9-(0)

6-()

i(end)

0