TOCPN: Interactive Temporal Model for Interactive Multimedia Documents Kyoungro Yoon
P. Bruce Berra
Dept. of EECS
Dept. of CSE
Syracuse University
Wright State University
Syracuse, NY 13244
Dayton, OH 45435
[email protected]
[email protected]
Abstract Recently, much attention has been paid to interactive temporal models which let authors de ne possible interactions with users and represent dynamic organization of hypermedia documents. The possibility of user interaction makes it very hard for authors to predict the course of presentation and to author a document without an authoring tool. The authoring tool then must be designed with a rich model representing the dynamic temporal structure of interactive multimedia documents. This temporal structure model should be able to specify the temporal composition of interactive documents and synchronization. We investigate possible interactions of users and design a new interactive temporal structure model based on the time interval model called Transitional Object Composition Petri Net (TOCPN). The novel features of TOCPN include capability of expressing dynamic structure of interactive multimedia document, providing a synchronization mechanism for various media with various temporal relations, supporting sharing and reusing of existing objects, and supporting composite objects as well as atomic objects. In this paper, we also de ne requirements of the authoring tool and helper tools, and provide a preliminary architecture of the authoring tool.
1. Introduction With the recent advances in software and hardware technology, multimedia documents have been paid a great deal of attention by researchers and scholars. Unlike conventional documents, multimediadocuments have multiple objects presented simultaneously and the objects presented tend to change with time. In addi-
tion, multimedia documents can provide various ways to interact with viewers changing the nature of the presented document [3] [11]. The possibility of user interaction makes it very hard for authors to predict the course of presentation and to author a document without a good authoring tool. A good authoring tool then must be designed with a good model representing the temporal structure of interactive multimedia documents. This temporal structure model should be able to specify the temporal composition of interactive documents and synchronization. Recently, much attention has been paid to temporal models which let authors de ne possible interactions with users and represent dynamic organization of hypervideo documents. Some of the temporal models for interactive multimedia documents include Hirzalla, Falchuk and Karmouch's time-line based model [3], Wahl, Wirag and Rothermel's TIEMPO [11], Rossum et. al.'s CMIFed [9], and Buchanan and Zellweger's Fire y [2]. However, these models have limitations in expressing interactive multimedia documents with multiple source/multiple destination hyperlinks such as interactive hypervideo documents, and time-line based models still have the complexity and limitations of conventional time-line models. In this paper, we investigate possible interactions of users and design a new interactive temporal structure model based on the time interval model called Transitional Object Composition Petri Net (TOCPN)[4] [5].
2. Requirements Requirements in temporal composition include temporal representation of documents, temporal access control, and support for temporal authoring of new documents [6]. The composition of hypermedia documents can be
divided into two major categories; loose composition and tight composition. In loosely composed hypermedia documents, each document has hyperlinks to other documents without any tight boundary for itself. In other words, the authors do not specify the limitation on how far the given document can be hyperlinked. Documents in this category can mainly be used for browsing or search purposes. On the other hand, a tightly composed hypermedia document has hyperlinks with a tight boundary between other multimedia data and itself. That is, documents in this category are associated with the limitation, speci ed by the author, on how far the document can be hyperlinked. Therefore, they are mainly used as presentation material with a speci c purpose, such as a multimedia kiosk. The main focus of the new interactive model is on the representation of tightly composed multimedia documents, but does not exclude the possibility of supporting loosely composed documents in addition to other requirements satis ed by OCPN model of Little et. al. While a presentation is in progress, a user can change the content of the presentation or the temporal structure of the presented document. Three possible types of such user interaction are changing the content of the presentation, selecting an alternative object, and presenting an additional object.
3. Presenting an Additional Object While a multimedia document is presented, a user can choose to view another document with detailed information such as a help document, pausing the original document, as shown in Figure 3.
1. Changing the Course of the Presentation A user action can change the course of presentation. For example, a presentation focused on subject A can be changed to another presentation focused on subject B, as shown in Figure 1, terminating the original presentation.
In addition to the requirements of the temporal composition model listed previously [6], an interactive temporal model should be able to represent these three types of user interactions eciently and clearly.
Original presentation Selected presentation
time
Figure 1. Changing the Course of the Presentation
2. Selecting an Alternative Object In presenting multiple multimedia objects, a user can select an object among multiple objects. For example, while presenting a general topic, i.e. the economy of Korea, composed of text, image, and video, a user can select the video clip of the Foreign Exchange Market of Korea among other objects, as shown in Figure 2.
TEXT AUDIO VIDEO 1 only one of three objects become active
VIDEO 2 VIDEO 3 time
Figure 2. Selecting an Alternative Object
Original Presenation Paused Temporarily Original Presentation Additional Presenation
time
Figure 3. Presenting an Additional Object
3. Related Work The synchronous hypertext petri net model of Trellis system [8] [7] is a timed petri net model for interactive hypertext system. The major di erences from the TOCPN are how the user interaction is modeled, and how the temporal control is modeled. Also, the Trellis model is a nondeterministic petri net while the TOCPN is a deterministic petri net. The key element in modeling user interaction and temporal control is the transition in Trellis model. The user interaction is modeled as a logical button, and is embedded in a transition which has temporal attributes. The major disadvantages of the Trellis model comes from the fact that it does not consider the synchronization and the temporal attributes are given only to the transitions. Set aside the lack of synchronization mechanism, the temporal attributes given only to the transitions requires speci c identi cation of temporal parameters for pre-de ned temporal objects.
The timeline-tree model of Hirzalla et. al. [3] expands the conventional timeline model to model interactive multimedia document scenarios. A new object called choice object, which de nes user interactions, region of interaction, and pointer to new scenario structure. Representations of each media object in a timeline are extended to represent asynchronous events. Also a tree representation of timelines each of which represents independent scenarios activated by user interactions is introduced. Using combinations of choice objects and extended object representation, the document scenarios of possible user choices are reprented for each branch of the timeline tree. The advantages of the timeline-tree model are the representation of asynchronous start and end of events, and the tree representation of variable scenarios of a given interactive multimedia document. However, this model does not consider the support of database, especially how the timeline-tree should be represented in a database, and lack of representability of iterative document scenarios. When there are parts of the document which can be played repeatedly by user selection, the possible timeline can be inde nitly long, and such a scenario cannot be represented in a timeline tree. The CMIF model of Rossum et. al. [9] has six entities which are presentations, events, channels, timing constraints, hyperlinks, and attributes. A presentation (or a document) of CMIF model is represented as a tree structure just like structured documents. The temporal structure of a presentation can be shown as a timeline graph with parallel and sequential relations. Detailed synchronization information can be obtained using synchronization arcs de ning a delay and allowable deviations. User interactions are supported by hyperlinks which have anchors and links. An anchor is represented as part of media data item, and a link is represented as a directed connection between two anchors. Just like any other timeline based models, it is very hard to represent iterative document scenarios using the CMIF model, and it is unclear from the literature [9] how hyperlinks are represented, i.e. how the change of presentation course can be represented, in the CMIF model with other elements. The TIEMPO of Wahl et. al. [11] [10] supports various user interactions and temporal synchronizations through interval based temporal relations and hierarchical grouping of media items. It supports various temporal relations of equalities and inequalities. However, the graphical illustrations of document scenarios become very complicated for a large document with many user interactions as a small presentation shown in [11] is more complex than most of the models. The Fire y of Buchanan and Zellweger [1] de nes
media items, temporal synchronization constraints, and operation lists as three parts in a document. In [11], it is pointed out the re y model does not support hierarchical structuring and the structure of media items, temporal synchronization constraints and operation lists get very complicated to understand or author a complex hypermedia document.
4. TOCPN Model The conceptual temporal model that we are proposing here is an extended model of OCPN (Object Composition Petri Net) proposed and extended by Little and Ghafoor [4] [5]. As mentioned previously, Little and Ghafoor's OCPN model is one of the most complete time interval based temporal models, but does not take user interaction into account. In this paper, we propose to extend the OCPN based on the state transition diagram of nite state machine theory in order to incorporate the interactive nature of hypermedia applications, and call it Transitional Object Composition Petri Net (TOCPN) model. 4.1. Definition of TOCPN
TOCPN is de ned as STOCPN = fT; P; A; D1; D2; R; M; C; W g where, T (Transitions (bars)) = ft1; t2; t3; tng where n 0, P (Places (circles)) = fp1; p2; p3; png where n 0, IA (Incoming Arcs) : fT P g ! I where I = f1; 2; 3; g, OA (Outgoing Arcs) : fP T g ! I where I = f1; 2; 3; g, D1 (Delay): P ! R where R is real numbers, D2 (Duration): P ! R where R is real numbers, R(Resources): P ! fr1; r2; r3; rk g, M (Marking) : P ! I; I = f0; 1; 2; g, C (Transition Condition) = fc1; c2; c3; cn g, and W (Wait Duration) is real numbers. In this de nition, OA (Outgoing Arcs) can be either an aggregative or atomic type, and also either a destructive or suspending type. Aggregative types of arcs can be considered as a set of outgoing arcs sharing the same destination transition, while atomic types of arcs are single arc de ned as an element of A : fP T g ! I. Each type of destructive or suspending arc tells whether the source place of an outgoing arc terminates or is temporarily paused when the mark leaves that place. The default type of arc is an atomic, destructive type, and the source places of the aggregate arc are assumed to be all the places active at the time of arc activation without speci c notation. Figure 4 shows each type of arc.
S
(a) Default Arc (Atomic, Destructive)
(b) Atomic, Suspending Arc S
or
or S
(c) Aggregate, Destructive Arc
(d) Aggregate, Suspending Arc
Figure 4. Outgoing arcs of each type
A transition condition (ci ) can have one of the following values: d denotes default value for a transition condition. If the transition condition has this value, the arc is activated when the wait duration is passed. Normally, the wait duration is the same as the duration of the source place when the transition condition is default. When the transition condition is default, the transition condition and the wait duration are usually not explicitly shown for simplicity. n denotes nil value. When the transition condition is n, the arc is active as soon as the source place gets a mark (or token), but the place does not lose the mark until the transition res. Normally, the transition condition is n when the source place does not de ne duration, and this case occurs when text is to be displayed as long as a certain video or audio object is presented. c denotes most of the transition conditions for conditional activation of arcs. ci can be any value denoting user interactions including mouse-click of a prede ned object or mouse-click within given a time interval. The transition condition (ci ) can be used for checking the user selection or interaction as follows: 1. user input: By de ning a place (pi ) as a user interface for user input, a speci c action can be taken based on the user input. For example, input to a database entry from users can be handled as one type of transition condition and can be modeled in TOCPN. 2. query: By de ning a place (pi ) as a general querying engine, a speci c query can be handled as one type of transition condition and can be modeled in TOCPN. Furthermore, selecting each object shown in a video clip (clicking on a speci c object) can be de ned as a transition condition, i
activate speci c queries on the selected object, and present speci c presentation. 3. navigation: In most cases, transition condition, ci , can model user interaction for navigational purposes such as going back to the previous object, jumping to other presentations, and selecting a course of presentation as speci ed by the author. A temporal model should be able to specify objects such as text and image objects with an unspeci ed time interval. For example, when a video clip is presented with the background of a text object, the duration of the text synchronizes with the video clip even though the duration of the text is not speci ed. Figure 5 shows the TOCPN model of an unspeci ed time interval object. As shown in Figure 5, the attributes related to δ
1
ur 1 p1
δ τr 2 2
t
1
p2
nnn a
1
a
2
dτ τ
t
2
Figure 5. Unspecified time interval
the object with an unspeci ed time interval are de ned as u for duration, n for transition condition, and n for waiting time. In this model, the arc a1 is active as soon as the place p1 gets a token, and the transition t2 res as soon as the arc a2 becomes active i.e. the actual duration of p1 speci ed as u, depends on the duration of p2 . 4.2. Firing Mechanism
The ring mechanism of the TOCPN described so far can be de ned as follows: 1. A transition ti res when all of its input places have at least one token and all of its input arcs are active. 2. A transition ti res by removing a token from each of its input places and by depositing a token into each of its output places. 3. When a token is deposited into a place pi , pi becomes active immediately and remains so until it loses its token. 4. An output arc ai of a place pi becomes active when pi is active (i.e. has at least one token,) and the transition condition of ai is satis ed.
t δ2 τ 1 r 1 p t
δ3 τ 2 r 2
2
d τ1 τ1
p
s δ1 τ 1
1
t dτ 2 τ 2
2
1
t
t dτ 1 τ 1 a1 sδ τ1
p1 t
δ4 τ 3 r 3 p
δ2 τ 1 r 1
4
1
t
3
δ3 τ 2 r 2
1
a2
dτ 3 τ 3
3
2
dτ 2 τ 2 a3
p2 t
t
3
4
5
Figure 6. Changing the Course of Presentation
Figure 8. Presenting an Additional Object c1 ω 1 ω 2 c 1ω 9 ω 10
δ5 τ 2 r 2 p
p
s 1δ 1τ 1 t
1
δ4 τ 1 r 1 p
1
a1
t
2
s 2δ 2τ 1 a a
4
δ8 τ 5 r 5 p
a
a
dτ 4 τ 4 t
5
dτ 5 τ 5
c 5 ω11 ω12
c4 ω 5 ω 6 c 2 ω13 ω 14
c 2 ω7 ω 8
4
5
a6 δ 10 τ 7 r 7 p
5
7
a9
t
8
Figure 9. Parallel Subnet Substitution
dτ 7 τ 7
a7
2 3
P2
dτ 3 τ 3
P3
δ2 τ 2 r 2
dτ 2 τ 2
3
δ7 τ 4 r 4
δ 3τ 3 r 3
δ1 τ 1 r 1
2
δ6 τ 3 r 3 p
c3 ω 3 ω 4
P1
t
s 3δ 3τ 1
3
δ9 τ 6 r 6 p
t
6
t
7
dτ 6 τ 6
6
a8 t
4
Figure 7. Selecting an Alternative Object
5. A transition condition ci of an output arc ai of a place pi is either a default condition, nil condition or a speci ed condition. A default condition can be satis ed after pi is active for a given duration of i , and a speci ed condition can be satis ed if the user interaction speci ed by the transition condition ci is given while pi is active. A nil condition is satis ed as soon as the source place pi becomes active. 6. An object (place) pi loses its token when the transition ti of its outgoing arc res. 7. When an object pi loses its token, either the object is destroyed or its execution is suspended depending on the type of outgoing arc. When the outgoing arc type is destructive, pi is destructed (terminated). When the outgoing arc type is suspending, pi pauses temporarily until it gets its token back. The three types of user interaction described previously can be clearly modeled using TOCPN as shown in Figure 6, Figure 7, and Figure 8. 4.3. Composite Objects
It has been shown that the temporal relationships among multiple objects can be expressed by combinations of two parallel objects and two sequential objects in Petri Net[5]. Consequently, if two parallel objects
and two sequential objects can be reduced into a single composite object respectively, any arbitrary object can be reduced into a single composite object in Petri Net. In Figure 9 and Figure 10, we show that two parallel objects can be reduced into a single composite object and two sequential objects can be reduced into a single composite object, respectively in TOCPN. In Figure 9, the new place P3 has attributes of, delay 3 = Min(1 ; 2), duration 3 = Max(1 ; 2), and resource type r3 is a composite resource type of r1 and r2; the new arcs has attributes of, transition condition c5 = c3 ^ c4 , start of active durations !9 = 1 ? 3 +!1 , !11 = Min(1 ? 3 + !3 ; 2 ? 3 + !5 ), and !13 = 1 ? 3 +!7 , end of active durations !10 = 1 ? 3 +!2 , !12 = Max(1 ? 3 +!4 ; 1 ? 3 +!6 ), !14 = 1 ? 3 +!8 . In Figure 10, the new place P3 has attributes of, delay 3 = 1 , duration 3 = 1 + 2 + 2 , and resource type r3 is a composite resource type of r1 , delay of 2 , and r2; the new arcs has attributes of, transition condition c5 = c4, start of active durations !9 = !1 , !11 = !3 + 2 + !5, !13 = !3 + 2 + !7 , and end of active durations !10 = !2 , !12 = !4 + 2 + !6 , and c1 ω 1 ω 2 P1
c 2 ω7 ω 8
c3 ω 3 ω 4
P2
δ1 τ 1 r 1
c4 ω 5 ω 6
δ2 τ 2 r 2
c 1ω 9 ω 10 δ 3τ 3 r 3 P3
c 5 ω11 ω12
c 2 ω13 ω 14
Figure 10. Sequential Subnet Substitution
!14 = !4 + 2 + !8 . By applying the reduction methods shown in Figure 9 and Figure 10 repeatedly, any subnet of TOCPN can be substituted for by a single place without changing the property or action of the TOCPN. By using the subnet substitution in TOCPN, composite objects can be constructued and various levels of granularity of synchronization and the logical structure which is hierarchical tree structures connected by arcs can be modeled. 4.4. Database Schema for TOCPN
To show how the TOCPN model can be used in database systems, we outline database schema to store and retrieve a TOCPN model. As shown previously, since the transitions and arcs play the most important role in the TOCPN model, we build the database schema based on the transitions and arcs. Class arc is to keep the hyperlink information coming out of each place, and shown in the following. public class arc { int doc_id; /* Document id */ int arc_id; /* unique id */ char A_type;/* Atomic or Aggregate */ char DS_type; /*Destructive/Suspending*/ transition To_transition; place From_place; char condition; /*activation condition*/ int[ ] act_duration; /* How long should wait to check and keep checking the condition in msec */ activate_arc (int doc_id, int arc_id) { /* activate arcs */ } deactivate_arc (int doc_id, int arc_id) { /* deactivat arcs */ }}
Class transition is to keep the transition bar in the TOCPN model, i.e. to keep the synchronization point information, and shown in the following. public class transition{ int doc_id; /** Document id */ int transition_id; /* unique id */ arc[ ][2] in_arc; /* arcs required to fire and see if they are active. */ place[ ] to_place; /* places to be activated when fired */ fire (int doc_id, int transition_id) { /** activate places*/ } arc_arrived(int d_id,int t_id,arc i_arc) {/* set flags of activated arcs*/}}
Class place is to keep the actual resource information in the composite documents. Class place has two subclasses which are atomic place and composite place, and shown in the following. The attribute hier level of composite place is used to keep the level of each composite place in the tree structure. atomic place is considered to have zero value of hier level, and the composite place for the root of the tree has the largest value of hier level. public class place{ int doc_id; int place_id; arc[ ] arcs_to_activate; place parent;} class atomic_place extends place{ int resource_id; /** HO id */ char resource_type; int duration; /** in msec */ int delay; /** in msec */} class composite_place extends place{ transition[ ] initial; transition[ ] finish; int hier_level; /** level in logical tree struct. atomic_place has default value of 0, and root node has the largest number */}
5. Document Composition and Tools In this section, we show an example of document composition scenario to help understand the procedure of document composition using the TOCPN model, and extract further requirements of an authoring tool and helper tools. 5.1. Composition Scenario
Composing an interactive hypervideo document using TOCPN can be done as follows: 1. Browse existing hypervideo objects and select objects to use, or create one. (a) Using browsing and querying function, select interested objects from the database of objects. (b) If the selected object needs editing, do editing and register as a new object. (c) Creating new object involves taking new pictures and is out of scope of this paper. Once picture is taken, store in the database, and register as an object. Then proceed as existing objects.
2. De ne synchronization points (transitions) among selected objects. (a) For default transition, i. Select set of objects to synchronize. ii. Insert transitions at appropriate point. iii. Connect objects and transitions by de ning link type and giving appropriate attributes. (b) For conditional transition (hyperlinks), i. Browse possible links from linkbase. ii. Select possible links or add new one. iii. De ne link type, and give appropriate attributes. iv. Give activation conditions for each link. v. De ne wait time for link activation. 3. Check the reachability of composed document. 5.2. Requirements of Tools
From the scenario of document composition, we can extract the list of necessary tools to aid authors and functionality of each tool. Among them are the object browser, the object editor, and the authoring tool. The object browser is required toidentify and select desired object to include in the document with the following functionality. Feature based search Authors should be able to search the video data in the database based on the feature values, such as contents, texture, and shape of image Preview capability Once a set of hypervideo objects are selected, authors should be able to preview the selected video clips to check the contents and quality of images. List of links Each selected hypervideo object can have possible links. Authors should be able to see description of the possible links of the selected object. Navigation support From the rst set of selection, authors should be able to retrieve other related hypervideo objects based on the description of the links. The object editor is required to modify the selected object with the following functionality. Cut and paste function Needless to say, cut and paste are most fundamental functions for editing.
Resize functionality To coordinate the size of pre-
sentation, authors should be able to resize the original video object. Special e ects To provide the smooth transition between hypervideo objects such as fade-out and fade-in, special e ect functions are required. Basic lter functions Along with special e ects, basic lter functions should be provided to modify original hypervideo object. The authoring tool is required to guide and help authors to follow the steps as described in the scenario with the following functionality. Graphical user interface To show the relation among objects, transitions, and links, graphical user interface should be supplied. Automatic data entry generation When authors de ne transitions and links based on the selected objects, the data entry for the structure of composed document should be automatically generated and stored in the database. Interactive data gathering When values of attributes are required, the authoring tool should be able to prompt the author and gather data from the author, so that the authors who are not familiar with the internal data structure do not leave important data unde ned. Veri cation of composed document Once the document is composed, the system should be able to verify the composed document, so that unreachable components or con icts in transitions can be prevented. A prototype of the authoring tool using the TOCPN model is currently under development and named TIMDAT (TOCPN Interactive Multimedia Document Authoring Tool). The architecture of TIMDAT is shown in Figure 11. Among the modules of TIMDAT, temporal structure management module, object management module, physical object management module, and database module originally belonged to LHDDS (Layered Hypervideo Document Database System) [12].
6. Example Document Using TOCPN In this section, we show an example document expressed in TOCPN to show the advantages of this model over conventional models with brief explanation of TOCPN execution.
Document Composition Module
d1 τ 8 r 2
0 ur 1 T3
A3 0 τ9 r 3
Temporal Structure Management Module
c2 Object Editing Module
Search & Browsing Module
Object Management Module
V2
0 τ2 r 1
0 u r1 T1
d2 τ 5 r 2
0 u r1 T4
T3
c3
A4
Physical Object Management Module
0 τ1 r 2
0 τ4 r 2
A1
A6 0 τ3 r 3
c1 Database
c1 τ1
0 τ6 r 1 T4 0 τ7 r 3 V3
V4 c1 0 τ 12 r 3 V1
LHDDS
0 τ 11 r 1
0 τ 13 r 2
T2
A2
Figure 11. Architecture of TIMDAT A1 T1
Figure 13. Interactive Multimedia Document Example Using TOCPN Model
A2 T2 V1 Timeline1 C2
Choice1 Actor1_text
(T1)
narration1
(A1)
Actor2_text
A3
Choice3
Choice2
T3 (T3)
Actor3_text
(T4)
V2
Choice3 A6 T4 V4 Timeline2
narration2
video_of_some_great_actors
(A6)
A4 T4 V3
(V4) Timeline0
Timeline3
Figure 12. Interactive Multimedia Document Example Using Time-Line Model
Consider an interactive multimedia document as shown in Figure 12 and Figure 13 using the timeline model as shown in Hirzalla et. al. [3], and the TOCPN model respectively. The possible scenarios of the given example document are as follows: 1. When there is no user input, the document ows as given in Timeline0 of Figure 12, showing V4 in full, giving the narrations A1 and A2 in sequence, and displaying T1, T2 and T3 in sequence. 2. When the user selects Choice1 (i.e. selects to see the details of Actor1,) presentation of V4 is replaced by V1 while the T1 and A1 continues to be presented. After A1 is done with presenting, T2 replaces T1 and after a certain time, A2 replaces T2. The A2 and V1 co-end when the document ends. 3. When the user selects Choice2 (i.e. selects to see the details of Actor2,) presentation of V4, A6, and T3 are all temporarily suspended and a new presentation composed of V2, A3, and T3 is given. At the end of this new presentation, the original
presentation of Timeline0 is resumed at the point of suspension. 4. When the user selects Choice3 (i.e. selects to see the details of Actor3,) the original presentation of Timeline0 is killed, and new presentation composed of V3, T4 and A4 is given. The rst scenario does not re ect the user interactions. Hence neither time-line based models nor TOCPN has any problem in expressing the rst type of scenarios. The second scenario re ects a partial replacement case where only V4 is replaced by V1 while A1 and T1 continue to play. As shown in Figure 12, it is hard for the time-line model to show the case clearly. The third scenario re ects a suspending case where the presentation of the current document is temporarily suspended while the document of Timeline2 is presented. This case causes an unlimited number of possible timelines due to the loop caused by the reactivation of the original document which is impossible to express in a time-line model. The fourth scenario re ects a simple change of course case, in which the original document is killed and the document of Timeline3 is activated when Choice3 is selected. The third scenario of TOCPN shown in Figure 13 can be executed as follows: 1. Initially A1 and V4 are active for the duration of 1 and 3 , respectively with T1. 2. After 1, A1 and T1 becomes inactive, and T3 and A6 becomes active. 3. User input of c2 deactivates T3, A6, and V4, which are still active at the moment, and activates A3 with delay of d1 and V2.
4. A3 is active for the duration of 8 , and it becomes inactive, when the time expires, which causes activation of T3. 5. T3 is active as long as V2 is still active. 6. V2 has lifetime of 9 and when the time expires, T3 also becomes inactive. 7. As soon as T3 and V2 becomes inactive, T3, A6 and V4 becomes active which is the state of step 2. 8. If the user input of c2 does not occur, T3 becomes inactive after 2 and activates T4. 9. T4 and A6 becomes inactive synchronously after A6 has been active for 4 . 10. When V4 has been active for 3 , and T4 and A6 becomes inactive, the execution of the document ends. 11. If user interaction of c3 occurs while T4 is active, the document proceeds as the fourth scenario activating A4, new T4, and V3 and deactivating old T4, A6, and V4. The limitations of the time-line based interactive temporal document model are clearly shown in the second and third scenarios of the example document. One of the advantages of the TOCPN is that it can clearly express all these cases, in which conventional models have limitations, as shown in Figure 12 and Figure 13.
7. Conclusion In this paper, we analyze possible activities of user interaction on an interactive multimedia document, propose a new temporal model for an interactive multimedia document (TOCPN). TOCPN have several advantages over conventional interactive data models such as clear support of user interactions (i.e. changing the course of presentation, selecting an alternative object, and presenting an additional object), support of both atomic and composite objects, support of sharing and reusing of existing objects, and powerful expression of dynamic structure of interactive documents. In addition, object-oriented database schema for TOCPN document is developed. As TOCPN is not a physical but a conceptual data model, TOCPN is independent of physical database systems, and any of the relational and object-oriented database systems can be used to implement TOCPN. Requirements of an authoring tool and helper tools are also brie y introduced with architecture of the authoring tool.
Further research problems include optimization of database schema for TOCPN, extension of TOCPN for distributed collaborative environments, and extension of the TIMDAT authoring tool as the TOCPN is being extended. Extension of TOCPN and TIMDAT for the other structures of interactive multimedia documents can also be a topic for further research.
References [1] M. C. Buchanan and P. T. Zellweger. Scheduling multimedia documents using temporal constraints. In Proceedings of the Third International Workshop on Network and Operating System Support for digital Audio and Video, La Jolla, CA, November 1992. [2] M. C. Buchanan and P. T. Zellweger. Specifying temporal behavior in hypermedia documents. In Proceedings of ACM ECHT'92 Conference, pages 262{271, Milano, Italy, December 1992. [3] N. Hirzalla, B. Falchuk, and A. Karmouch. A temporal model for interactive multimedia scenarios. IEEE Multimedia, pages 24{31, Fall 1995. [4] M. Iino, Y. F. Day, and A. Ghafoor. An objectoriented model for spatio-temporal synchronization of multimedia information. In Proceedings of 94 International Conference on Multimedia Computing and Systems, pages 110{119, Boston, MA, May 1994. [5] T. D. C. Little. Synchronization for Distributed Mulitmedia Database Systems. PhD thesis, Syracuse University, Syracuse, NY 13244, 1991. [6] T. D. C. Little. Time-based media representation and delivery. In J. F. K. Buford, editor, Multimedia Systems, chapter 7, pages 175{200. ACM Press, 1994. [7] P. D. Stotts and R. Furuta. Petri-net-based hypertext: Document structure with browsing semantics. ACM Transactions on Information Systems, pages 3{ 29, January 1989. [8] P. D. Stotts and R. Furuta. Temporal hyperprogramming. Journal of Visual Languages and Computing, pages 237{253, 1990. [9] G. van Rossum, J. Jansen, K. S. Mullender, and D. C. A. Bulterman. Cmifed: A presentation environment for portable hypermedia documents. In Proceedings of ACM Multimedia '93, pages 183{188, Anaheim, CA, August 1993. [10] T. Wahl and K. Rothermel. Representing time in multimedia system. Technical Report TR-93-12, Universitat Stuttgart, Germany, 1993. [11] T. Wahl, S. Wirag, and K. Rothermel. Tiempo: Temporal modeling and authoring of interactive multimedia. In Proceedings of the International Conference on Multimedia Computing and Systems, pages 274{277, Washington, D.C., May 1995. [12] K. Yoon. Layered Hypervideo Document Database System: Design and Modeling of Hypervideo Document Database. PhD thesis, Syracuse University, Syracuse, NY 13244, 1998. in preparation.
