models for storing and presenting multimedia documents - Larim

Telemacics and Informa~ics, Vol. 13, No. 4, pp. 233-250, 1996 Copyright 0 1996 Elsevier Science Ltd Printed in Great Britain. All rights reserved 0736-X353/96 $15.00+0.00

Pergamon

PII: SO73&5853(96)00024-X

MODELS FOR STORING AND PRESENTING MULTIMEDIA

DOCUMENTS

Samuel Pierre Hai’dar Safa

Abstract - This paper deals with models for storing and presenting multimedia documents integrating large amounts of data of different types. Usually, multimedia systems integrate a variety of data such as voice, graphics, text video and other types of images into a single document. Most of these data are not structured and therefore require a huge storage capacity. Such a requirement causes many problems for traditional database management systems which were not originally designed to manipulate data other than text. The presentation of multimedia documents incorporates documentary research that depends upon storage model and mechanisms. In this paper, we present two data storage models and several other associated models for the presentation of multimedia documents. The analysis of these models reveals their capacity to synchronize various temporal scenarios while allowing users simultaneous and secure access to multimedia systems. Copyright @) 1996 Elsevier Science Ltd Keywords - storage model, presentation

media synchronization,

model, multimedia document, multidistributed multimedia database

1. INTRODUCTION

The development of computerized multimedia applications and the technical requirements resulting from the particular nature of these applications call for a document architecture adapted to the characteristics and the types of documents to be integrated. In the case of distributed applications, such an architecture is meant to structure, represent and manage the exchange of data between two users, or two workstations. These functions are also meant to take place with centralized applications that involve an exchange of data between a server and a user/client. In the current computerized environment, there is an urgent need for better storage and presentation models of multimedia documents integrating a large amount of data of different types. Indeed, current multimedia systems integrate a variety of data such as voice, graphics, texts, video and different types of other images into a single document. However, most of these data are not structured and therefore require massive Both authors are at Centre de recherche en informatique cognitive et environnements de formation (LICEF), T&-universitk, Universiti du Qdbec, 1001 rue Sherbrooke Est, 4e &age, CP 670, Succ. C, Mont&al, Qutbec, Canada H2L 4L5. Address all correspondence to Samuel Pierre; E-mail [email protected]. 233

234

Samuel Pierre and Ha’idar Safa

storage capability. Furthermore, the lack of suitable data structure creates many problems for traditional database management systems (DBMS) which were not originally conceived to manipulate data other than text. There is a fundamental difference between a traditional database and a document base. By definition, a traditional database is a collection of encoded and structured data that are independent of any particular application. Conversely, a document base is a collection of computerized documents associated with standards for storage, manipulation, display and transmission. Hence, a document base is a place where document modeling is linked to the development of abstract models. In these models, documents are represented as complex objects comprised of text, graphics, images and sound (Guidon & Pierre, 1995). An example of this is the computerized office document architecture (ODA) (Horak, 1985). The initial task that leads to suitable presentation of multimedia documents is the search for documents appropriate for computerization. This preliminary task depends on the chosen abstract model and the mechanisms used for storing document. Such a task is motivated primarily by the necessity of identifying relevant information (to users) which could be drawn from either a single source or several sources. Traditional systems of documentary research could be divided into two categories: the keyword-based research systems and the full text research systems (Meghini, Rabitti, & Thanos, 1991). In keyword-based research systems, a document is represented by a set of words and each word is related to an index. In such systems, the most important problem lies in the choice of the terms that best define the content of a document. This problem has a major drawback and results in severe limitations when one uses specific descriptors to represent the content of a document. The full text research systems are often used to support documentary research that emphasizes documents’ contents. In these systems, a document is defined by its attributes and its content; the latter being represented by an index or a list of words. It follows that, when documents are large, their contents are often divided into chapters which are in turn broken into sections. While each chapter or section is represented by a list of words, attributes are also ascribed to chapters and sections. Thus, it is possible to ascribe attributes to both levels: the entire document and the chapters that constitute it. The major inconvenience of such systems is the fact that large portions of documents must be scanned before the required documents are found. Content-based documentary research systems are convenient for multimedia documents because voice messages, fixed and animated images and graphics contain information that enriches the textual content of documents. Such information is meant to be used by both the user and the system to formulate their requests and to access the document bases. This paper deals specifically with the storage and the representation of multimedia documents. Section 2 outlines the theoretical foundations on which the storage and the processing of multimedia information are based. Section 3 deals with multimedia architecture, Section 4 presents storage models. Section 5 focuses on presentation models and analyzes a few questions related to access mechanisms that support multimedia documents. Section 6 concludes the paper with a summary of the proposed models. 2. PROCESSING AND STORAGE OF MULTIMEDIA INFORMATION The main functions of a multimedia information system can be divided into two stages: the retrieval and manipulation of data, and the support for appropriate access mechanisms. The first stage is the retrieval of data necessary for the creation of a

Models for storing and presenting multimedia documents

235

document that is able to integrate different types of data. The second stage consists of bringing into play access mechanisms and models of presentation. In contrast to traditional databases, multimedia databases deal with several types of data originating from different sources and having different structures. The retrieval process is followed by storage mechanisms and the appropriate compression for each type of data, integrating data into a single application. The result of the second stage is a set of information that combines data of different types: several links that model the relations between data, and constraints that guarantee consistency. These sets of links and constraints constitute the multimedia document, which is stored in a multimedia document base. Once the document is created, it is delivered to the user through a network or by means of CD-ROM. Figure 1 illustrates this scenario. A multimedia application system essentially operates at three layers: (1) the storage of data, (2) their integration and presentation and (3) the search for and access to documents. However, a distributed multimedia application system requires an additional transport layer. While the first layer is a host for the models and storage techniques, the second layer proceeds with integration of data and introduces the presentation models. The third layer is dedicated to the appropriate access mechanisms. Figure 2 illustrates the superposition of these layers. Resources I

Storage Structured Data Ctaphics/Images Video/Audio

IIText, Graphic.

Image Editor

II

‘7

1

Presentation Presentation

I

Ftgurel. Multimedia data processing.

236

Samuel Pierre and Ha’idar Safa

Access Mechanisms to Multimedia Databases

Layer 3

Presentation and Composition Models of Multimedia Document

Multimedia Storage Models

Layer 2

I

Layer 1

Figure 2. Superposition of layers for multimedia application

Usually, the storage of multimedia data, particularly that of video and audio, proceeds through digitization. Whereas the digitization of video produces a sequence of frames, the same operation with sound generates a sequence of samples. Thus, a continuously registered sequence of frames and samples is referred to by the notion of strand. Each strand is stored onto a disk which is divided into several media blocks separated by empty spaces called “gaps”. The storage shape of a strand is determined by the size of the media block (M) and that of the empty spaces (G) that constitute the number of sectors on a disk. Media blocks are supposedly homogeneous, in the sense that each block contains a single type of media. This homogeneity allows the file system to use the characteristics of each media in order to optimize the storage space. On the other hand, heterogeneous blocks would require additional processing for the combination of media data during storage and for their separation during research. Each size of a media block (M,) and each size of an empty space (G,) of a strand (S,) are supposed to cover an integer number of sectors. As shown in Figure 3, each relation (M,, G,) refers to the shape of strand St which is composed of storage forms. The factor that determines the shape of a strand’s storage is the requirement for research continuity. In return, such research continuity can be guaranteed only when each media block is appropriated for visualization through a panel control before or after its reading. However, when the blocks of a strand are randomly disposed onto a disk, there is a chance that the separations between the successive blocks of a strand will be incapable of supporting continous research for media strands. At the other end of the spectrum, the contiguous allotment of blocks ensures research continuity, but at a high cost which is caused by the processes of insertion and deletion. These two drawbacks could be avoided if the blocks are distributed in a way such that their sizes and the void spaces between them are determined by the requirement for research continuity. The latter is defined by the expression: Mt+Gt -= Dtd

T

a,

where Mt refers to the size of the media block, Gt indicates the void spaces between blocks, T, the media display time measured in seconds, and Ddt the flow of data transfer from the disk measured in sectors/s.

Models for storing and presenting

I

M2

a

M2

I ^_,

W

0

Ml

Gl

Ml

G3

M3

I Cz

G3

Gl

_

I W

1 m

M3

IImIImIImIImIImlll

multimedia

Cz

‘I

G3

M2

_

I G2

1

m

G3

documents

237

Gl

MI -

(SO

I

M2 0

M3

G;! I

G3

I

I

.,,,,,I,

(S2)

1I,, . ,, ,,

M3

G3

,,,,I,#, 03)

Free sector Sector with multimediadata Figure 3. Sequence of three media strands.

It is supposed that the reading of a disk and the visual display of a media are executed in a continuous way, as illustrated in Figure 4. Furthermore, it is also presumed that the flow of data transfer Dtd is the same as the flow of disk rotation (in sector/s). The requirement for continuity is met if the time to jump from one block to another, as expressed by the left part of the relation (l), does not exceed the visual display time (i.e., the right part of that relation). The shape of (M,, G,) storage of a strand St must satisfy the relation (1). However, the shape of a strand is not unique when there are two variables and one relation. The exact shape of a strand could be determined by fixing one of these variables and by substituting its value in the relation in order to obtain the value of the other variable. Technically if the disk is often empty, then each strand could be stored according to its storage shape. This is not possible, however, because a disk always contains a large number of strands. In order to efficiently use a disk, the blocks of a new strand must be stored onto the available void spaces as shown in Figure 5 (Rangan, Kaeppner, & Vin, 1992). Unfortunately, there is no guarantee that the blocks of the new strand can be arranged in conformity with the disk’s storage form. Therefore, the requirement for continuity could be strictly maintained for each media block. 3. MULTIMEDIA DOCUMENT ARCHITECTURE Multimedia document architecture is composed of a logical structure, an arrangement structure and content. The logical structure provides the method for the setting

Figure 4. Continuous research for media blocks in a strand

238

Samuel Pierre and Ha’idar Safa

Ml

Ml

G1

I I I 0

G1

I I I 0

Ml

Gl

I I I I (Sl)

S2 fusioneddam S1

cl Free sector q S1 data sector W2

data sector Figure 5. Fusion of two strands.

up of the document content, whereas the arrangement structure supplies the method for the organization of the document content on each page. The content represents the information that is usually associated with the documents. This section deals with the logical and arrangement structures first, then the conceptual structuring of multimedia documents. 3.1. Logical and arrangement structures of documents The logical structuring of multimedia documents is based on concepts such as abstract object, hierarchical links between objects, requested and non-requested objects and distributed components. All these concepts are represented in Figure 6, which describes the logical structure of a document. The ODA mentioned in Section 1 is considered here as the basic frame of reference that allows the design of efficient document models. ODA has the capacity to merge into a single frame the representation of a document structure (logical structure) and the formats of document presentation (arrangement model). Our objective is to build a model which provides a method that is able to describe the electronic representation of documents while recognizing the specificity of each type of information included in the documents. For this purpose, we find that, in principle, an ODA-structured document could integrate text, graphics and images and is, therefore, a multimedia document. Unfortunately, ODA does not provide the methods capable of supporting continuous multimedia data such as video, audio, animated images, and so forth. In addition, while a multimedia document is constituted of spatially and temporally related objects, the ODA architecture is unable to perform the continuous synchronization of these objects.

Models for storing and presenting

I

I

I I

I

I

1

,

I

(+r-) I

I

,

239

documents

I

(z+-)

SectiOtl

Title

multimedia

I

Figure 6. Logical structure of a document

Multimedia objects can be elementary or composite. While an elementary object is a unit of a media representation, a composite object is constituted of several elementary or composite objects. The behavior of an object is described by a program or script (also called a scenario). The latter calls upon the methods of elementary objects to establish the methods for the composite objects. Figure 7 illustrates the hierarchy of composite objects. Hence, documents whose elementary objects ought to be presented simultaneously cannot be arranged by using current ODA architecture. This is because ODA is an architecture designed to handle only static documents (i.e., those created and manipulated by an editing software).

Multimedia

I

Document

I

SCenSliO Composite object 3

Composite object 1

Ekmentary object 1

Compositeobject

2

Scenario

I

I

Elementary object 2 Elementary object 3 Elementary object 4 Figure 7. Hierarchy of composite objects.

Bementary object 5

240

Samuel Pierre and Hdidar Safa

As the ODA architecture is not suitable for synchronizing multimedia objects, a multimedia document architecture should provide a structuring model for both categories of media: the continuous media such as video and audio, and the discrete media such as text, images and graphics. Thus, in addition to the logical and arrangement structures provided by ODA, continuous media require the specification of the temporal relations between the objects that are described by these structures. For this purpose, it is useful to mark the difference between the time-dependent and the time-independent data. While time-independent data is static, time-dependent data such as video, audio and animated images are considered dynamic; that is, their expressive interpretation conforms continuously to a progressive change on a time scale. Thus, temporal information is an integral part of this type of data. This leads us to differentiate between intra-temporal information and inter-temporal information. Indeed, whereas the former is the factor that makes a set of data, time-dependent, the latter is the element that determines the temporal relations between the components of a multimedia document. An arrangement structure could be directly created in ODA by following the latter’s logical structure. However, such a method is not practical for multimedia documents because arranging a document in ODA architecture is simply an act of displaying and sharing out a document into several pages which in turn are divided into frames and blocks. These pages, frames and blocks are the locations where pieces of information are displayed. This ordinary method is not suitable for multimedia documents because arranging a multimedia document is a more complex process that requires primarily the identification of the objects to be presented and a scenario that includes all temporal relations between objects. Furthermore, multimedia arrangement supposes that a scenario must be created before even presenting the multimedia document. Such a scenario should describe when and for how long each object is presented. Therefore, it can be said that the arrangement structure of a multimedia document is always associated with a scenario that takes into account the time factor in the process of document presentation. A multimedia document architecture is used for organizing, representing and structuring the information that is exchanged between a user and a machine. However, the structure of a document does not enable one to search for a document in a large document base. This is because the perception of a user could vary from one document to another. For example, a user could perceive certain documents from the view of their arrangement structure; he could see other documents from their logical structure and still see some others from their content. Therefore, in addition to the logical and arrangement structures, a document needs a conceptual structure that facilitates the representation of its content. 3.2. Conceptual structuration of documents In information systems, conceptual modeling always begins with the introduction of data models, notably the relational model. Recently, powerful tools for data modeling including semantic data models have been proposed for a more adequate representation of semantic applications. Among these models, the object-oriented models are the best for modern database application modeling. While these models conceive the world as an interconnection between objects, they also offer the abstract mechanisms of a semantic model that is necessary for the classification, aggregation and generalization of objects. The term classification represents the relation instance ofthrough which several objects with common characteristics are brought into other objects named classes. Aggregation

Models for storing and presenting

multimedia

241

documents

indicates the relation part of, or the relationship between an object and its components. Through aggregation an object could be seen as a compound of elementary objects. Generalization is the relation “is” through which classes could be organized in taxinomies ordinarily called the “is” hierarchies. These mechanisms do two things: they specify the relationship between objects and they interact with each others through the concept of inheritance, which can be subdivided into two categories: the first is the category of the inheritance ofinstances through which an instance of a class can also be an instance of its own generalizations, the second is the inheritance of characteristics allowing the characteristics of a class to also be the characteristics of its own specializations. Figure 8 shows a small database described by this semantic model. Figure 9 shows a student card where the content is made of multimedia data (images, graphics and texts). Figure 10 illustrates the conceptual modeling of this multimedia document. 4. STORAGE MODELS In this section, we propose two scenarios of multimedia document storage. The first is called centralized model because it considers that multimedia data is centered on a single server, whereas the second scenario is called distributed model and processes multimedia data located on many specialized servers.

I

PERSON

I

I

I\

-_-_

Instance of __)

3

Paltof

Figure 8. A semantic database model

1 POSTALCODE

242

Samuel Pierre and Ha’idar Safa

I

University

of Quebec

EXPIRATIONDATE JUL 3119%

at Montreal

MICHELEWRIGHT wRlM1901Moo1 We

student

A456789320 Figure 9. A student’s card processed as a multimedia document

Student Card

FigureIO. Conceptual modeling associated with Figure 9

4.1. Centralized model

As Figure 11 shows, in centralized models, multimedia data are stored on one support and managed by a single server. Several types of data can be processed differently but stored in the same file. For example, data such as text, images and graphics that are generated in different servers could be stored in the same file. However, changing the

243

Models for storing and presenting multimedia documents

User 1

IWtimedia Server

4 P R

E s

Multimedia Data

Multimedia Client

Figure 11. A centralized storage model.

E N T A T I 0 N

configuration of images in this file would require their transfer to an image processing software. Therefore, it would only be after the use of appropriate integration mechanisms that this model would be capable of storing different types of data in the same file. 4.2. Distributed model

In distributed models which are normally constituted of servers and multimedia data of different types, data are acquired and stored on specialized and sometimes remote servers. The types of servers used in almost all the applications are: the image data servers, the graphic and textual servers, the audio data servers and, finally, the video servers. The main advantage of this model lies in the fact that the access to data is accomplished using a single command. Moreover, during the presentation of data, it would not be necessary to have predetermined synchronization mechanisms; otherwise, the multimedia document would be considered a unique object. The role of an image data server is to manage the storage of data and to search for images. It also supplies the multimedia system with a permanent strategy for the manipulation of stored images. Such a strategy often operates at two levels of storage. The first level is the active storage that makes available to clients the data that is frequently accessed during a work session. The second is the level of archive storage that processes the data which is not frequently accessed. The server of graphic and textual data manages character-based data including texts, graphics, and some types of images. For a multimedia system, it is not sufficient to process and present textual data as chains of simple characters. As mentioned earlier, a useful representation of textual information must contain logical as well as arrangement information. Here, the notion of graphics integrates all the concepts necessary for the management of drawings and other images based on formal descriptions, programs or data structures. The typical elements of a graphic are: lines, regions and elements of text. To be able to process structured data in a complex manner, the server must integrate the graphic data with the other types of media.

244

Samuel Pierre and Ha’idar Safa

The server specialized for graphic and textual data is sometimes used to manipulate low resolution scanned data and continuous images such as pictures and manuscripts. Many multimedia applications use a single server to manage simultaneously image, graphic and textual data. The common characteristic of these three types of data is their time-independent feature; they are not subject to a coding system similar to the one used to process audio and video data. Conversely, audio data is continuous and timedependent. While operations such as cut, copy and paste are performed in a static fashion, the recording and reading of audio are associated with a time scale and are carried out through an appropriate network that links the server to the audio terminal of a workstation. The latter controls the audio server by sending the control commands through the network. It follows that the audio server must be capable of processing and reflecting the characteristics of audio data such as time-dependence, the expression of temporal relations and the support to storage and compression. Such capability is necessary for the semantic interpretation of audio data. This server is also able to process speech. Although speech data is always processed as audio data, it remains that the variation of speech from one individual to another makes speech data different from the other audio data. The search for video data is a complex problem. However, the development of image processing techniques based on the motion and the similarity of images can be used to foster a possible solution. The type of basic video data is the video sequence, a set of frames linked by the different times (integers) of their presentation. The main operators associated with a video sequence include: the speed which allocates the number of frames to be presented in a time unit, the cZipthat extracts the frame sequence between two points in time, and finally the insertion that interpolates one video sequence into another. A video data server must convert analog data to a digital form. This process of conversion uses several animated data techniques such as MPEG that enables the storage of video segments on a server. When a video segment is read as a part of multimedia, the workstation must send an adequate command with an appropriate address to the server. This process is necessary to start the reading, and then to present the video segment.

5. MODELS OF PRESENTATION

Before dealing with the problems related to multimedia synchronization, this section presents the types of servers on which the presentation of multimedia documents could be modeled. For this purpose, a difference should be made between the presentation of ordinary data and that of multimedia data. Generally, ordinary data essentially includes types such as symbols and characters processed by programming languages. Their representation is performed according to the computer at hand. Contrary to this, the presentation of multimedia data is not canonical in the sense that it does not conform to a general rule or procedure. It follows that multimedia documents are not directly supported by programming languages. Rather, their presentation depends upon special devices and supplementary information related to their specific image formats, compression techniques and arrangement descriptions. Furthermore, the presentation and the control of multimedia documents from a user site require many things: a client/server architecture (Sinha, 1992) a buffer space, networks that support the protocols for continuous transport. With this in mind, let us now proceed with the types of servers.

Models for storing and presenting 5.7.

multimedia

245

documents

Types of servers

In considering the two storage scenarios presented in the previous section, one can distinguish two models of multimedia documents presentation. While the first is a model of presentation without preliminary composition on specialized servers, the second is a model that uses a single and, eventually, a temporary server (of already composed documents) which is associated with a work session involving a user. The model of presentation using a unique server corresponds to the model of synchronized storage. It does not have a scenario for document composition since each stored document is already composed. The document appears as a unique object. It is sufficient, therefore, to send the appropriate command to start its transfer to the client’s site. To guarantee good quality service during the presentation of a document, it is preferable that the latter be entirely transferred on the client site before starting its presentation (Vogel, KerhervC, Bochmann, & Gecsei, 1995). The client/server architecture is necessary for the model that seeks multimedia documents and uses a single storage support. As shown in Figure 12, the document found by this model is manipulated and then transported via a network to a workstation where it can be presented. The presentation model using specialized servers corresponds to the distributed storage model mentioned earlier. In fact, each presented object could involve several others. If these objects are physically stored on several separated servers, then a document composition model would be required to integrate and connect the multimedia objects into a single document. Thus, the composition of the document is realized according to a scenario that determines the relations between the objects involved. Users can create such a scenario by utilizing the tools appropriated to their workstations. Figure 13 is a platform that illustrates both the composition and the presentation of a multimedia document. The presentation model which is dedicated to specialized servers requires a database server that supplies a set of applications for multimedia storage, retrieval, composition and manipulation. One important aspect of this model is that the integrated objects of a document originated in several servers contain data of different types. This aspect Multimediadatabase server User

Multimediaclient

Figure12. Presentation of centralized data on a single server.

I

t-l

246

Samuel Pierre and Hdidar Safa

User 4 Request of Document User Interface Synchronization

n

Quality of Service Puffer ing

Time Line Video 1 Presentation Text 3

I’ ‘I’ ’ lmagellPmage1(Video1

(Imageltmage

Audio Data Server

Image Data Server

4

1

(Video1

Image Data Server

Gaphical and Textual Data server

Figure 13. Multimedia application platform with distributed storage.

shows why multimedia synchronization should be carried out from the beginning to the end of the presentation process. In complex multimedia applications, a flow of media can go across several intermediary systems before reaching the system where it can finally be presented. In addition, the distributed application itself can be constituted of several processing elements that form a stream of graphics. Each of these elements can cause a delay and jitters that influence the quality of service during the presentation of a document. This is the problem that raises the importance of synchronization in multimedia applications. 5.2. Media synchronization In multimedia systems, synchronization is considered a fundamental problem. In fact, each multimedia system should be provided with mechanisms clarifying the constraints of synchronizing different media types. There are two levels of synchronization: the continuous and the event-based. Whereas the former is oriented towards the reading

Models for storing and presenting multimedia documents

247

of synchronized data flow, the latter articulates the presentation of the separated multimedia objects by using several servers. These objects can be made of several temporary related subobjects such as video clips, text fragments and images. However, if a multimedia object is made of continuous media, then the synchronization of objects includes media flow at the lowest level. It follows that the mechanisms of object synchronization are based on the flow synchronization functions. To better clarify the diverse notions of synchronization, it is necessary to use a terminology that consists of modeling, specifying and representing time in a computer. Such clarification is indispensable in explaining multimedia scenarios and synchronization techniques. The objective of temporal information is threefold: the development of techniques required for the synchronization of temporal scenarios, the reproduction and the handing over of these techniques to the user’s workstation. The challenge, however, is the coherent reading of the continuous recorded data. To overcome this challenge, the synchronization of flow needs a rapid bus, temporary memories and decompression devices. To achieve the desired presentation, the characteristics of the operating environment (communications, operating systems, reading and displaying devices) must be processed as soon as the required characterization is completed. The problem of synchronization consists of modeling, representing and specifying the time requirements of multimedia scenarios. This is because one sequence of events can be alternatively presented by means of several time models. As shown in Figure 14, one sequence of events in a movie can be described in four ways:

(9 A textual description. In the beginning, the car stops. Then, the red light turns green and the car starts to move until a policeman orders it to stop. (ii) A description of time occurrence. At time t=O (the beginning of the film), the stopped car appears. At t = 5, the light turns green. At time t = 5, the car starts to move. At t = 25 a policeman orders the driver to stop, the car stops immediately. At t = 29, the film ends. (iii) An instant-based description of a temporal relation. The car starts to move when the red light turns green. Then, the car stops and, at the same time, the policeman orders the driver to stop. (iv) An interval-based description of a temporal relation. Initially, the car is stopped for 5 s. The traffic light remains red for 5 s. After this first period, the car moves for 20 s. Then, at the policeman’s request, the car stops for 4 s at the end of the film.

Figure14. Frames representing a real-time recorded film

243

Samuel Pierre and Ha’idar Safa

These are four descriptions of the same film. Each of these descriptions represents a different temporal sequence. They illustrate the way in which a single scenario could be modeled according to several temporal approaches. The representations of the scenario could differ from each other, nonetheless they lead to the same scenario. Figure 15 shows the temporal modeling process where the left side describes the temporal scenario and the right side is the scenario itself. The transformation of the scenario during the presentation can be achieved by means of different temporal modeling techniques. A temporal scenario represents an instance of timely related set of activities. There are two types of temporal scenario: the determined and the undetermined (Perez-Luque & Little, 1995). Determined temporal scenarios represent a set of activities in real time. In this context, the temporal constraints are determined by the physical and real environment; Figure 14 is an example. Figure 16 illustrates a category of undetermined temporal scenarios. The left side of the figure describes the symbols of an undetermined temporal scenario. This scenario is exemplified by a person’s overnight activities such as: opening a door, reading a newspaper, watching TV at dinner time, and sleeping. This scenario is the same every night. However, its performances could vary in time from one night to another. The left side of Figure 16 illustrates two different timely performances of this scenario. In this context, several time processing models are required for such an undetermined case. To capture and describe a temporal scenario, one needs a temporal model. In a temporal scenario, activities are events that occur in time. Such occurrences could be instantaneous or could have a longer duration. In this sense, a temporal scenario is described as an ensemble of independent events (e.g., he arrives at 7:00) or a description that captures some temporal relations between events (for example, he eats while he is watching the TV).

Figure 15. Modeling of temporal information

Figure 16. Example of an undetermined temporal scenario.

Models for storing and presenting multimedia documents

249

In a computerized environment, a technique of temporal representation describes the way in which time is captured and mechanized. Consequently, any representation is the result of the application of a time model. This model is associated with the basic time units such as instant and interval, and with the contextual information specifying the type of temporal information associated with these units (Allen, 1991). 5.3. Synchronization, interoperability and presentation When a user accesses a database in order to read a file, the multimedia database verifies whether or not the user’s site is equiped with the appropriate devices for the presentation and the transfer of data from the server to the user’s workstation. Indeed, traditional workstations are not always suitable for multimedia data processing because most of them are not suited for continous data. To support the interaction between several users, a multimedia application system needs two categories of devices that allow data input and output. Input devices such as microphones, cameras and videos are used for the recognition of movement and speech and output devices such as windows, monitors and speakers are required for the presentation of data. The multimedia system must control these different devices simultaneously and manage the interruptions that are provoked by the users. The remote access to a multimedia system uses the same principle as the simultaneous access. However, in the case of remote access, data is located in several servers and the document is composed during the access process. Therefore, when a user asks for a document, the request is transmitted to the DBMS. Then, the DBMS sends the presentation scenario to the user and to all of the multimedia servers involved in the document. Here, the servers that supply the objects are the same as those that temporarily integrate these objects. Moreover, these servers interoperate in order to achieve the remote temporal presentation of data according to a determined scenario. Such interoperability implies the synchronization of all servers’ operations. Whenever the servers receive the command to deliver the objects at their disposal, they open for the clients several connections supplied with the appropriate characteristics for the transmission of each data flow. Then, the clients indicate the network delay and the other delays associated with the process of data decoding on different servers. Thereupon, the delivery signal is received by the servers and they start the delivery of their objects to the user’s workstation. If a server cannot open a connection because of its transmission characteristics, it informs the planning system which in turn notifies the DBMS. Finally, the work session is terminated. We have also mentioned that, because of the free spaces between the flows and the jitters involved, the network delay sometimes interrupts the continuity of data during the transmission. Such interruption causes several synchronization errors and necessitates compensation at the user’s site. 6. CONCLUSION In this paper, we have presented data storage models and several associated models for presenting multimedia documents. To be efficient, the models defined in this paper must ensure to several users the simultaneous access to multimedia systems. Whereas the access mechanisms to multimedia data depend upon both appropriate software and hardware, the client workstation must be suitable for multimedia data. Since access to documents is influenced by the structure of specified models, it varies according to the documents stored on a single or several remote servers. Although a multimedia application allows users simultaneous access, it is possible only when users read the stored

250

Samuel Pierre and Hdidar Safa

data. Nevertheless, when a user accesses a database in order to modify the content, the other users cannot access at the same time the same database. The analysis of our storage and presentation models revealed their capacity to synchronize various temporal scenarios while allowing users simultaneous and secure access to multimedia systems. REFERENCES Allen, J. F. (1991). Time and time again: The many ways to represent time. International Journal of Intelligent Systems, 6, 341-355. Guidon, J. 8r Pierre, S. (1995). Hypertext and hypermedia for the production and utilization of interactive and distributed documents. Telematics and Znformatics, 12(2), 11l-123. Horak, W. (1985). Office document architecture and office document interchange formats: Current status of international standardization. IEEE Computer, 18(10), 5&60. Meghini, C., Rabitti, F. & Thanos, C. (1991). Conceptual modeling of multimedia documents. IEEE Computer, 24(10), 23-29. Perez-Luque, M. J., & Little, T. D. C. (1995, May). Temporalreferenceframeworkfor multimediasynchronization. Paper presented at the International Workshop on Multimedia Synchronization, Tysons Comer, VA. Rangan, V. P., Kaeppner T., & Vin M. H. (1992, February). Techniques for efficient storage of digital video and audio. In Proceedings Multimedia Information Systems, Tempe, AZ (pp. 68-85). Sinha, A. (1992). Client-Server computing. Communications of the ACM, 35(7), 77-98. Vogel, A., Kerherve, B., Bochmann, G. V., & Gecsei, J. (1995). Distributed multimedia and QoS: A survey. IEEE Multimedia Summer, 10-19.