MMDD { A Distributed Meta-media Simulation Framework Sha Xin Wei, Deborah Zimmerman, Rick Wong, Siew Sim ASD, Stanford University December 6, 1994 Abstract
We describe an experimental system supporting the creation of highly interactive simulations which are rich in media and structure. Strengths include author-modi able schema, author-recon gurable front ends, and relatively straightforward extensibility to new media types and search methods. The framework of meta-media distributed databases (MMDD) manages associations of arbitrary renderable objects (such as structured text, 3D-graphics, musical score les, video, certain executables) using a hybrid object-oriented, relational database abstraction. Key features of the architecture include (1) atoms that are not traditional documents but abstract entities, which endows the MMDD with unusual exibility, and (2) object-oriented core mediating classes which provide rational, extensible meta-data- and content-based services. The MMDD was designed around the projected needs of faculty and their assistant authors, based on extensive experience with creating multimedia simulations. It currently supports several distributed media testbeds in the elds of drama, history, music, foreign languages, and art.
Sweet Hall 415, Stanford University, Stanford, CA 94305.
[email protected]
1
1 Introduction This paper describes a system designed for the analysis and composition of rich media structures. Various relational structures can be imposed on, and operators applied to distributed media of arbitrary type. It can be viewed as a simple databased persistent store to which one can apply services including: relational database engines, grammar-driven indexers, format conversion utilities, natural language lters, and commercial or public domain applications such as WWW hyper-document servers. It is designed to rest atop existing (NFS, AFS, AppleShare) lesystems; accept media without any mark-up of content; and, most importantly, extend in a simple, coherent fashion to arbitrary content-based operators through its object architecture. The core classes include a meta-database, content-based searcher, hyperlinker, media abstracter/ lter, and user interface classes which assemble into widely varying network-based applications. The user interface classes have been constructed as Macintosh XCMD's, Hypercard objects, NeXTSTEP Interface Builder and Objective-C classes. This paper sketches the design, architecture and program of research for the MMDD. In section 2, we discuss the users and tasks we wish to support with our multimedia management and authoring system. In section 3, we list a few design desiderata, followed by a discussion of the architecture in section 4. Section 4 contains a discussion of the principal frameworks of the MMDD: the core meta-database classes, the distributed object server with lter/abstraction services, and the user interface frameworks. In section 5 we brie y indicate related work, and in section 6 we discuss present limitations of the MMDD and directions for future work.
2 Audience and tasks We are placing the MMDD in the hands of faculty and student authors who are integrating multimedia technology into their work. These authors typically have very rich media organized in associative structures which do not map well onto static, linear forms like lm. We have chosen to apply the MMDD in several pilot projects: a History of Silicon Valley which is a growing web of research to be used and extended in a class this winter quarter, a History of Renaissance Theater used by members of the Drama department, 2
and a multimedia messaging board supporting collaborative learning. Other applications are a research and teaching archive of electroacoustic music at Stanford's Center for Research in Music and Acoustics, and an extensible foreign language dictionary service which can be bound to di erent lexica, grammars and corpora.1 A typical use is the composition, close analysis and annotation of a distributed compound document by several people. Another use is the application of content-based analysis and relational modelling operators to any personal or commonly-held media in the distributed lesystem. Some related tasks include entering digitized data, text composition, annotating still images or video with text, or conversely, annotating text with sound, images or video. In general we want to make it easy to associate any piece or stream of renderable media { a blob2 { to another, to view blobs from multiple phenomenological or epistemic perspectives, and to repurpose media across projects. We identify several classes of human users of the MMDD: (1) the browser who makes few permanent changes in the corpora's blob content or topology, (2) the author/designer who may edit corpora or re-arrange the interface in simple ways, (3) a designer/programmer who creates new interfaces and scripts behavior using graphical or textual programming environments. The MMDD provides frameworks which support all these activities. Of course, humans should be able to pass freely from one sort of interaction to another, but we believe that this model clari es the tasks that confront a prospective participant.
3 Some desiderata for distributed media systems Faced with the need to manage and depict compound media in very uid simulation projects based on numeric, algebraic, relational and rule-driven models, we sought a exible media management system which t existing authoring practices and support new research practices in client communities See the Stanford Multimedia Distributed Library report [26]. The MMDD blob model includes at least certain types of interpretable or executable media such as Hypercard stacks or Mathematica scripts. 1 2
3
as they evolve. We also needed a system which would let us migrate client knowledge bases work to other systems as the state of the art evolved.3 We discuss here a few of the desiderata which guided our design.
distinguish between conceptual model, data format, and data depic-
tion, and support maps between each of these ontologies. To illustrate, word-processors typically have a xed depiction { e.g. running text with embedded graphics { allow no control over internal data format, and do not encode models. (Of course this simplicity is also their strength.) GUI frameworks such as Hypercard give a great deal of control over the depiction, but limited control over data format and model. On the other hand, traditional RDBM's which are designed to handle complex relational models typically have xed data formats, and xed UI models. Most importantly, we must let authors dynamically reshape their meta-data schema.
anticipate a moving target in multimedia formats. This has at least two
corollaries: (A) admit multiple representations of the same data, (B) make as few assumptions as possible about the internal structure of a blob; This distinguishes the MMDD model from single-representation schemes such as HTML and HyTime.4 support multiple, concurrent, variable depictions and associations of media. For example, the same simulation of a socio-historical model can be presented via a map on desktop, a spreadsheet on a second, and as a scatterplot of micons ( lled with video) on a third. decay gracefully on low-end personal computers, but take advantage of more powerful services if available. In the example above, the third desktop may enjoy local video codec/transport services not available to the other two.
We investigated a variety of commercial multimedia databases, but found none which satis ed all our requirements at that time. We evaluated FileMaker, Atlas Pro{ a geographic information system, Art Access { a at database aimed at curators of art museums and at wirephoto services, Fetch { a at multimedia database, Oracle, Sybase, and Versant { a robust object-oriented database aimed at large medical, geographic and engineering systems. We also examined some more experimental systems, including Parabase, Gain, and Illustra. A full discussion lies beyond the scope of this paper. 4 We refer to PREMO [19] for some cogent remarks on this issue. 3
4
co-exist intimately with applications running across heterogeneous op-
erating systems. This includes inter-operation with commercial applications which may be supplied by authors. Such unpredictable environments precludes the use of monolithic libraries (qv. [19] for further analysis on this point) but instead assumes some minimal support for distributed objects. be useful to non-technical authors in non-technical endeavors, yet remain extensible to arbitrary media formats, annotation/indexing schemes and search metaphors. Consequently, we chose to store source media and meta-data in traditional systems5
4 Architecture For clarity, we rst give a concise description of the MMDD's media object model, followed by a more leisurely description of the architecture. In section 4.1, we discuss the abstract media entity model, which allows the MDDD to uniformly treat arbitrary renderable data, including structured text, musical score les and video streams. Section 4.2 treats the core classes, 4.3 the Object Server and other services, 4.4 the graphic user interface framework. Our prototype system uses NeXTSTEP (NS) and Macintosh computers, supported by Sun le-servers and an IBM database server, with inter-object communication by Objective-C, a TCP-IP based API, and Distributed Objects.
4.1 Media Object Model
The MMDD media object (blob) model has a simple structure which can be more clearly expressed in precise terms. 5
unix, AFS le storage for source media, Sybase or equivalent RDBM for meta-data.
5
link σ=
blob γ=
blobs database
links database σ1, σ2,σ3...
γ1, γ2, γ3...
γ= σ
σ
γ=
[r]
σ
σ
...
γ=
source media
[r]
Figure 1. The blob structure is stored with references to selections within source media. Hypermedia structures are kept in a links database. This allows multiple, custom structures (such as hypermedia graphs or time-based score les) to be constructed over common media of arbitrary and varying type. The unshaded is a reference to an abstract blob with no associated media.
A typical blob structure (Figure 2) can be viewed as a directed graph f?; gwhere the nodes 2 ? are logical entities representing generalized selections in multimedia objects, and arcs 2 are associations of ordered pairs ( ; ); 2 ?; carrying extra structure. Both ? and carry simple but extensible structure. Each is represented by , where is a a vector of annotations and abstracts in various media types, and points to a selection within persistent source media { for example a le, or more generally, a selection in a blob. The persistent data [r] is actually speci ed modulo an equivalence de ned by the application. In practice the are stored in a relational database while the r are stored in the lesystem. For eciency and to keep the database relatively compact with an eye toward detachable sub-databases, our policy dictates that the sizes satisfy j j
