Also the entire application doesn't have to move; only objects dependent on a .... We see the importance of mobile scripting languages but our view is that ... Section 3 discusses developments at the distributed programming platform level to.
Baltzer Journals
A Framework to Support Mobile Users of Multimedia Applications John Bates, David Halls and Jean Bacon
University of Cambridge Computer Laboratory Pembroke Street, Cambridge UK
In this paper we describe how we have extended our distributed multimedia application platform to support the fact that users may change their physical location whilst continuing to interact with an application. We are assuming an environment of ubiquitous computer deployment. Constructing multimedia applications from reusable active objects leads to ecient and dynamically recon gurable systems. These objects have been made mobile so they can follow users as they move. This work di ers from other approaches (e.g. teleporting or mobile scripting languages such as Java) in that there is support for more than just mobile GUIs. Also the entire application doesn't have to move; only objects dependent on a user's location need follow him/her. We have implemented all concepts discussed and have built working multimedia applications which can support user mobility. In our experiments, we found it appropriate to use compiled objects as media processors and interpreted objects as high level managers.
1 Introduction 1.1 Background and Motivation Early experiments with application support for user mobility, such as the work on teleporting [13] has shown the bene ts of allowing users to change location, either within a room, between rooms or between domains, whilst retaining their application sessions. This frees users from some of the the traditional \sti ness" of computer-based activities. Until now, underlying frameworks have not been able to support complex application types, such as multimedia and cooperative working. In this paper we present a platform which supports complex applications and takes into account mobile users. Results from our previous research provided a starting point for this work. We have built a platform to support the authoring and run-time management of distributed active systems [2, 1]. Examples of such active systems are interactive multimedia applications, cooperative working applications, agent architectures, active databases and applications which respond to changes in people's locations. Instead of building one-o monolithic applications, our platform supports the construction of applications from reusable components. We call these components active objects. Each object can function autonomously, for example by detecting and responding to events.
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
2
Our approach to supporting mobile users is to make our objects mobile. The movement of users can be detected and objects dependent on their location can move with them. We are not looking speci cally at supporting users with portable computers. We are assuming ubiquitous deployment of computer equipment around our environment. We are also assuming a means of locating users and equipment. One way is to use electronic tags and we have considerable experience using the active badge system [8]. It is also possible to use simpler techniques such as determining the location of the workstation at which a user is logged in. 1.2 Related Research This section discusses two di erent research areas relevant to our work, teleporting and language support for mobile programs. Teleporting [13] is a technique to support user mobility by the user interface of an application following a mobile user. In order to \teleport", a user clicks a button on their active badge near any X windows workstation. Their current X session then pops up on that workstation. The teleporting system is based on the technology of the X window system. A proxy X server is able to forward X protocol requests to another server thus providing a level of indirection. The display of X applications can thus move with a mobile user. An active badge database provides a mapping between current location and workstation display. Teleporting is a powerful technique, the main advantage being that it makes any existing X application mobile. This approach also has some disadvantages in that
It is tied to the X system. Only the display connections follow a user; there is no dynamic recon guration at an application level. If it is used with any medium that does not go through X, e.g. audio, then the approach fails.
It can be quite inecient as control messages have to go to the original site and then be forwarded.
The applications do not themselves move, are unaware of their user interface moving and are not built around the premise of mobile users. This means that factors explicitly related to mobility cannot be capitalised on, e.g. the application behaving di erently depending on location.
Our approach provides for the next generation of teleporting, for applications authored with active objects. Recently, there has been a lot of interest focussed on programs which can be sent to run on another machine. We call these REPs - Remotely Executable Programs. Generally, each machine on which such programs can be executed is equipped with an interpreter and a mechanism to send and receive REPs. In order to support mobility, when the user of an application moves, the process is suspended and some or all of the execution state is saved. The process can then be moved to a workstation at the user's new location, the state restored and execution continued. One system currently in the limelight is Sun's Java [6]. Java is a general purpose object-oriented scripting language. It has come to predominance as Java REPs can be downloaded and run in an extended Web browser. Java's disadvantage is that it does not
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
3
support state saving explicitly. There are also many other REP systems currently under investigation, such as Omniware [10], Dreme [5] and Kali Scheme [4]. For a more detailed discussion of these systems see [7]. We see the importance of mobile scripting languages but our view is that applications should be composed of multiple objects, each of which may be running on separate machines. When a user moves, only those objects tied to that user's physical location need to be moved. This provides a more exible arrangement than moving monolithic applications in their entirety. Location Active Database
Room 1
John John’s Management Object
Room 2
David
Locator Objects
David’s Management Object
stream connections
media source and sink objects abstracting physical devices
Figure 1: Mobile active objects 1.3 Active Systems Figure 1 illustrates a simple application which can be built using our framework. The users have video and audio sessions to each other, using video and audio source and sink objects. The locator objects report on user locations. Each application session has a management object which controls higher level functions, such as when to move object components. In this example the following forms of inter-object communication are employed:
Streams - Each object can act as a data processing or monitoring node, continuously examining or adjusting data. It can receive continuous streams of data from other objects using named stream endpoints. In the example, streams are used to connect video and audio sources and sinks. Objects inherit methods to allow third parties to connect or recon gure them.
Events - An application does not necessarily have a sequential ow of control. The event mechanism is a generic way of objects informing each other of asynchronous
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
4
run-time occurrences, including user interaction and temporal occurrences. Each event has a parametrised class which is used in noti cation. Objects inherit methods to register and deregister interest in event occurrences. In our example, a management object can register interest in events of class Location("John",R) with a locator object. It is noti ed asynchronously if John changes location, for example Location("John","Room1").
Method invocations - The standard technique by which an object can invoke a method de ned in the interface of another. The noti cation of events can trigger rules to perform invocations on other objects. For example a management object can be set up to move objects associated with John when it gets an event that John has moved.
Our latest work supports the notion that if a user moves then the active system can recon gure itself to the user's new location and the users can continue as if nothing had happened (see gure 2). This recon guration is achieved through the ability of objects to move from location to location followed by reinstantiation of the communication status between them. Only the objects directly related to a location change have to move. 1.4 Overview This paper describes the design and implementation details involved in supporting mobile users within active systems. Section 2 outlines some decisions on how to implement mobile active objects. Firstly it discusses the rationale behind the two main types of mobile object: media objects and management objects. It also outlines why the support for movement is implemented in user space and discusses the relative advantages of using compiled and interpreted approaches in building active objects. Section 3 discusses developments at the distributed programming platform level to o er powerful facilities for mobile active object construction. These include an enhanced object speci cation language with named and typed events and stream endpoints. Tools and libraries have been added to support these enhancements. Section 4 describes how existing classes of media objects are organised. Section 5 discusses how dynamic requirements, such as prototyping a management object, are supported by interpreted objects. Section 6 outlines services required for naming and location of people, equipment and objects. The issues discussed here are how to track users, how mobile objects can be mapped onto equipment near these users and how mobile objects can be located for other objects to bind to. Section 7 describes how we are investigating the construction of higher level services and authoring to support exible automated movement. Two relevant areas are dynamic QoS adaptation within applications and moving objects relevant to a user's location without explicitly programming the move. All of the concepts described have been thoroughly investigated by realistic implementation. Section 8 shows how the construction of working applications has proved that mobile users can be supported in a multimedia environment. Section 9 describes how we envisage our work evolving in the future. We outline our conclusions in section 10.
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
5 Room 3
David’s Management Object
Location Active Database
Room 1
John
Room 2
David
John’s Management Object
Figure 2: Mobile active objects II
2 Mobile Active Objects 2.1 Object Roles As discussed, our view of an application is a set of objects connected together. Our framework strongly encourages the use of existing classes, however, a glue is required to connect the objects and to use run-time events from one object to trigger invocations in another. Thus there are two levels of abstraction:
media objects - existing objects classes, instances of which can be plugged together. Our architecture assumes a class hierarchy of media objects in each domain.
management objects - the bespoke aspect of an application, responsible for connection, event-handling and movement. Management objects are usually on a per user basis and can communicate in a cooperative application.
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
6
Referring back to the mobility scenario of gures 1 and 2, each user's management object is responsible for setting up the con guration of media objects. In order to move a con guration, a management object saves the state of all media objects which must move, transports itself to the new site and reinstantiates the objects there. If David's locator object noti es his management object that he has moved to another room, the management object saves the state of the video and audio source and sink objects (which are dependent on David's location), moves itself to the new room's workstation (after database lookups) and reinstantiates the sources and sinks. 2.2 User Space vs Operating System There are two levels at which support for mobile objects can be provided:
Within the operating system - process migration.
Within the object framework - script-based object migration or state saving and reinstantiation.
Process migration is a well-established technology which allows a running process to be saved, and restored on another computer. Some systems [12] even support the transparent rebinding of communication endpoints. We have not employed such systems to move our objects for the following reasons:
They either make assumptions about the operating system on which the program is running or are too inecient.
In our investigations we found that lightweight script-based objects are more suitable for experiments on mobility. These objects can be used to save and transport any necessary state including the communication state of compiled objects.
2.3 Compiled vs Interpreted An interesting issue is the compromise between using compiled and interpreted code in both the implementation of media objects and in higher level management objects. The bene t of compiled objects is that they are ecient. One bene t of interpreted code is that it is more portable. It can be run on any machine since the source code is used directly instead of a compiled version. Also, it can be more ecient in a prototyping situation since runtime changes do not have to be recompiled. Both types of object are useful and we have thus made provision for both in our framework. We have found that compiled objects naturally ful ll the role of media sources, sinks and processors. These objects have little requirement for dynamic behavioural change in that they fetch media, possibly perform some function on it and then expel it. They must have methods to control this processing. These objects bene t from being compiled in that they are ecient. They also need to save little state (often only communication state) when they move; they usually perform some continuous memory-less function. They do not need the dynamism of an interpreted approach. In most domains the systemindependent characteristics and availability of such media objects will be similar. REP objects can be used to prototype the functionality of a new class of media object but
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
7
overheads such as interpreting code and garbage collection often add excessively to already computationally intensive tasks, e.g processing continuous media. We found that REP objects ll the role of high level con guring entities. This role involves dynamic changes within an object if it, say, enters a new environment. It may even involve some AI techniques. It is also likely to involve the saving and reinstantiating of all object state.
3 Extending Distributed Programming To support the construction of mobile active objects we have enhanced a distributed programming platform. Our starting point was the ODP-ANSA model [9] which provides object speci cation, remote invocation and interface trading facilities. We assume a multithreaded implementation environment. The programming system we used as the basis of our prototype implementation was developed in Cambridge and is called MSRPC2. The DEC Pthreads package was employed to provide us with kernel threads. Video Sink Object
Video Source Object
vidsink Stream Endpoint
vidsrc Stream Endpoint
Video Stream
Video Source Object vidsrc Stream Endpoint
VideoSource: DEVICE = BEGIN vidsrc: PLUG VideoStream; OPERATION Play(..); OPERATION Stop(..); END
Video Stream
vidsink Stream Endpoint
Video Sink Object
VideoSink: DEVICE = BEGIN vidsink: SOCKET VideoStream;
IDL Definitions
OPERATION Move(x,y:INTEGER); OPERATION Resize(w,h:INTEGER); Click:EVENTCLASS(); ClickArea:EVENTCLASS (x1,x2,y1,y2:INTEGER); END
Figure 3: Extending OODP with stream endpoints and events 3.1 An Extended Object Speci cation Language It is useful to declare explicitly stream endpoints and event de nitions, along with the usual types and methods, as part of an object's speci cation. This provides a speci-
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
8
cation of the object's facilities to its clients. It allows a third party to connect two stream endpoints together. Also, naming every communication endpoint makes it easier to re-establish communication channels after movement. A trading service can be used to re-locate named communication endpoints. For example in gure 2, when David's video window object moved it was necessary to reconnect to John's video source. In the meantime John could have moved. A trader lookup using the object and endpoint name can reveal the most up-to-date physical location of the stream endpoint. Our model assumes objects keep the trader up-to-date on their location. We have extended an ODP-compliant object speci cation language to allow named event classes and named stream endpoints to be declared. An example is shown in gure 3. This illustrates how the video source and sink object can be declared. In order to provide programmers with access to the enhanced object facilities it was necessary to build libraries and tools and integrate them with the programming environment. Stub generation tools were extended to generate the code necessary for marshalling, unmarshalling and binding associated with events and streams. We also built stream, event and communication state libraries which are described below. 3.2 The Stream Library The stream library implements generic stream management facilities. Operations are provided so that third party objects can connect two named objects together with reference to typed stream endpoints. For example a management object can connect a vidsrc endpoint to a vidsink endpoint. Endpoints can read and write typed data down a stream, for example a video source will write video frames and a video window will read them in. Endpoints are capable of multicasting when writing to multiple clients. Various types of stream are available depending on transmission reliability and eciency requirements. QoS parameters can adjust the characteristics of a stream on-the- y. Named streams are also supported in this model. This allows any authorised third party object to send and receive to/from a stream. For example, it is possible to write some data of the correct type (a video frame) to vidsrc of John's video source. Whatever streams this endpoint is connected to will receive the data. The bene t of the stream library to mobile objects is that they support the suspension and resumption of data transmission. When one end dies, the other end automatically freezes the stream for that connection. Reconnection can be performed when the object has relocated. The example shown in gure 3 shows two video source objects connected to two video window sinks. This example illustrates how video front and back ends can be implemented as separate objects. On the workstation itself, there is only one video source- the camera. A low level object is responsible for retrieving live video from the camera. However, there may be more than one person at a time at that particular location. Thus there will be more than one video source object there. No one person can \own" the camera since they may move leaving others still in the location. Thus \front-end" objects can be used to abstract the many-to-one relationships between people and hardware devices such as cameras and microphones. Movement is more easily managed since these objects contain no hardware-speci c code. When they arrive at a new location they must be attached to the back end object representing the physical camera devices. The logical video sources can exist in separate address spaces from the physical video sources and receive video via
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
9
streams or can run in the same factory site and use shared memory. 3.3 The Event Library Objects inherit methods for registration and deregistration of events. The event library provides generic implementations for these methods. Events are transmitted in a generically marshalled state and thus they can be sent to any other object, even if the class is not immediately understood. Stubs are automatically generated for the marshalling and unmarshalling of event instances by event stub generation tools operating on an object interface. The event library calls the relevant stub to encode or decode an event instance. Site A
Site B
John:VideoSink
John:VideoSink
Registered Events
Registered Events
Click() -> John Management Object
Click() -> John Management Object
Stream Connections
Stream Connections
VideoSource From David Video Source
Save Communication State Destroy Object
VideoSource From David Video Source
Communication State
Create Object of Same Class Instantiate Communication State Rebind Connections
Figure 4: Saving of object communication state 3.4 The Communication State Library All of the state associated with an object's communications can be accessed by an authorised third party. This enables an object to be suspended and restarted elsewhere without any heavyweight migration facilities. A third party keeps the state of its communications and uses it to reinstantiate the object elsewhere. Objects inherit methods to allow third parties to access their state. Figure 4 illustrates the principle of saving and instantiation of communication state, including streams and registered events. To implement media objects, state, including communication state, can be made accessible to a third party via methods. In the case of management objects, state includes everything associated with the running process' status and special facilities are required to enable this to be accessed (see section 5).
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications RegisterEvent UnRegisterEvent
....
Filtering Object
Media Object
Display Object
Monitoring Object
10
Source Object
Sync Object
....
play() stop()
Video Source
StoredVideo Source
MPEG
Mips-Ultrix
Audio Source
Picture Source
LiveVideo Source
JPEG
Linux
Text Source
System Independent Pandora
System Dependent
Alpha-OSF
Figure 5: Part of a media object class hierarchy
4 Media Objects For a media object to \move", the old object's communication state is stored by a third party, an object of a compatible class is created in the new location and the communication state is used to instantiate the new object. The old object is destroyed or suspended. A hierarchy of active object classes (see gure 5) provides the model for recreation of objects at new sites and hence \movement". To support the transition of objects between domains a distinction is made between system-independent and system-dependent classes. Consider the example of a video source object. With reference to gure 5 one can see that the system independent video source object is generic. It has methods such as play and stop, a stream endpoint on which it can multicast its video and events in which interest can be registered. However, there are various system dependent sub-classes of this class depending on the operating system and hardware combinations which are available on a particular host. The framework supports the movement of such objects by making the appropriate system-dependent selection for the user. The user can use the system-independent facilities without being aware of system speci cs. For example if John moves from his oce where he is using an Alpha and jvideo card to Tim's oce and starts using a Linux workstation with a di erent video card his video source object will still be able to follow him. The mechanisms to realise this are factories and a con guration service. Factories are sites at which objects of certain classes can be created. For instance, on every workstation with live video facilities there will be a factory capable of creating video source objects. When creating objects, a factory can either spawn them as independent processes or
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
11
manage them within its address space. A con guration service has knowledge of these factories and which classes they can create. By posting a request for an object creation in a speci c location, the appropriate system-dependent class will be created. If there is not an exact match to the required object at a speci c site then translator objects can be employed. For example if a JPEG video source is not available but a Pandora video source is, then a translator from Pandora to JPEG can be inserted into the con guration by the con guration service. The con guration service can also be used to connect stream endpoints together, assuming compatible typing. Objects to other Tube sites
Objects from other Tube sites
Tube Site Active Object
Active Object
BigLoo Interpreter Lisp Stub Compiler
MSRPC2 Platform
Figure 6: A Tube REP site
5 Interpreted Objects This section introduces the Tube, our infrastructure for the management, execution and transmission of REP-based objects. We elected to use our own REP system rather than one of the publically available systems, such as Java. The Tube is more suited to our work on mobile objects for the following reasons:
It uses Scheme as the basic language of implementation. Scheme is an ideal language for mobile code because of its program/data duality. Since both are implemented as lists, it is easy to create data from REPs and REPs from data. The Tube also uses a continuation-passing meta-circular evaluator so that a REP's running state can be captured at will: a computation can easily be stopped on one machine and restarted on another.
We are able to change it at the lowest level because we wrote it. We have thus been able to link it with our extended OODP system and compiled active objects (see below).
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
12
Each machine in our experimental system runs a software service called a Tube Site (see gure 6). These sites listen on sockets for the arrival of objects. An object is transmitted as a marshalled Scheme expression. When an object arrives, it becomes \active". Our prototype implementation of Tube Sites is based around the Bigloo Scheme compiler/interpreter [15]. Additional functionality that has been added and compiled in is described below.
Low-level communication facilities Unix sockets, les, redirection of input and output are accessible from scripts. A generic addressing scheme is provided; addresses are typed and contact and resolution procedures are associated with each type. In this way, the act of communication is removed from how the communication is initiated and maintained. Input and output functions are overridden so that type-speci c procedures can be installed (to handle errors, for example).
Marshalling of s-expressions Any Scheme expression can be marshalled into a string of characters. REPs are marshalled at the sender and unmarshalled by the receiver before execution.
Multi-threading using the underlying operating system REPs that arrive at a site are each executed in a separate thread and in a separate environment. Each site is actually arranged as a number of server threads which interact with each other and with arriving REPs. A thread-safe version of Boehm's garbage collector [3] is being used.
The abstraction of a \farm" Each farm (server) accepts work packets and guarantees to process them and produce results (synchronously or asynchronously). How each farm achieves this is not known by its clients; it can be implemented serially or in parallel using a number of threads.
Prede ned farms to handle the reception of REPs, outgoing communication and jobs submitted by REPs themselves.
A table of data references This is global to a site and is where REPs share information. Areas within the table can be created and destroyed, and values posted and retrieved. Locking of the table is handled automatically unless overridden; there is an access control list associated with each area.
Authentication of REPs A REP can optionally be signed using public key encryption, in which case the receiving site will tag the thread it runs in with the signee. This will then be used in limiting its access to the data references table and to the built-in Bigloo symbols.
A continuation-passing meta-circular evaluator This sits on top of Bigloo as another layer and evaluates Scheme programs. It is for REPs which require the capture of their dynamic state (i.e. which might be stopped,
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
13
shipped somewhere else and then restarted from where they left o ). Something like the abstract continuation-passing style [14] is used to convert Bigloo lambda objects (which cannot be marshalled) into a form suitable for marshalling. Dynamic loading of object les and shared libraries This enables modules to be plugged in at run-time; scripts can arrive at a monitor site and link to domain-speci c facilities. Parts of object components which require access to speci c facilities can thus be written to exploit interpretation. A class hierarchy of shared libraries is required so that the naming of library components to link is recognised across domains. An interface to the XForms GUI library Requests made by client REPs to the GUI server thread are serialised. Published variables, which are tied to speci c sites and enable roaming computations seamlessly to refer to the same value instance from anywhere. Support to allow Scheme scripts to use the extended MSRPC2 facilities has also been added. REPs can performs RPC and use the name resolution facilities of the trader. Stub generators from MSRPC2 to Scheme have been developed. It is also possible to send Scheme programs via RPC. A novel RPC system has been built on top of the Tube itself. This allows for exible and parametrised client-server interaction. A client requests a proxy object from a server by specifying some constraints and then communicates with the server via it. The proxy is sent from the server as a REP and sets itself up and installs itself at the client. The concrete methods for client-server interaction are therefore hidden in the proxy, which can also be replaced dynamically by di erent implementations delivered from the server. Work is underway to link Scheme objects with a Tube-RPC interface seamlessly to static MSRPC2 IDL les. Our initial experiments (described later in section 8) with Tube are encouraging. Part of our current work is adding higher level functionality above the raw Scheme model. We are designing a language which can be used to build programs that can arrive at any REP site and organise an application session. Instead of the whole world having to conform to a particular classi cation schema, REPs (with some intelligence) can be sent to unknown domains to set up part of a remote con guration of objects. Also, if users go to a new domain then their application can still follow them. Tube sites share a naming and location model with other active objects, as described in section 6.
6 Naming and Location Our naming and location model supports the tracking of physical objects, such as users. Mobile objects can be made to follow these users through the ability to retrieve details of physical hardware near to user locations, onto which the objects can be moved. The set of mobile objects associated with a user are named in terms of that user. This enables us to perform operations such as connecting to \John's video source" and nding out \the location of David" rather than using some internal machine referencing scheme. This is
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
14
also leading on to building applications which can be instantiated for particular users and in which objects are \owned" by people and exist in \places". 6.1 A Model of Naming and Location The rst requirement is a model of physical location, e.g. \Jean is in room T13" or \the workstation Berg (which belongs to John) is in room T12". Models for location and containment are being investigated [11]. We have used a simple model, dividing our world into rooms and parts of rooms. Each location has a real world name, as in the examples. We also named objects with real world names such as the person \David Halls", or the workstation \Britten". Active objects can be physically located on workstations (i.e. they are running on that machine). They can also be bound to people, e.g. \David Halls' location object". Peripherals dependent on a workstation, such as cameras and microphones, are named as belonging to that particular workstation, e.g. \Britten's camera". 6.2 Location Objects Having de ned a basic model of naming and location based around people and places it is possible to develop objects which monitor the physical location of a particular person. These objects abstract locator devices, such as active badges. Interest in various movement events can be registered with such objects. For example interest can be registered in Location("David Halls",Loc). Whenever the location of David changes, his location will be noti ed. In our prototype implementation we have developed badge objects which can report on changes of location of the individuals they are monitoring. These badge objects are managed within an active badge database. Meta information on location is stored in a separate location information database, described below. Our future work will allow badge databases in multiple domains to interoperate so it will be possible to get the location of any user by making a request to one's local service. 6.3 Location Information Database We have developed a prototype active database service to store location information. This is used in various mobility situations, such as when David's management object receives an event that David has moved to oce T14, it wants to nd out the nearest workstation to his location to move his display objects to. Also if Jean wants to look at David then she must connect a video window to David's video source. This will require a lookup of David's location and of the nearest video source object to him. The following types of information illustrate what can be stored in the database:
The names of people and the facilities for locating them that are available. For example, if they have an active badge then the badge ID can be stored.
The names of rooms and their geography, i.e. how they interconnect. This can be used to ascertain things like \which room is next to John's oce?"
The names and capabilities of pieces of equipment. For example, workstation Berg
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
15
has camera, microphone, screen and speaker. The classes of object which it can support are stored (see section 4).
The equipment stored in each room. This information can either be static or equipment can be tagged. Using this information it is possible to determine whether a speci c room or part of a room o ers a certain facility, e.g. video.
6.4 Use of the Trader The MSRPC2 trader is used as a location and binding forum for mobile active objects. In the ODP-ANSA model, attributes can be stored in the trader with object references. This provides a limited database lookup service to assist in nding objects of interest. Our objects are stored with attributes of ownership and location. For example, to register interest in an event with Jean's video source object rst requires the interface reference (an encoding of its physical location) for this object. This can be retrieved from the trader by asking for an object of class VideoSource with attribute owner set to \Jean". Similarly, interfaces for the binding of stream endpoints are all stored in the trader. When an object moves it can rebind to the objects it was previously attached to by reacquiring their system-speci c address by name from the trader. For example, when John moves, a reference to the video source object at his new location must replace the old reference in the trader. Thus any object which subsequently binds to it will have the new address. This system relies on objects keeping the trader up-to-date on their location. Obviously race conditions can occur due to communication delays. These can be detected by communication timeouts. It is also possible for objects to notify events when they move, in order to force other objects to retrieve their new location.
7 Managing Applications This section describes several additional services which are being built to assist in the development of management objects. These are in the form of libraries and extensions to existing services, to enhance the facilities available to developers of management objects. 7.1 Automating Object Movement Not every object has to move when a user moves. Those associated with the physical location of a user such as window or camera objects must move but others, such as personal agents need not. As described so far, object movements have to be programmed explicitly. Based on object classes, it is possible to generate policies to specify which objects should move under certain circumstances. For example, when a user moves, all his/her user interface components must go with him/her. Thus every object class derived from class display component can be marked in this way. Likewise, whenever John moves and he is using video then his video source object must move with him.
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
16
7.2 Adaptive Policies A trend in modern networked operating systems is support for Quality of Service (QoS), i.e. the system can guarantee a portion of its resources to a process. As objects move around, it is possible that the hardware onto which they are mapped varies, e.g. moving from a machine on a fast network to one on a slow network. Rules can be encoded to respond to the QoS negotiation process. For example, if the network bandwidth is inadequate to support full motion video with audio then the video can be turned o or the frame rate cut. 7.3 Language Support As part of our multimedia platform research we have developed a language to allow the con guration of objects and the speci cation of event-driven rules. This language is useful in the construction of management objects. We are enhancing it so objects can be instantiated for particular users and each instance can communicate with dynamic numbers of other instances.
8 Experiments 8.1 Experimental Objects We have enhanced various media object classes for mobility and used them in experiments, including the following:
Video source and video window
Audio source and sink
Drawing board - Each user has a drawing board which they can use their mouse to draw on. Other users can register an interest in drawing strokes and can be noti ed of them. Event-driven rules can be used to trigger local drawing when noti ed of another user's drawing strokes thus creating a shared drawing environment.
Text I/O object - In a similar way to the drawing board, users can register an interest in other users typing text into their text box. This can be used to create a text conferencing facility.
8.2 The Experimental Framework Figure 7 shows that in various rooms we have multimedia workstations, each of which is running a suite of software. Firstly there is a REP site running on each machine. This enables interpreted objects to be sent to any machine. Each machine also runs factories to allow new objects to be instantiated. The diagram shows a display factory which can manage the creation of window-based objects, including video and audio sinks, text I/O objects and drawing boards. It also shows video and audio factories on which video and audio sources can be created for each person present at that location. These objects
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
17
Equipment within rooms
Location Information Database
Location Active Database
Configuration Service
Trader
Management Object
Tube Site Audio Source
Display Factory
Video Source
Audio Source
Audio Factory
Video Source
Video Factory
Video Source
Figure 7: Experimental Framework are independent of the backend object and can move. The diagram also shows higher level services used to manage mobility, including location databases and con guration and trading services. Our initial experiments have been constructed on DEC Mips-Ultrix and Alpha-OSF architectures. We are currently developing a PC-Linux implementation. 8.3 Prototype Applications Our platform designed to support a wide range of mobile application types. We have built applications ranging from simple demonstrations, such as \follow-me" video, to more complex applications involving multimedia collaboration. By combining components of di erent media into one application, users can collaborate via true multimedia facilities. The strength of the framework is that it is very easy to prototype a new application using pre-existing components. The potential bene ciaries of this work are mobile professional people who require the use of computing equipment at many sites both within a building and between buildings. These include IT professionals, medical workers, academics and oce managers. Using mobile applications removes the need to log in and initialise a working context at every
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
18
location, as well as maintaining location-independent consistency of access. It also has the advantage that the system makes the best use of available equipment and system resources in a particular location.
9 Future Work We are currently working in several related areas including the following:
Support for storage and retrieval components We are building objects to store and retrieve media associated with mobile users, e.g. video and audio storage objects, to record all streams from live collaborative sessions.
Live indexing and retrieval of cooperative, mobile sessions Building on top of our storage architecture, a novel indexing approach is being extended to allow retrieval from complex multi-party multimedia sessions. All of the media will be stored and indexes constructed. Powerful querying will then be available, for example \retrieve me the picture Ken drew after Jean gave a presentation about owers".
Higher level language for mobile cooperative applications As already introduced, we are interested in extending our existing multimedia authoring approaches to build applications explicitly around mobile users.
10 Summary and Conclusions We have described a framework for supporting the construction of mobile multimedia applications. Our previous work has been to design application development paradigms which address emerging application classes. The building blocks for these applications are active objects. These can be connected together by streams and can communicate by invocation and event noti cations. Higher level services and languages can be used to build complex active systems out of them. In this work we have shown how we have enhanced our original object design to support the fact that objects within an active system can move to support user mobility. We distinguish between media objects and management objects. Media objects represent media sources, sinks and processors. Well-known classes of media objects can be created using factories, running on workstations which can support the relevant functionality. Management objects provide the glue to plug media objects together and manage run-time interaction and movement. In order to move a complex con guration of objects, a management object saves their state, moves itself and reinstantiates them at the new location. We have investigated both compiled and interpreted approaches to mobile objects. Compiled objects work well as media objects and interpreted objects work well as management objects. Mobile objects are speci ed using an enhanced object speci cation language. Stream
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
19
endpoints, methods and events are all named. The system-speci c aspects of the current location are exported to a trader. A system-independent name can be used to get the upto-date physical address with which to communicate. We have extended the facilities of a distributed programming environment to support streams which can be suspended and reconnected as well as event registration and noti cation. The communication state of an object can be saved for later reinstantiation of a clone. The Tube REP system allows the executing state of an interpreted object to be captured, for it to be transmitted between sites and for its execution to be restarted at the new location. A mobility support framework allows objects to be named in terms of people and physical locations. The locations of real world objects are determined by locator objects which abstract location devices such as active badges. Information about where equipment is located and what media functions it can support are stored in a database. We have successfully evaluated our approach by building and experimenting with multimedia applications which can support mobile users. Wide deployment of this paradigm will enable mobile professionals to carry their computing environments with them anywhere in the world. Acknowledgements We would like to thank the EPSRC for supporting this work through its studentships and under grant GR/J42007. Dr Bates is grateful for the support of Michael and Morven Heller and St Catharine's college, Cambridge. Thanks are due to Giles Nelson and C.K. Toh for their expertise on mobile matters and to Ian Piumarta of INRIA for assistance with garbage collection. We are grateful to ICL for general support of our research group.
References [1] J. Bacon, J. Bates, R. Hayton, and K. Moody. Using events to build distributed applications. In 2nd International Workshop on Services for Distributed and Networked Environments. IEEE, 1995. [2] J. Bates and J. Bacon. Supporting interactive presentation for distributed multimedia applications. Multimedia Tools and Applications, 1(1):47{78, March 1995. [3] H. Boehm. Space ecient conservative garbage collection. In Proceedings of the ACM SIGPLAN '91 Conference on Programming Language Design and Implementation, pages 197{206. SIGPLAN, 1993. [4] H. Cejtin, S. Jagannathan, and R. Kelsey. Higher-order distributed objects. ACM Transactions on Programming Languages and Systems, 17(5):704{739, September 1995. [5] M. Fuchs. Dreme: For Life in the Net. PhD thesis, Department of Computer Science, New York University, September 1995. [6] J. Gosling and H. McGilton. The Java language environment. Technical report, Sun Microsystems, 1995. [7] D. Halls. Remotely Executable Programs. Internal Report, University of Cambridge Computer Laboratory, June 1995. [8] A Harter and A Hopper. A Distributed Location System for the Active Oce. IEEE Network, January 1994. [9] Architecture Projects Management Limited. Advanced Networked Systems Architecture Testbench Implementation Manual, 1993.
J. Bates, D. Halls and J. Bacon/ Mobile Multimedia Applications
20
[10] S. Lucco, O. Sharp, and R. Wahbe. Omniware a universal substrate for web programming. World Wide Web Journal, 1(1):359{368, December 1995. [11] G. Nelson. System Support for Location Aware Computing. PhD thesis, University of Cambridge Computer Laboratory, 1996. In preparation. [12] S. Pope. Application migration for mobile computers. Submitted to The 3rd International Workshop on Services in Distributed and Networked Environments, 1996. [13] T Richardson, F Bennett, G Mapp, and A Hopper. Teleporting in an X Window System Environment. IEEE Personal Communications, August 1994. [14] David De Roure. QPL3 { Continuations, Concurrency and Communication . Technical Report CSTR 90-20, Department of Electronics and Computer Science, Unive rsity of Southampton, 1990. [15] M. Serrano and P. Weis. Bigloo: a portable and optimizing compiler for strict functional languages. In Second International Static Analysis Symposium (SAS95), 1995.
