Communication Support for Distributed Collaborative Applications Raj Yavatkar and K. Lakshman Department of Computer Science University of Kentucky, Lexington, KY 40506-0027 fraj,
[email protected] To appear in the ACM/Springer Verlag Journal on Multimedia Systems, Vol. 2, No. 2, 1994. July 19, 1994
Abstract The Development of distributed, multimedia, collaborative applications requires resolution of communication issues such as concurrency control, and temporal and causal synchronization of traffic over related data streams. Existing transport and/or session layer protocols do not include desired support for multi-stream, multipoint communication. In this paper, we propose new communication abstractions and mechanisms that facilitate implementation of the necessary coordination and concurrency control semantics in a collaborative application. We also propose a protocol suite called MCP (Multi-Flow Conversation Protocol) for realization of these abstractions and describe its prototype implementation in an internetwork of workstations. The paper also describes our experience with the prototype and results of a performance evaluation.
KEYWORDS: multimedia communication, distributed collaboration, temporal and causal synchronization, Unix networking.
This research was supported in part by the National Science Foundation Grant No. NCR-9111323 and Grant no. STI-9108764.
1 Introduction 1.1 Motivation Distributed collaborative systems allow a geographically dispersed group of users or applications to use computers and multiple media (text, audio, video, and images) to solve problems together. An example of such an application is a collaborative, software engineering environment [SFB+ 87, AEHL88]. In such an environment, a group of designers located at different sites collaborate on a design document (or a program) using interactive tools to edit and test parts of the design under development. Interaction may involve a group editor (based on shared windows), an image display (that displays resulting design), and a voice (and video) channel that allows them to view, discuss, and edit the suggestions made by each other. Moreover, the interaction may not necessarily involve human users, but may also involve cooperative programs that work together in automating a particular task [DL87]. Development of such collaborative tools requires research in several areas including user interfaces, communication infrastructure, collaboration paradigms, group toolkits, and shared object systems [PD93]. In this paper, we focus only on the communication aspects of distributed collaborative systems. Communication in such applications involves several, related, multipoint streams. Each stream carries data specific to a particular medium and is shared among several geographically dispersed participants. Because several participants share one or more streams, some form of concurrency control is desired. In addition, data sent over separate streams are typically related in time and context, and such relationships must be preserved in delivery of data at each recipient. Thus, distributed collaboration using multiple media requires resolution of coordination and control issues such as concurrency control, and temporal and causal synchronization of traffic over related data streams. Currently, conventional transport and session layer protocols do not include desired support for multistream, multipoint communication. Instead, collaborative application designers must build their own mechanisms to achieve desired semantics. This approach has several disadvantages. First, lack of suitable abstractions requires designers to concentrate on low-level communication aspects instead of focusing their attention on investigating higher-level issues such as collaboration paradigms, flexible toolkits, and user interfaces. Second, application layer solutions for coordination and control must use an existing transport protocol such as TCP(transmission control protocol) or UDP(user datagram protocol) designed only for a 1
single communication stream between two parties. Therefore, application layer implementations are not efficient and require duplication of some of the transport and session layer functions including flow and error control, buffer management, asynchronous event handling, and timer management. Third, lack of standard interfaces and abstractions leads to collaborative systems that are not inter-operable. In contrast, distributed applications based on traditional client-server model utilize well-defined communication abstractions designed for two-party communication. Necessary semantics are efficiently implemented in transport and session layer protocols allowing application designers to concentrate on higher level issues. Clearly, there is a need for similar functionality and support from communication protocols to facilitate development of collaborative multimedia applications.
1.2 Our Contributions In this paper, we investigate design and implementation of suitable communication abstractions and mechanisms to allow realization of the necessary coordination and control semantics in a collaborative application. In particular, we have designed a new protocol called Multi-Flow Conversation Protocol (MCP) that provides two communication abstractions: First, MCP includes a token-based mechanism for concurrency control among participants of a multipoint connection. The degree of concurrency control achieved varies from a strict “floor-based” control at one extreme to no control at the other extreme. Second, MCP provides a communication abstraction called a multi-flow conversation to allow temporal and causal synchronization among traffic over multiple, independent streams. A conversation may consist of one or more (two-party or multipoint) connections and provides both temporal and causal synchronization in delivery of messages sent over participant connections. Temporal and causal ordering in a conversation is restricted based on a notion of delay-constrained or ∆causality that takes into account the deadlines on delivery of multimedia data. Additional features of MCP include an out-of-band signaling channel for exchanging control information and an asynchronous interface that allows an application to specify event-triggered actions to facilitate fast processing of periodic data and changes to the status of multipoint connections. MCP is an upper level protocol designed to support a wide variety of collaborative applications. We have implemented it on top of both a connectionless inter-network layer (IP) as well as over a connection-
2
oriented internet protocol (Co-IP). We have conducted a systematic performance evaluation of the MCP kernel implementation and have also compared the performance of user applications with and without MCP. Our experience with MCP has revealed the utility of its abstractions in implementing collaborative software. It also demonstrates the performance gains achieved by designing a new protocol instead of implementing the MCP functionality at the application layer. The rest of this paper is organized as follows. Section 2 presents the background motivation for our research. In Section 3, we describe the proposed communication abstractions. A protocol framework for implementation of the abstractions is proposed in Section 4. Section 5 describes a prototype implementation and Section 6 presents the results of a performance evaluation. In Section 7, we compare our work against other relevant research, and finally, Section 8 concludes the paper.
2 Coordination and Control Problem In the design of multimedia collaborative systems, one must address three coordination and control issues: Concurrency Control: Multimedia collaboration typically involves one or more multipoint flows1 . Each flow involves several participants. Each participant may either be an application program (such as a design tool component) or a human user (e.g. a voice conferee). When several participants may concurrently transmit data over a shared flow, the need for concurrency control arises to ensure consistent views and smooth interactions. Examples of such control include avoiding interruptions and crosstalk in a multipoint voice channel [Sch92, CLSF93], or maintaining consistency of a shared text document or an image [Dew90] in a collaborative application. As explained in Section 2.1 below, the desired degree of concurrency control varies over a wide range depending on the application or the context. Temporal Synchronization: A multimedia application needs to synchronize traffic over multiple flows, where each flow carries traffic from a different medium (text, voice, video, or image). A collaborative, software development group effort described earlier (in section 1.1)is an example of such an interaction. For such an application, the communication software must allow related streams to be 1
We refer to a two-party or a multipoint connection as a flow [Zha89, FV90, CY89] throughout this paper. The notion of a flow assumes certain performance guarantees on delays and bandwidth as described later.
3
grouped together, recognize the order and sequencing of messages sent over them, and preserve such ordering in delivering messages to each participant. For example, when a participant in a group editor scrolls through an image browser (or a shared editor window) and says, “look at the middle of the display”, the statement should be heard at the same time (or just after) the scrolling is completed. Such temporal relationships must be captured in the delivery of traffic over related streams. Another example of such a synchronization is “lip-synching” when voice and video are transmitted over separate flows in a network. Causal Synchronization: When collaboration involves multiple participants communicating either over a single flow or using multiple flows, causal relationships among messages sent over the flow(s) must be maintained to preserve the context in which a message is sent. Consider the case of a voice conference where a single flow is shared among multiple participants. When a participant B responds with message M2 after hearing message M1 from another participant A, a third participant C must receive and interpret messages M1 and M2 in the proper causal order to make perfect sense. Maintenance of such causality is especially important when interaction involves programs (rather than human users) updating a shared document or image. Moreover, in collaboration involving multiple media, such causal relationships exist among messages sent over separate flows. Consider a collaborative design editor involving two flows: an image flow to carry messages containing drawings and updates of shared images, and a voice flow for conversation among participants. In such an interaction, if a participant scrolls through or modifies the shared image (resulting in a message update over the image flow) in response to a voice message from another participant, then a causal relationship exists among messages sent over two different flows. Again, as explained below, the desired degree of causality varies depending on the media involved and the application.
2.1 Degree of Coordination In the following, we describe how different applications and contexts require a varying degree of concurrency control and temporal/causal synchronization.
4
Concurrency Control: The degree of concurrency control needed varies from one application to another. In addition, an application may go through several phases of interaction and each phase may need different degree of concurrency control. At one extreme is the case of a shared window package [SFB+ 87, AEHL88] where a single window is displayed on the display screens of multiple users, and each user gets an identical image of the window. Many shared window systems provide no concurrency control for simultaneous actions, allow users to constantly see the actions of other users who are responsible for manually ensuring that there are no conflicts. At the other extreme lie teleconferencing systems where interference is avoided by strict control based on the notion of a “floor”, where only the current “speaker” that has the floor is allowed to “speak” (or transmit data) at any time. Both the extremes have their limitations. On one hand, lack of any concurrency control puts additional responsibility on application designers to resolve conflicts. On the other hand, strict floor-based control does not allow applications to exploit inherent concurrency and may sometimes unnecessarily increase latency due to the waiting involved. Needs of some applications may lie between the two extremes. For instance, users in a conferencing system may sometimes decide to enter smaller discussion groups that may separately hold concurrent conversations [SFB+ 87]. An interaction based on the “chalkboard” metaphor [SFB+ 87] in a cooperative environment is another example where less stringent coordination is appropriate. In the case of a group editor, strict coordination is not necessary when one user is browsing through a part of the shared text while another user is annotating a different part of the text. Thus, a flexible concurrency control mechanism is necessary to allow a wide range of semantics. Causal Synchronization: Problem of causal delivery has been addressed before in the case of distributed systems [BJ87, PBS89]. When collaboration only involves shared text or still images, enforcement of strict causality is similar to the previous work in distributed systems because there are no real-time constraints on message delivery. However, multimedia collaboration involves delay sensitive media such as voice and video. Voice or video communication is delay sensitive in nature in that the data must be presented at specific instants (or at specific rate) at a receiver to convey its original meaning. The delivery of data units must follow better never than late semantics and multimedia messages that are delayed beyond an upper bound are rendered useless for playback.
5
Thus, the problem of both causal and temporal synchronization is complicated when delivery of delay-sensitive multimedia data is involved. Within an application, desired level of causality may depend on the context and the media involved, and may vary from strict causality to less strict notions of causality. The communication software must provide mechanisms that allow an application to enforce the necessary semantics.
3 Communication Abstractions Given the need for flexible coordination and synchronization mechanisms, our design is based on the principle of separation of policy and mechanism. We believe that the right way for accommodating a wide range of coordination and synchronization requirements is to provide communication mechanisms that allow specification and selection of the degree of coordination and control needed and leave the task of choosing the appropriate policy to the application designer. We propose two methods of coordination: token-based control and an abstraction called multi-flow conversation. Together, these two methods yield a flexible and adaptable coordination mechanism as discussed below.
3.1 Token-based Concurrency Control When a multipoint flow is created, a token is assigned to that flow that acts as an authorization for data transmission. A sender must hold a token to be able to send traffic over a flow. However, token management primitives are provided so that other participants can obtain transmission privileges. Thus, applications are free to transfer, replicate, and delete tokens to govern the degree of concurrency control needed. This type of concurrency control is entirely different from the token-based synchronization provided by some session-layer protocols. The latter method of synchronization allows session participants to insert resynchronization points (or checkpoints) in the data stream to allow rollback and to reduce the amount of retransmitted data in case of a transmission error. The token mechanism allows several levels of concurrency control. For example, we describe below a concurrency control hierarchy derived based on the collaboration paradigms used in existing systems [SFB+ 87, AEHL88, CF89, SG85]. 6
Floor Control Real-time teleconferencing systems employ such a strict concurrency control. A single token enforces control over a multipoint connection. The token will be passed on from one speaker to another whenever the floor is transferred. To obtain control of the floor, a participant must explicitly request transfer of the token. Chalkboard Interaction Applications based on the “chalkboard” metaphor in a cooperative environment interact in two phases. In the first phase, a speaker addresses a group of listeners with no interruptions. The application may switch to the second phase any time. In the second phase, a group of questioners may address questions to the speaker. The token-based method can accommodate both types of interaction. Only the speaker may hold the token during the first phase, whereas the token may be replicated and distributed to the questioners during the second phase. Replicated tokens will be destroyed at the end of the second phase. Discussion Groups Some environments such as real-time conferencing systems [CF89, SG85] envisage a session breaking up into smaller discussion groups and thus holding multiple, concurrent conversations. In such a system, initially only a single token would be created and passed on from one speaker to another to achieve strict “floor-based” control. However, the token can be replicated and transferred to each discussion group, and each group may then use the token independently as it sees fit. Brainstorming This form of coordination is common in shared window systems where there is no concurrency control for simultaneous actions by multiple users. For such applications, the token is replicated and distributed to all the participants.
3.2 Multi-flow Conversations To allow temporal and causal synchronization among traffic over multiple flows, we introduce a communication abstraction called a multi-flow conversation. A conversation is a logical entity and consists of one or more (two-party or multipoint) flows. An application will typically first create individual flows with appropriate performance requirements [Fer90]. It will then create a conversation that includes one or more related flows to achieve the necessary temporal and causal synchronization. For example, Figure 1 shows a conversation C that consists of two flows,
Flow1 and Flow2. 7
If a sender
S1 sends a message M1 over
S1 Conversation
Control Flow
S2 Flow2 (video)
Flow1 (voice)
S3 Figure 1: Conversation C consists of two flows Flow1 and Flow2 that span three participants S1, S2 , and S3 .
Flow2 and then sends another message M2 over Flow1, all the participants should receive those messages in the same temporal order. Moreover, if the participant S2 sends a message M3 after receiving message M2 , the participant S3 must “see” the correct causal order (M1 followed by M2 followed by M3). The conversation abstraction ensures such ordering. It uses message boundaries as synchronization markers. Messages sent over flows from the same sender are assigned monotonically increasing sequence numbers and are also timestamped to capture the temporal relationships among messages sent over different flows. In the following, we describe the details of temporal and causal synchronization.
3.3 Temporal and Causal Relationships Formally, let F denote the set of flows in a conversation C and let M denote the set of messages sent over the constituent flows in F . Definition 1. We define (“precedes”) to be a transitive relation on M, such that m1 if the following conditions hold: 1. both m1 and m2 are sent by the same sender, and 8
m2 if and only
2. both m1 and m2 are sent over flows fi and fj (both fi ; fj
2 F ), and
3. the message m1 is sent before m2 is sent. The sequence numbers assigned to messages sent over constituent flows by the same sender capture the partial order defined by the relation in MCP. In addition, timestamps assigned to each message precisely identify the temporal relationships among related messages. However, this temporal relationship must be augmented further to provide causally ordered delivery of messages originating from multiple senders. Stricter notions of causality provided by a causal broadcast [BJ87, PBS89] achieve a global ordering of messages necessary for this purpose. Following Lamport’s definition of causality [Lam78], we define a causal dependency relation “!” among messages in a conversation as: Definition 2.
m1 ! m2 if and only if at least one of the following two conditions hold:
1.
m1 m2, or
2.
m2 is sent over some flow fi by sender S1 after receiving m1 over some flow fj , and both fi ; fj 2 F
Thus, Definition 2 defines a partial order among all the messages exchanged over a conversation that captures both temporal and causal relationships. In the following, we refer to such an order simply as causal order even though it includes both causal and temporal relationships. Enforcement of such a causal order in the presence of varying delays and losses across a wide area network can cause problems for multimedia information. Delivery of a message may sometimes have to be delayed indefinitely waiting for other messages that precede it in a causal order. Such delays are not acceptable to real-time flows involving voice or video that have better-never-than-late semantics. For instance, some real-time voice flows typically require delivery of each sample within 100 milliseconds to maintain the desired playback rate at a receiver. In such cases, an application can adapt to some increases in delays, but delays of 300 milliseconds or more render a voice sample useless for playback. To take such delivery constraints into account, we discard strictly causal ordering in favor of a more adaptive notion of causality called ∆-causality. Informally, ∆-causality works as follows. Given a set of flows and a set of participants in a conversation, message delivery to an application at a recipient site is both temporally and causally ordered provided an 9
upper bound ∆ on end-to-end delay is not violated for any of the related messages. In the case of example
S3 will see the proper causal order provided M1 and M2 were generated and sent in such a way that their arrival at S3 is not delayed beyond an upper bound ∆. ∆ is a function of individual delay constraints for flows Flow1 and Flow2. involving Figure 1, the receiver
To take delay constraints into account, we define a ∆-Causal order as: Definition 3.
m1 !∆ m2 if :
1.
m1 ! m2 and
2.
m1 is sent at most ∆ time units before m2
It must be noted that our notion of real-time assumes large-grain clock synchronization among the participants. This is not an unrealistic assumption as fault-tolerant clock synchronization algorithms exist that achieve such synchronization. In the TCP/IP Internet, Network Time Protocol (NTP) achieves global clock synchronization across the country within a few milliseconds [Mil89]. Thus a small upper bound (typically 5-10 milliseconds) on difference between clocks at any two participants can be assumed. In the following, we describe ∆-Causality in more detail.
3.3.1
∆-Causality
We assume that the network layer provides a flow abstraction with performance guarantees to provide predictable service needed by real-time voice or video channels [Zha89, FV90, CY89]. Desired performance characteristics are specified when a flow is established. Apart from the usual packet rate and error rate parameters (not considered here), there are two delay constraints associated with each flow Fi : Desired delay i is the maximum end-to-end delay that the flow
Fi
can tolerate before quality of ser-
vice deteriorates. Example of such a constraint is 100 milliseconds delay bound in packet voice. Ferrari [Fer90] gives several examples of delay constraints for different classes of traffic. Loss delay i is an upper limit on end-to-end delay for flow traffic beyond which the delivered traffic delivery is useless. For example, packet voice or video have better-never-than-late delivery semantics
10
and specify such a constraint. Packet voice traffic with desired delay constraint of 100 ms has a loss delay constraint in the range of 200 to 300 ms. Consider a conversation C consisting of flows F1 : : :Fi : : :Fn with respective delay constraints i ’s and
i ’s. We compute two conversation-wide desired delay and loss delay constraints: ∆1
= maxfi; i = 1; : : :ng
∆2
= minfi; i = 1; : : :ng
Based on these two constraints, we define the causality interval ∆ for each conversation. ∆ is computed
< ∆ < ∆2 and defines a window of causality for each message sent in a conversation. For example, if a message M4 is sent by a participant after receiving messages M1 through M3 over some constituent flows, M4 must normally be received by applications at all other sites in the correct causal order (i.e., after M1 through M3 are received). However, ∆-causality specifies that a conversation provider at each receiving site maintain such causal relationship between M4 and each of its predecessors only if messages related to M4 are received within interval ∆. as ∆1
The value of ∆ is chosen based on the following considerations. First, ∆ must at least be equal to ∆1 + 2
( is the maximum possible clock skew). Second, to allow some flexibility in the presence of fluctuations in
network conditions and delays, MCP provider may choose value of ∆ to be higher than ∆1 + 2 without
compromising individual flow semantics as long as ∆ remains much below the upper bound ∆2 . The interval ∆ is used as follows: 1. Each conversation is assigned a network-wide unique conversation identifier conv_id; each message in a conversation is timestamped (msg_time) and is also assigned a conversation-wide unique mesg_id2 . 2. Whenever a participant at a site
S1
sends a message
M1
over one of the constituent flows of a
conversation, the conversation provider at that site includes some context for that message. 3. The context information included with each message is a pair for
each participant. That is, the context includes the sequence number of the last message received at S1 2
A conversation-wide unique message identifier is easily constructed by appending a local sequence number to the address of the participating site.
11
from each participant before sending M1 . Thus, the amount of context is bounded by the number of conversation participants. The amount of context is also bounded by the causality interval ∆, because not all the context information may be current and useful if last message from some participant was not received in recent past. In particular, causality interval implies that a message M received earlier at
S1 provides no context for M1 if M was received at time less than T ? ∆ if T is the time at which M1 is sent by S1 . 4. When a receiver R1 at another site receives message M1, it checks to see whether or not it has already received all the messages specified in the context of M1. If not, it buffers M1 waiting for the missing messages to be delivered and notes the timestamp Tm1 for
M1 .
It may have to buffer additional
messages in M1’s context if they also arrive out of order. However, if the missing messages are not received within the interval ∆ after Tm1, the conversation provider delivers
M1 to its application and
deallocates buffers. The parameter ∆ essentially acts as a tuning knob; when ∆ is zero a conversation only guarantees causal synchronization among messages originating from the same sender and when ∆ goes to infinity you get strict causality similar to causal broadcast in ISIS [BJ87]. The latter is useful when dealing with non real-time (“batch”) multimedia applications such as mail or a slide presentation playback where a remote server is sending data from a multimedia storage. Note that the idea of using and preserving context information is not completely new as it has been used before in distributed systems [PBS89, BJ87]. However, we have extended it in two directions. First, we use the context information across traffic over multiple flows in a conversation rather than treating it separately within each individual flow. Second, the notion of causality provided is weaker and bound by real-time traffic delivery constraints as demanded by multimedia distributed applications.
4 Multi-Flow Conversation Protocol Framework To realize the communication abstractions proposed earlier, we have designed a MCP protocol framework and Figure 2 shows the proposed communication architecture based on layering in the ARPA Internet protocol suite. At the network layer, we assume a reservation-based service such as the one provided by the 12
Session Token Control
UDP
MCP
TCP
Conversations
Transport Flow Control
IP
COIP (or ST-II)
Figure 2: Organization of the MCP protocol framework in the ARPA Internet hierarchy. Tenet Scheme Channels [FV90, FBZ92], ST-II [LT90], and RSVP [ZDE+ 93]. However, our current design is based on McHIP and we assume that the network layer service guarantees an upper bound on end-to-end delay for each flow. Above the network layer, the MCP functionality is subdivided into transport and session layer mechanisms. At the transport layer, MCP needs a flow control mechanism and can use an existing transport protocol such as UDP 3 . At the session layer, MCP implements the conversation abstraction, multipoint flow management, and token management. A session layer MCP provider resides at each site and interacts with its peers residing at other sites to to coordinate management of flows and conversations. Applications interact with the MCP session layer provider through a set of service primitives.
4.1 Flow Management Under flow management, MCP provides service primitives for flow creation, flow termination, and primitives for sending and receiving messages. An application establishes a flow by using the service call init_flow(flow name, params, list of participants). The init_flow returns a flow identifier (a flow_id) and a token for coordinating data transfer. The user-specified (that is agreed upon in advance) “flow name” (in addition to the flow identifier) serves 3
In fact, we have an MCP library implementation that uses UDP as its transport protocol and IP multicast at the network layer. In principle, MCP coordination and synchronization mechanisms can be implemented on top of IP multicast. However, no bounds on end-to-end delay are guaranteed in this case.
13
to identify a flow at each participating site. One or more participants are specified using a transport-level multicast or unicast address, and flow parameters include performance (delay constraints and bandwidth requirements) and error control specifications. The init_flow primitive only creates a flow at all participating sites. Each participant must explicitly invoke the primitive accept_flow() before receiving and sending any traffic over the flow. accept_flow returns a flow identifier and (optionally) a token if the creator has requested distribution of the token to that participant. Flow termination is achieved by each participant using a terminate_flow primitive.
4.2 Conversation Management The service primitive init_conv(a list of Flow_IDs) creates a conversation. The list of constituent flows may be empty as flows can be added to a conversation later. Init_conv is symmetric and requires all the participants to issue the primitive init_conv before a conversation can be used. The primitive init_conv returns a conversation identifier to the requesting participant and is used in subsequent conversation-specific primitives. The tokens from constituent flows are implicitly transferred to the conversation at each holding site and are then managed as part of a conversation. The primitives join_conv(,) and leave_conv(,) are used for adding and removing a flow to and from a conversation. Any participant may issue the primitive join_conv; the local conversation provider adds the requested flow to the conversation and forwards that request to its peers at other participating sites. The local conversation provider will also start exchanging control information with its peers to enforce temporal and causal synchronization for messages sent over the newly added flow and other constituent flows. However, at other peer sites, the local participant must explicitly issue the primitive join_conv before temporal and causal synchronization is enforced for outgoing messages over the requested flow. This restriction is necessary to avoid confusion resulting from delayed delivery of earlier flow traffic at some sites. Likewise, the primitive leave_conv simply results in removal of the requested flow from the conversation at that site. The local conversation provider informs its peers about the change. Each participant must explicitly issue the primitive leave_conv at each site before outgoing traffic over the requested flow is
14
not considered subject to temporal synchronization.
4.3 Token Management MCP also provides service primitives for replicating, distributing, transferring and deleting a token. An application can create multiple copies of a token by using the primitive replicate_token (token and no. of copies) and distribute a replicated token using distribute_token(token, participant list). The MCP provider permits token distribution only if appropriate number of copies of the specified token exist locally. Sometimes, it may be necessary to divide a discussion group (a multipoint flow) into smaller, independent discussion groups. In such a case, controlling token for the flow must be explicitly transferred to each group so that each group has its own concurrency control. The service primitive transfer_token (token, list of participants) must be used for this purpose. Finally, a copy of the token may be destroyed at a site using the primitive delete_token(). However, the original controlling token for a flow always remains in existence and is only destroyed when the flow is terminated.
4.4 Other Features An interesting feature of MCP is the use of an out-of-band signaling channel for exchanging control information to manage a flow or a conversation. Because the transmission and delivery of data over a multipoint flow is controlled by transfer of tokens and relevant control information, processing of normal data is complicated if one uses “in-band signaling” as in traditional transport protocols. To facilitate realtime protocol processing and to simplify processing of user data, we separate the control information flow from normal data transfer using a separate signaling channel. Another interesting feature is an upcall-based user interface to allow asynchronous, event-triggered actions. Under such an interface, an MCP user can specify an action to be taken on receipt of data over a flow or in case of an event such as a flow joining or leaving a connection. The service primitive conv_status(, ) specifies the upcall procedure to be called when one of pre-defined status changes occurs on a conversation.
15
The arguments to the procname include conversation identifier, an integer code specifying the kind of change, relevant flow identifiers (if any), and the user data received. A similar primitive is also provided for a flow. Such an asynchronous interface is useful for a multimedia application that must simultaneously handle incoming traffic and changes in status of multiple connections. For example, a shared image-display system may specify a routine to be called for automatic updating of the screen when data is received on a connection that only handles remote mouse and keyboard events.
5 Implementation One of the goals of our research is to integrate MCP abstractions into a general purpose operating system such as Unix so that we can utilize a large collection of toolkits and application software libraries [Dew90] to build interactive multimedia applications. Towards that goal, we have implemented MCP in the framework of the Unix 4.3 BSD networking software. Our implementation is based on the CoIP-k kernel which provides a toolkit for implementing and experimenting with connection-oriented internet protocols [CP93]. Details of our implementation are described elsewhere [YLK93] and we summarize here some of the salient points to keep the discussion clear and relevant: 1. MCP retains the conventional TCP/IP programming interface in Unix. In a multipoint application, a single participant acts as a client and all other participants act as servers (or passive entities). An example of such a program is shown in Figure 4. 2. Control primitives for conversation and token management are provided through the setsockopt interface in the Unix socket library. 3. MCP transport and session layers reside below the socket layer. 4. MCP transport layer is responsible for establishment of point-to-point and multipoint data and control flows and for the transfer of data to and from the MCP session layer. 5. Given a conversation, MCP also establishes a separate control flow among the transport-layer peers at participating sites. The control flow is devoted to exchange of state and control information related to
16
Socket
Internet
Xerox NS
COIP-K (MCP)
Phys. Net
Figure 3: Location of the MCP protocol suite in Unix networking software. the conversation. Communication over a control flow is typically only two-party and requires reliable delivery. The following section describes relevant details of the session layer implementation.
5.1 MCP Session Layer Implementation MCP session layer implements two important functions, namely, token management across a multipoint flow and temporal/causal synchronization among messages sent across constituent flows in a conversation. Data transmission across a flow at each sender consists of a sequence of messages. Data sent using each write call by an application constitutes a single message. The MCP session layer preserves the message boundaries in delivery of data at each participant and also uses the message boundaries as synchronization markers (or points) for both temporal and causal synchronization.
5.1.1
Token Management
For each participant of a flow, MCP allocates a separate MCP PCB (see Figure 5) that maintains the state information about the token for the participant. MCP enforces token semantics in the following way:
When a sender tries to send a message across a flow using MCP, the MCP session layer transmits the message only if the sender PCB holds a token. Otherwise, MCP returns an error indicating that the token is not present and it is left to the application to explicitly request transfer of a token from the 17
CLIENT s = socket(PF_COIP, SOCK_MCP, 0) s = setsockopt(s, MCP, MCP_SETFLSPEC, &flowspec, sizeof(flowspec) ) connect(s, addr, addrlen);
SERVER s = socket(PF_COIP,SOCK_MCP,0) ..... bind(s, addr, addrlen); listen(s,5); snew = accept(s, addr, addrlen)
/* Establish a conversation */ setsockopt(s, MCP, MCP_INITCONV, sockarray, sizeof(sockarray)); Figure 4: A typical multipoint MCP application consists of a single client and one or more servers. Servers are passive listeners waiting for the client to establish a flow. The client initially has the token when a flow is established. The token holder always gets to transmit and a server must first obtain a token before sending any data. current token holder.
When a token request is made, the MCP session layer multicasts the request across the multipoint flow. When the request is received by MCP layer at a token holder, it immediately acknowledges receipt of a request so that the sender may not repeatedly multicast its request. MCP layer at the token holder then notifies its application using an asynchronous event notification mechanism that identifies the requesting participant (see Figure 5).
MCP does not implement any particular token management policy and it is left to the application to decide whether and when to transfer the token to the requesting entity.
An issue that is policy specific is “what to do with pending requests when a token is transferred”. MCP provides a call to flush pending requests before a transfer, this call is useful in resolving contentions. The default is to forward pending requests with tokens.
MCP also implements other token management calls such as copy token, delete token, and distribute token.
5.1.2
Temporal and Causal Synchronization
MCP session layer implements both temporal and causal synchronization following ∆-causal order described earlier. To facilitate causal ordering, each message carries three pieces of information: 18
Participant B
Participant A
ctlpcb pointer
request
mcp_tknreq(s)
reqhead pointer token type
notification
Waitfortoken(s)
copies
CoIP-K pcb pointer cprev
ack
cnext status
mcp_tkntrans(s)
transfer
convid
exception condition or SIGURG
Flow QOS info mcp buffer pointer
mcp_getevent context info pointer
ack
Figure 5: Example of token request and transfer using setsockopt call and unix signal is shown above. MCP PCB (Protocol Control Block) is shown on right hand side. 1. a unique sequence number; when multiple flows are part of the same conversation, sequence numbers for messages sent by the same sender across constituent flows are drawn from a single sequence number space. 2. a timestamp that records the time at which the message was sent. 3. context information that gives the context in which the message was generated. The context includes a pair for each participant in the conversation. That is, the context includes the sequence number of the last message received from each participant before the message was sent. Figure 6 shows an example of the context information in the message number 18 sent by a participant
A. At a receiver, MCP session layer enforces the ∆-causal order using these three pieces of information. For messages over a flow from the same sender, the MCP session layer can easily derive a delivery deadline for each message using the flow’s QOS specification that includes the delay bounds for individual messages. Thus, maintaining necessary playback rate for messages withing a flow is straightforward.
19
Message from A has following context info Flow id
Sequence number
Participant
B’s
C’s
D’s
number
Info
Info
Info
5
18
A
10
2
12
Context info at B Contents of
A
15
C
2
Message 18 Waiting for 12 Waiting for 17
D
10
Waiting for A
17
Waiting for C MCP Buffer pointer Waiting for D
12
Dep. lists after Message from A is queued up
Figure 6: At the top, figure shows the context information included by A in its message number 18. Context information in the message specifies that the message should be delivered ONLY after message number 17 (from A), message number 2 (from C), and message number 12 (from D) have been delivered. However, the context information at B indicates that B has received so far first 15 messages from A, first 2 messages from C and first 10 messages from D. Therefore, message 18 from A is queued and added to the dependency lists. To ensure synchronization among messages over multiple flows and from multiple participants, the MCP session layer uses the context information. Consider an example illustrated using Figures 6 and 7. When a new message arrives at another participant B , the MCP session layer does the following: 1. MCP checks to see whether or not B has already received all the messages specified in the context of the new message. If yes, MCP schedules new message delivery. 2. If not, MCP builds the dependency list of messages for the new message. Figure 7 shows an example. 3. Pending reception of previous messages in the context, MCP computes the delivery deadline of the new message using the ∆ value, sets a timer to go off at the deadline, and buffers the message. The message is buffered in a message dependency list according to its position with respect to pending messages. 4. Whenever another message M2 arrives over the conversation, MCP checks the head of the dependency list to see whether there are any messages waiting for the arrival of 20
M2 .
If so, those packets are
Message number 4 arrives from C Flow
Sequence
Participant
id
number
number
A’s Info
B’s Info
D’s Info
5
4
C
18
10
12
Context info at B
A
Message 18
15
from A
Message 4 From C Waiting for 12
C D
2 10
MCP Buffer pointer
Waiting for A
Waiting for 12
Waiting for 3
Waiting for 17
Waiting for 18
17
18
Waiting for C
3
Waiting for D
12
12
Dep. Lists after message number 4 from C is queued
Figure 7: Figure shows MCP actions when another message (seq. no. 4) arrives from C. The new message depends on the messages 18 and 12 from A and D respectively. dequeued and are scheduled for delivery along with M2 according to their delivery deadlines. If message M2 itself contains some unresolved dependencies, it is also queued up (see Figure 7). 5. When a timer goes off for a message
M , MCP dequeues M
and any messages that were queued
behind M according to dependency information and queues them up for delivery to the application layer. An exception is reported to the application to notify it of missing messages.
5.1.3
Asynchronous Event Notification
One of the interesting MCP implementation issues concerned an user interface concern. A multimedia application involving multiple data streams must simultaneously handle several events including arrival of traffic over individual flows, mouse and keyboard events, and asynchronous events such as token arrival or a token transfer request. Conventional approach to building such applications in Unix is to use the select system call which is clumsy at best. It is our belief that system designers must allow an upcall-based user interface to allow notification of asynchronous, event-triggered actions and MCP assumes such an interface. Under such an interface, an MCP
21
application can specify an action to be taken when a particular event occurs. For example, the MCP service primitive flow_status(, ) specifies the upcall procedure to be called when one of the pre-defined status changes or events occurs on a flow. The arguments to the procname include flow identifier, an integer code specifying the kind of change, and optionally the user data received. However, Unix does not support upcalls. We are currently investigating alternate ways of implementing upcalls in Unix. Currently, we have a crude implementation of such a facility in a MCP library routine. The library routine in conjunction with MCP kernel simulates an upcall as follows. MCP reports an asynchronous event by sending a signal (SIG URG) to the user process and setting the outofband flag in the socket structure. The upcall handler library routine first uses getsockopt call to retrieve the cause of the event and any additional information from the kernel, and then calls the corresponding upcall routine that is previously registered by the application. This implementation cannot prevent race conditions if many events are posted in quick succession. However, our experience shows that this is not a common occurrence. A more elaborate implementation would require major modification of Unix kernel.
6 Performance and Experience The experimental set up for MCP prototype consists of an internetwork consisting of two ethernets and a set of SUN Sparcstations equipped with audio devices in our Distributed Systems Laboratory. To simulate an internetworking environment with delays (varying from a few milliseconds to a few hundred milliseconds) and packet losses, we have instrumented the router kernel to introduce additional delays and losses. Using MCP, we have implemented many multi-party, interactive applications to evaluate the effectiveness of MCP abstractions and to measure performance of multimedia applications with and without using MCP. In the following, we describe our experience with MCP and Section 6.2 discusses the results of a comparative performance evaluation of the MCP implementation against implementations of other protocols.
6.1 Experience with MCP 6.1.1
Multi-party Applications
To test and gain experience with use of MCP primitives in a collaborative setting, we ported two multi-party applications , namely, mshell and mtalk. Mshell is an interactive shell that provides command-level access 22
to various MCP primitives such as flow creation, conversation creation, joining/leaving a conversation, and token request/transfer. Thus, Mshell was useful in testing each MCP service primitive separately and in various combinations with other primitives. Mtalk (for Multi-user talk) is a multi-party conferencing program that allows a conversation among multiple users [CP93]. At each user, the program divides the screen into two parts: an input region and display region. Lines of text typed in input region are transmitted over a MCP flow to all the other users and text from other users is displayed in the display region of the screen. In combination with Mshell and Nevot (see below), the set up allows us to create a multi-flow application involving text, voice, and, to a limited extent, images (using a bit map editor). In general, we have found the MCP abstractions extremely useful in enforcing the necessary coordination and control in a collaborative setting. However, more experience involving a classroom setting such as a software engineering course is necessary.
6.1.2
Nevot
NEVOT is a network voice terminal that supports multiple concurrent conferences (both two-party and multi-party) on top of a variety of transport protocols and using a range of audio encodings from vocoder to multi-channel CD quality. It is developed as an experimental tool by Henning Schulzrinne of Univ. of Massachusetts and has been implemented on top of TCP, UDP and ST-II. We have ported Nevot to run on top of MCP and used Nevot/MCP in two ways. First, Nevot allowed us to gain some experience in using MCP abstractions for implementing different conference control styles. Second, we evaluated the cost of using MCP by comparing the performance of audio conferencing applications implemented on top of Nevot/MCP, Nevot/UDP, and Nevot/TCP. Using Nevot, we experimented with the following conference control styles: Floor Control: This is a user driven conference control style used commonly in teleconferencing applications. Under this model at most one speaker holds “floor” at any time and is allowed to talk at any time. Other users must request control of floor before speaking. MCP supports such a user-driven control using tokens. For each flow, there is only one token allocated initially and the applications can transfer token from one speaker to another based on a policy that is chosen by the application. In 23
addition, MCP allows an application to make multiple copies of a token and distribute copies among few participants. In Nevot, for instance, we can implement a hierarchical conference control where a subgroup of participants have access to the floor at any time and can separately control arbitration of speaking rights among the members of the subgroup. Our overall experience so far has been that token management in MCP works very well in implementing a range of floor control policies. However, users must be made aware of token management because a speaker must typically be told to wait until a token is requested and obtained from another speaker. Activity Sensing: Floor-control based conference control using explicit transfer of tokens is not suitable for some collaborative multimedia applications involving shared windows, image browsers, and group text editors. In such an environment, strict control is desired to ensure that at most one user is allowed to manipulate parts of a shared workspace. However, transparent transfer of transmission rights is desirable whenever a user finishes her update (idle for some time), or current sender of data moves mouse (focus) away from a shared window. An example of transparent floor control is a distributed, voice activated, collision sensing algorithm proposed in CECED [CLSF93]. Under an activity sensing algorithm, the underlying software transfers token to another requester when it notices that the token holder has been idle for some time. However, such an algorithm requires choosing a correct time interval to decide whether or not a speaker is in idle (quiescent) state. Our experience showed that an idle interval of 200 to 400 milliseconds gives good results in terms of detecting quiescence. The algorithm works well in a local area environment and does not work so well when propagation delays approach that found over a wide area network (WAN). When the delays are larger, additional delays introduced by activity sensing increase the interactive response time beyond the tolerable limit. Recently, we have devised alternatives that combine activity sensing with token-based control and are currently experimenting with these alternatives. Thus, MCP platform has proved beneficial in further understanding of appropriate collaboration control paradigms. Brainstorming and Discussion Groups: An extreme alternative to strict floor-based control is free for all in which no explicit conference control is exercised and all participants are free to transmit whenever they wish. We have found this to be a useful and efficient form of conferencing especially when a 24
conference enters a discussion phase where there is no obvious speaker at any time. Such a control does not require any support from MCP as every participant has a copy of the token. However, MCP allows the application to switch from brainstorming to a “single speaker” mode or to a group discussion mode. In the group discussion mode, a separate token circulates in each group, MCP enforces the concurrency control in each group, and the application writer need not maintain the necessary state information or exert any additional control.
6.1.3
Impact of ∆-causality
To investigate the impact of ∆-causality, we constructed a multi-flow application involving a voice flow, a text flow for an interactive application (mtalk), and a background flow, Ffile that repeatedly transferred a large file (similar to a page update or advancing to the next page in whiteboard [McC92]). The tests involved at least three participants with variable delays and random packet loss simulated inside the kernel. In the experiments, about 1–5% packets were chosen at random for drop and, in addition, the end-to-end delay was varied between 20 milliseconds to 500 milliseconds. The parameter ∆ was chosen to be 200 milliseconds which happens to be an acceptable delay for packet voice before delays cause noticeable breaks in audio reception. The tests served two purposes. First, they helped us verify correctness of our implementation of causal and temporal synchronization. Second, we wanted to see the impact of enforcement of ∆-causality on the perceived ordering of communication across multiple flows and the quality of the audio playback at participants in the presence of variable delays and occasional packet losses. Our experience is that MCP can easily maintain the deadlines associated with ∆-causality in the presence of occasional packet loss. Packets are lost either due to random drop (network errors) or because packets that arrive beyond the delivery deadline are discarded. In terms of causal ordering, loss of delayed audio packets does not cause easily perceptible degradation in playback quality. On the other hand, participants can discern the proper causal relationship among audio playback, received text, and background file update (“whiteboard transfers”) with little confusion.
25
6.2 Performance Measurements Goals of our performance evaluation were two-fold. First, we wanted to measure the impact of using MCP on the performance of a multimedia application. In particular, we were interested in comparing the performance of an application implemented using MCP against the performance of the same application implemented using other transport protocols such as UDP and TCP. Such a comparison would indicate whether additional functionality of MCP protocols introduces significant overhead defeating its purpose. Second, we wanted to evaluate the impact of implementing MCP in a kernel rather than implementing it as a run-time library. In the following, we describe the results of our evaluation of both the aspects.
6.2.1
Relative cost of MCP as compared to other protocols
We implemented MCP services as part of a run-time library on top of three different protocols: TCP, UDP, and COIP-RAW. COIP-RAW provides direct access to COIP network interface through Unix sockets. Note that the purpose of this exercise is to determine the relative cost of additional MCP functionality and not to compare the protocols themselves. We ported Nevot to run on each of the MCP library implementations and on top of MCP kernel. Tables 1 and 2 show results of a Nevot experiment involving a sender and a receiver. Sender transmitted contents of an audio file (audio playback) in a loop. Nevot prints statistics collected using Unix getrusage call after every 20000 packets and statistics include application’s user and system times. Packets arriving late at the receiver are discarded. MCP kernel performance is comparable to that using other protocols. On the receiving side, Nevot/TCP loses packets due to late arrivals even across an ethernet because TCP queues up data at sending side to send in chunks. Using TCP NODELAY option prevents TCP from queuing up data and that reduces the number of late packet arrivals, but the total times are then much worse as shown in Tables.
26
user time (sec) system (sec)
TCP 45.3 29.6
TCP (NO DELAY) 45 41
UDP 41.8 33.6
COIP-RAW 35 34.3
MCP-kernel 35/45 35/43
Table 1: Statistics at the Nevot sender side for 20000 packets sent at the rate of 50 packets per second. Each packet contains an audio sample worth 164 bytes. The second (larger) time for nevot on the MCP kernel take into account the cost of activity sensing.
user time (sec) system (sec) late packets
TCP 38.2 29.2 21-11
TCP (NO DELAY) 38.5 40.8 3
UDP 37.9 29.9 3
COIP-RAW 38.7 33 5
MCP-kernel 38.8 33.35 5
Table 2: Statistics at the Nevot receiver side. At the receiver side, row labeled late packets shows the number of packets that arrive too late for playout. 6.2.2
Kernel vs. Library Implementation
We had initially implemented MCP as a run-time library. A collaborative application using multiple media must be implemented as a multi-threaded application under MCP library where a separate thread is responsible for handling data transfer across each flow in a conversation and an additional thread handles the requests/responses across the control flow. However, constructing a collaborative application as a multithreaded program requires considerable expertise in concurrent programming and limits its use in many collaborative applications written using Unix socket library. Instead, we decided to implement MCP inside the Unix kernel and retain the conventional system call interface for inter-process communication. Our performance comparison (see Tables 3 and 4) shows some benefits of a kernel implementation. In particular, cost of operations such as creating a conversation and data transfer is significantly higher using the library as compared to the kernel implementation. Type of Connection Point to Point Multipoint
Implementation level MCP Library Kernel 35.6 6.7 57.6 7.8
Table 3: The Conversation create time (in milliseconds) shows the amount of time needed to establish a single-flow conversation involving two participants in each case. Establishing a conversation consists of creating a control flow and updating MCP state to add the flow to a conversation. Typically, a conversation consists of more than one flow, but the cost of creating a multi-flow conversation is similar to that of creating a single-flow conversation.
27
Type of Connection
Point to Point Multipoint
Implementation Method MCP Library MCP Library Kernel (COIP-RAW) (TCP) 257 400 78 441 – 185
Table 4: Data Transfer time for 100 packets each containing one byte of information. All the timings are in milliseconds.
7 Related Work The work described in this paper is related to current research in two main areas: user interface for coordination and conference control, and OS and protocol support for enforcement of QOS and temporal/causal synchronization. In the area of conference control and collaboration, CECED [CLSF93] system is an example of a software system that supports collaboration involving multiple users with minimal intrusion of software and user styles. For coordination and floor control, CECED uses an activity sensing algorithm as described earlier. We have used the algorithm as a basis for further experimentation with alternate activity sensing policies. Schulzerrinne [Sch92] discusses several issues involved in designing a real-time transport protocol (RTP) to support audio conferencing across the Internet. The main focus of RTP is on achieving session management and playout synchronization. The work is a discussion of various alternatives with no concrete implementation. MCP design [Yav92] precedes the RTP work and differs from RTP because MCP considers concurrency and causal synchronization in addition to the temporal synchronization. In addition, MCP has been implemented and evaluated in an internetworking environment. MMCC, Multimedia Conference Control system, developed at University of Sothern California/Information Sciences Institute [Sch93] concentrates on providing session level conference management functions such as conference establishment, inviting parties to join a conversation, merging conferences and starting a “sideconference” through an application level connection manager. MCP includes support for such functions and, in addition provides communication mechanisms to enforce several varieties of conference control and to achieve synchronization among several flows. In the area of temporal synchronization, the following work is of relevance to our work.
28
Venkat Rangan and others [RR93] have proposed elegant algorithms for intra-flow and inter-flow synchronization based on feedback techniques in the absence of global clock synchronization. However, we argue that practical algorithms for achieving global clock synchronization already exist [Mil89] and should be utilized by higher-level protocols. Julio Escobar and others [EDP92] have recently proposed a flow synchronization protocol that provides temporal synchronization by adapting to changes in network delays in the presence of globally synchronized clocks. No actual implementation or experience with the protocol has been reported to the best of our knowledge. Our work is similar to their work in that respect, but, in addition, also considers the problems of concurrency control, real-time causality, and maintenance of consistent views. It should also be pointed out that our proposal also precedes their work [Yav91, Yav92]. In the area of distributed systems, some researchers have designed group broadcast primitives to maintain globally consistent order of delivery when messages are broadcast among a group of processes. The causal broadcast in the ISIS system [BJ87] guarantees that all the processes will receive messages in the same predetermined, partial order. The emphasis in ISIS is on providing fault-tolerant and globally consistent communication in the presence of site and process failures and the aim is to support consistent updates to replicated, shared objects. The Psynch [PBS89] protocol is based on a “conversation” abstraction that defines a partial order on the delivery of messages similar to the causal broadcast in ISIS. Our approach shares some ideas with the Psynch approach where context information is represented by a global context graph maintained by each site. Context information is included in each message and a the protocol maintains a copy of global graph on each host. As explained earlier, a context graph is created independently by each site in our system and edges in such a graph are delay-dependent. Nicolaou [Nic90] presents a scheme for implementing synchronization among related streams originating at the same source by inserting synchronization points in each individual stream. His work is closely related to our proposal as far as achieving temporal synchronization is concerned. We allow an application to specify such points in terms of message boundaries in each individual flow. However, there are important differences between our work and Nicolaou’s work. First, Nicolaou does not consider the problem of causality or concurrency control in a collaborative system. Instead, his work is only concerned with
29
transporting real-time voice and video in a digital network without disturbing the temporal relationship specified by a source. Second, we provide mechanisms to achieve both temporal synchronization among related streams and concurrency control among multiple participants when any of the four media (text, image, voice, and video) are used. Third, the conversation abstraction allows an application to dynamically combine any number of flows into a logical entity.
8 Concluding Remarks In this paper, we have proposed new communication abstractions to meet the coordination and synchronization requirements of distributed collaborative applications. To realize these abstractions over channels with guaranteed performance, we have designed a new protocol framework called MCP. We have built a prototype implementation of MCP using CoIP-k in SUN OS 4.1.1 kernel for Unix workstations and evaluated its performance and suitability of its mechanisms for building multimedia applications. Our work is significant in following respects. First, to the best of our knowledge, this is the first comprehensive treatment of communication layer issues for distributed collaborative applications. Second, this is also one of the first Unix-based implementation of transport and session level services designed explicitly to meet the coordination and temporal/causal synchronization needs of multimedia applications. Third, we have shown that coordination and synchronization mechanisms can be efficiently implemented in a general purpose OS such as Unix. Fourth, we have identified alternate conference control styles that should be supported to meet a wide range of applications including audio conferencing, group editors, and shared workspace. We are using the prototype for further research in various conference control styles and user interfaces for collaboration using multiple media.
Acknowledgments Authors would like to thank Charles Cranor and Guru Parulkar of Washington University in St. Louis for making COIP-K sources available to us and for cheerfully putting up with our questions about COIP-k software during the initial stages of MCP implementation. We would also like to thank Henning Schulzrinne of AT&T Bell Labs for making Nevot sources available to us, and Giri Kuthethoor for help with the Unix implementation of MCP.
30
References [AEHL88] S.R. Ahuja, J.R. Ensor, D.N. Horn, and S.E. Lucco. The Rapport Multimedia Conferencing System. In Proceedings of the 2nd IEEE Conference on Computer Workstations, pages 52–58, March 1988. [BJ87]
Ken Birman and Thomas Joseph. Reliable communication in the presence of failures. ACM Transactions on Computer Systems, 5(1):47–76, Feb 1987.
[CF89]
T. Crowley and H. Forsdick. MMConf: The Diamond Multimedia Conferencing System. In IFIP WG2.7 Working Conference on Engineering for Human Computer Interaction, Palo Alto, California, August 1989.
[CLSF93] E. Craighill, R. Lang, K. Skinner, and M. Fong. CECED: A System for Informal Multimedia Collaboration. In Proceedings of ACM Multimedia ’93, Anaheim, pages 437–446, August 1993. [CP93]
C. Cranor and G. Parulkar. An Implementation Model for Connection-Oriented Internet Protocols. Journal Of Internetworking, 4(3):133–157, 1993.
[CY89]
D.E. Comer and R.S. Yavatkar. Flows: Performance Guarantees in Best Effort Delivery Systems. In Proceedings of IEEE INFOCOM ’89,Toronto, pages 100–109. IEEE Computer Society, April 1989.
[Dew90]
Prasun Dewan. A Guide to Suite: Version 1.0. Technical Report SERC-TR-60-P, Software Engineering Research Center, Purdue University, West Lafeyette, Indiana, February 1990.
[DL87]
T. DeMarco and T. Lister. Peopleware: Productive Projects and Teams. Dorset House, 1987.
[EDP92]
J. Escobar, D. Deutsch, and C. Partridge. Flow Synchronization Protocol. Technical report, BBN, March 1992. an unpublished draft.
[FBZ92]
D. Ferrari, A. Banerjea, and H. Zhang. Network Support For Multimedia. Technical Report TR-92-072, University of California, Berkeley and International Computer Science Institute,Berkeley, November 1992.
[Fer90]
D. Ferrari. Client Requirements for Real-time Communication Services. Internet Request For Comments 1193, November 1990. Internet Engineering Task Force.
[FV90]
D. Ferrari and D. Verma. A scheme for real-time channel establishment in wide-area networks. IEEE Journal on Selected Areas in Communications, 8(3):368–379, April 1990.
[Lam78]
L. Lamport. Time, clocks, and the ordering of events ina distributed system. Communications of the ACM, 21(7):558–565, July 1978.
[LT90]
Charles Lynn and Claudio Topolcic. Experimental internet stream protocol, version ii (st-ii). Internet Request For Comments 1190, October 1990. Internet Engineering Task Force COIP Working Group.
31
[McC92]
Steven McCanne. A Distributed Whiteboard for Network Conferencing. Technical report, Real Time Systems Group, Lawrence Berkeley Laboratory,Berkeley, September 1992.
[Mil89]
Dave L. Mills. Measured performance of the network time protocol in the internet system. Network Working Group Request for Comments: 1128, October 1989. Internet Engineering Task Force.
[Nic90]
C. Nicolaou. An Architecture for Real-Time Multimedia Communication Systems. IEEE Journal on Selected Areas in Communications, 8(3):391–400, April 1990.
[PBS89]
L. Peterson, N. Buchholz, and R.D. Schlichting. Preserving and using context information in interprocess communication. ACM Transactions on Computer Systems, 7(3):217–246, August 1989.
[PD93]
Editor P. Dewan. Special issue on collaborative software. Computing Systems, 6(2), 1993.
[RR93]
S. Ramanathan and P. Venkat Rangan. Feedback Techniques for Intra-Media Continuity and Inter-Media Synchronization in Distributed Multimedia Systems. The Computer Journal, Special Issue on Distributed Multimedia Systems, 36(1):19–31, March 1993.
[Sch92]
H. Schulzrinne. Issues in designing a transport protocol for audio and video conferences and other multiparticipant real-time applications. INTERNET-DRAFT, Internet Engineering Task Force, Audio-Video Transport Working Group, December 1992.
[Sch93]
Eve M. Schooler. Case study: Multimedia conference control in a packet-switched teleconferncing system. Journal of Internetworking: Research and Experience (1993), 4(2):99–120, june 1993.
[SFB+ 87] Mark Stefik, Gregg Foster, Daniel Bobrow, Kenneth Kahn, Stuan Lanning, and Lucy Suchmann. Beyond the Chalkboard: Computer Support for Collaboration and Problem Solving in Meetings. Communications of the ACM, 30(1):32–47, January 1987. [SG85]
S. Sarin and I. Greif. Computer-based Real-Time Conferencing Systems. IEEE Computer, 18(10):33–49, October 1985.
[Yav91]
Rajendra S. Yavatkar. Communication support for collaborative multimedia applications. Technical Report 181-91, Department of Computer Science, University of Kentucky, January 1991.
[Yav92]
Raj Yavatkar. MCP: A Protocol for Coordination and Temporal Synchronization in Multi-media Collaborative Applications. In Proceedings of the 12th International Conference on Distributed Computing Systems, pages 606–613. IEEE, June 1992.
[YLK93]
R. Yavatkar, K. Lakshman, and Giri Kuthethoor. Design and implementation of a multimedia protocol suite in a bsd unix kernel. In USENIX Summer Conference Proceedings, Cincinatti, pages 129–147, 1993.
[ZDE+ 93] L. Zhang, S. Deering, D. Estrin, S. Schenker, and D. Zappala. RSVP: A New Resource ReSerVation Protocol. IEEE Network Magazine, 7(5):8–18, September 1993.
32
[Zha89]
Lixia Zhang. A New Architecture for Packet Switching Network Protocols. PhD thesis, Massachusetts Institute of Technology, July 1989.
33
List of Figures 1 2 3 4
5 6
7
Conversation C consists of two flows Flow1 and Flow2 that span three participants S1 , S2, and S3. : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
: Organization of the MCP protocol framework in the ARPA Internet hierarchy. : : : : : : : Location of the MCP protocol suite in Unix networking software. : : : : : : : : : : : : : :
17
A typical multipoint MCP application consists of a single client and one or more servers. Servers are passive listeners waiting for the client to establish a flow. The client initially has the token when a flow is established. The token holder always gets to transmit and a server must first obtain a token before sending any data. : : : : : : : : : : : : : : : : : : : : :
:
18
Example of token request and transfer using setsockopt call and unix signal is shown above. MCP PCB (Protocol Control Block) is shown on right hand side. : : : : : : : : : : : : :
:
19
At the top, figure shows the context information included by A in its message number 18. Context information in the message specifies that the message should be delivered ONLY after message number 17 (from A), message number 2 (from C), and message number 12 (from D) have been delivered. However, the context information at B indicates that B has received so far first 15 messages from A, first 2 messages from C and first 10 messages from D. Therefore, message 18 from A is queued and added to the dependency lists. : : : : :
:
20
Figure shows MCP actions when another message (seq. no. 4) arrives from C. The new message depends on the messages 18 and 12 from A and D respectively. : : : : : : : : :
:
21
34
8 13
