Interfaces for Open Programmable Routers Spyros Denazis1, Kazuho Miki2, John Vicente3, Andrew Campbell4 1
Centre for Communications Systems Research (CCSR), University of Cambridge, UK, Industrial Research Fellow, Hitachi Europe Ltd, UK.
[email protected] 2 3 4
Hitachi Ltd., Japan.
Intel Corporation, USA
Center for Telecommunications Research (CTR), Columbia University, NY, USA {miki, jvicente, campbell}@comet.columbia.edu
Abstract. The ability to rapidly create and deploy new network services and architectures in response to new user demands is a driving force behind the emergence of programmable networks. The goal of open network control is being addressed in the IEEE P1520 Working Group (http://www.ieee-pin.org/) through the definition of a set of open network programming interfaces for networks. In this paper, we discuss a set of open programmable interfaces for IP-based router and switch networks. These open interfaces allow service providers to manipulate the state of the network through high-level languages and abstractions in order to construct and manage new network services with quality of service support. Keywords: Open Programmable Routers, Router Interfaces, Differentiated Services
1
Introduction
Over the past several years we have witnessed a growing amount of work in the area of open programmable networks [1], [2], [14]. The aim of this work is the design of new network architectures which facilitate rapid creation and deployment of new network services. Central to this goal has been the emergence of open interfaces, enabling control, management and composition of network resources through the introduction of new innovative services, which cannot otherwise be realized with today's proprietary (i.e., closed) network technologies. The need for programmable networks is becoming more apparent as open programmability has become a central theme to a number of standardization efforts and consortia [3], [4], [5]. As technology for open programmable networks matures, well-designed interfaces become more important for flexible customization, operation and extensibility. The development of open programmable interfaces has mainly been the result of academic projects which have fairly specific objectives. In addition, open 2 3
Kazuho Miki is a Visiting Scientist at CTR, Columbia University John Vicente is a Visiting Scientist at CTR, Columbia University
programmable interfaces have been constructed in an ad-hoc manner to support the introduction of services as a proof of concept for network programmability. As a result, a design framework for open interfaces in the context of programmable networks has not been addressed. Most existing proposals found in the literature have focused primarily on the control and management of ATM network elements motivated by the proprietary limitations of existing switching technology [6], [7], [8]. Through the activities of the DARPA initiative for Active Networks [9], [3] there has been an attempt to enable programmability within router-based networks. In parallel to the Active Networks initiative, OPENSIG [10] has been investigating open signalling and programmable network architectures. The open interfaces of a programmable network architecture are structured in a layered fashion whereby the higher interface relies on the services of the interface below it, while it exposes its own services to the layer above it. Hence, an interface is characterized by its scope and the services it offers. For example, the scope of an interface may be such that it distinguishes between node and network-wide services that it can offer. P1520 Reference Model Users V interfaceAlgorithms for value-added Value-added Services communication services created by Level network operators, users, and third parties U interface Network Algorithms for routing and connection Generic management, directory services, … Services Level
}
L interface CCM interface
Virtual Network Device (software representation) Physical Elements (Hardware, Name Space)
Virtual Network Device Level
} }
U Interface Mapping of P1520 RM to IP Routers
}
Applications invoking methods on objects below
Differentiated Services Scheduling
Customised Routing
RSVP or Other per-flow protocol
Routing Algorithms
Software representation of routing resources
CCM Interface
Controller PE Level
Hardware and other resources
L Interface
Routing table Data
Fig. 1: The P1520 Reference Model and the mapping to IP routers.
In this paper, we define a set of router (node) interfaces for programmable router-based networks. We believe that in order to define a node interface, it is first important to describe a framework that assists in the design process and consequently the use and maintenance of the interface. The basic principles underpin our framework are driven by experiences with existing router technology. The proposed model, terminology and interface definition may serve related initiatives (e.g., active networks) by offering a generalised, yet well stuctured interface model in the support of programmable network architectures. The contribution discussed in this paper has submitted to the IEEE P1520 for consideration. This paper is structured as follows. In Section 2, we present an overview of the IEEE P1520 reference model and terminology. Following this, in Section 3 we discuss a number of requirements for open router interfaces. Section 4 introduces our
framework and its three basic components. Sections 5 and 6 present resource and service specific asbtractions, respectively. Section 7 presents the realization of the propsed interfaces in support of Differentiated Services scenario [12], [13], [14]. Finally, we present some concluding remarks in Section 6.
2
P1520 Reference Model
The IEEE P1520 standardisation effort addresses the need for a set of standard software interfaces for programming of networks in terms of rapid service creation and open signalling [5]. The technology under consideration spans from ATM switches to IP routers to circuit or hybrid switches. The interfaces are structured in a layered fashion offering their services to the layer above. Each layer defines what is termed as a level. Each level comprises a number of entities in the form of algorithms or objects representing logical or physical resources depending on the level’s scope and functionality. This approach gives rise to the reference model depicted on the left of Fig. 1. More specifically, we can distinguish the four levels as follows: • • • •
The physical element (PE) level consisting of entities such as hardware and the device architecture that actually reflects upon the supported capabilities The virtual network device level (VNDL) has as its sole purpose to logically represent resources in the form of objects (entities) thereby isolating the upper layers from hardware dependencies or any other proprietary interfaces The network generic services level (NGSL) consists of entities in the form of distributed algorithms that bind (interconnect) together the objects of the VNDL level according to specific network functionality, e.g., routing, connection setup. Finally, the value-added services level (VASL) includes entities in the form of end-to-end algorithms that actually enhance the generic services of the NGSL level; adding user oriented features and capabilities in the applications.
The four levels give rise to four interfaces, namely, CCM (Connection Control and Management), L (lower), U (upper), V (value-add) interfaces. The CCM interface is actually a collection of protocols that enable the exchange of state and control information at a very low level between the device and an external agent. The L interface defines an API that consists of methods for manipulating local network resources abstracted as objects. CCM and L interfaces falls under the category of node interfaces. The U interface mainly provides an API that deals with connection setup issues. As in the case of the L interface, this interface also aspires to isolate the diversity of connection setup requests from the actual algorithms that implement them. Finally, the V interface (not shown in Figure 1) provides a rich set of APIs to write highly personalised software often in the form of value-added services. U and V interfaces constitute network-wide interfaces. The P1520 Reference model (RM) provides a framework for positioning possible programming interfaces and operations of networks, given a specific networking technology. A mapping of the diverse network technologies and their corresponding functionality to the P1520 RM is essential. The right side of Fig. 1 illustrates a mapping of the P1520 RM to IP routers. Given this mapping, it is important to establish an L-interface definition that abstracts router resources and
functionality such that it satisfies the service requirements above it, while it is flexible enough to accommodate future service requirements.
3
L-interface Requirements
The objective of the L-Interface is to create programmable abstractions of the underlying resources of the router or switch, whereby third-party service providers, network administrators, network programmers or application developers can influence or extend router or switch control through the use of high-level API's. The requirements of the L-interface design are driven from the perspective of the users of the L-interface. It is therefore relevant to detail and understand these as fundamental to the basis of the L-interface. We enumerate the following requirements: 1. Open programmability - This is an enabling requirement for the L-interface where separation is achieved between hardware and software, fostering third-party service creation and open competition. 2. Operational support - Management and administrative functions must be improved or facilitated via the L-interface, where over slow-time scales, greater control and intelligence data gathering are open to network administrators and architects. 3. Service provisioning - Through open APIs, third-party service providers must have the ability to modify existing network services as well the ability to introduce entirely new network services to the router or switch functionality 4. Extensibility - The associated L-interface model must provide flexibility for extension without intrinsically creating a proprietary format, limiting the extensibility and virtualization process. 5. Programmable abstraction granularity - The ability to allow network programmers to influence network control through open interfaces must also support granularity in router object programmability and service provisioning. 6. Timescale flexibility - Access and control of router resources through the LInterface must be achievable over different time scales (control, management and transport). 7. Resource partitioning – Through management and control plane operations, the Linterface should support partitioning of router resources thereby allowing network operators or service providers to deploy and operate their network architectures on the same physical infrastructure confined to the allocated portion of the router(s) resources. Finally, it is imperative that the L-interface requirements are not hindered or otherwise restricted to the abstracted resources level (VNDL) due to limitations of lower-level protocols (e.g., GSMP) or proprietary design features of a router kernel or hardware. The basic tenet argues that not only should the concept of open network programmability be supported by lower level protocols/interfaces, e.g. CCM, but also new router architecture designs should be prototyped to account for the requirements set by the above interfaces. We view this as critical for the success and widespread acceptance of open programmable routers.
4
Generalized L-interface Model
The framework for designing the L-interface or more generally node interfaces is comprised of three components. The first is a two-layer model representing the Linterface separation structure for abstracting router resources. The second is a hierarchical decomposition approach to representing router resources in the form of a tree-like structure with their corresponding inter-relationship. Finally, the third component describes how the interface definition structure of each abstracted resource, namely each node of the tree, should support control, transport, and management administrative operations on the router resources. In what follows, we further elaborate on each one of these elements. 4.1
Two-layer Abstractions
Fig. 2: Two-layer model for abstracting resources
The process of defining the L-interface through the abstraction of node resources is conceived as a two layer model depicted in Fig. 2. At its core lies the process of abstracting resources that are considered generic in the sense that they are not used in a specific service context. At the outer circle lies the process of abstracting resources that have only meaning when they are used within a specific service context, e.g. differentiated services. To this end, we view router resources from two different perspectives, corresponding to an association with a particular layer of the abstraction model. The first viewpoint is to consider router resources as general-purpose facilities; the abstraction of which leads to general-purpose interfaces that may be used and combined simultaneously by different service domains. The second viewpoint is to identify resources associated with the functionality they are intended. In this context, resources of the latter represent a partition of the general-purpose resources eventually used for certain tasks. In addition, L-interfaces that are the result of the generic abstraction process will form the basis on which service specific interfaces may be defined for a variety of purposes. This in turn may also become a part of the L-interface definition. The advantage of the two-layer model is that it allows for true interface openness and resource reusability under different service contexts. Thus, we are suggesting that such an abstraction model allows, for example, upper level interfaces (e.g. U interface) to create or program completely new network services using generic abstractions or modify existing services using service-specific abstractions, which are themselves built on generic abstractions. This makes the L-
interface flexible rather than static in the sense that as new services are conceived, they can form their interface representation in a seamless fashion. 4.2
Hierarchical Resource Decomposition
Abstracting resources however requires knowledge of an IP router architecture, its corresponding physical resources, identification of those resources pertaining to our objective, followed by some form of classification. Consequently, this activity will result in an IP router reference model that may be used as a guiding model for resource abstraction. Fig. 3, depicts a generalized IP Router Architecture and an example decomposition of the router resources, superimposed onto the P1520 interface model. As shown, the architecture is composed of a number of distinct functional elements: the line cards whereon input/output ports are instantiated, the forwarding engines wherein forwarding of packets takes place with routing and traffic control elements influencing the forwarding policy and traffic control services, respectively. A generalpurpose processor executes router kernel and network-level services; it is responsible for hosting a number processes like routing protocols and other housekeeping or special purpose functions. The connection module is used for transport interconnection among the other elements and may represent a switching fabric or a bus architecture. Each one of these elements may be considered as containers of resources that should eventually be reflected upon the L-interface and thus, become available to the user thereof. Router capacity, not unlike the other resources, plays a part in the functional architecture of the router, where local computation capacity, network bandwidth and static configuration are key abstracts for managing proper utilization of local processing and network-level control and transport services. To this end, it constitutes the quantitative representation of the router and as such it may be viewed as an orthogonal model to the actual router resource representation (Fig. 3). The importance of such a distinction will become more clear in later sections. Achieving generic abstraction requires viewing resources irrespective of functional service domain. Hence, we should identify generic router resources and translate them into generic L-interfaces that provide methods for resource partition and methods to forge each ensued partition according to the specifications of the caller of the method.
L Interface
Bandwidth
Other Buffer Mgmnt
Generic Abstraction
Scheduler
e.g.,OSPF hop cost
e.g., OSPF routing table
CPU Capacity
Other
Buffers
e.g.,DiffServ PHB
Various Tables
e.g., EF PHB buffers
Service-specific Abstraction
CPU Scheduling
algorithm
Routing Calculation
Routing Table
Other Packet Classifier
Database component
Software representation of routing resources
CCM Interface Processor
Line Card
Traffic Control
Route Control
Forwarding Engine
Data
Connection Module Line Card
Forwarding Engine
Capacity
IP Router Fig. 3: IP Router Architecture with example resource abstractions
The core router abstractions can be viewed in a hierarchical manner. As illustrated in, we depict this hierarchy and provide a layered mapping of the Generic and Service-specific abstractions. In this section, we focus on the generalized elements of the router, which are comprised of Base, Functional and Component abstractions, which we describe here. Base abstractions. The base abstractions serve as the major stems of the binding hierarchy for core router services, and represent the highest-level router binding abstractions. They refer to such base services provided by the router, more specifically transport, processor and capacity. Functional abstractions. Base resource abstractions are composed of functional element abstractions. For example, switch fabric and line card resources are functional router elements that serve the transport abstraction for forward processing. Component abstractions. Below the functional element layer, the component abstractions form router or switch functional abstractions. Through network service binding, static components (e.g., queues, schedulers, etc) are realized through the creation or binding of tables or software components composed via programmable instantiation. Service binding abstractions. The service binding interface realizes or binds new or existing network service abstractions to the generic component resources (e.g., scheduler) supporting the router functional elements (e.g., line card port). These tightly coupled interfaces cast service specific abstractions onto the component implementations, thereby binding service-specific policy, algorithms, protocols and the like to local router component resources.
Generic Abstractions
Base Functional abstractions abstractions
Resource Hierarchy Root
(Capacity, Controller, Transporter)
Service-Specific Abstractions
Component abstractions
(Examples: Connection Module, line or port, routing services, traffic control, capacity regions)
(Examples: Queues, tables, classifiers, databases, scheduler, path, ( flow or flow aggregates, threads, addresses, filters)
Service binding abstractions (Examples: Algorithms, protocols, profiles, code-points, policies, index types, entry )
Control Mgmt Transport
Fig. 4: Hierarchical Abstraction Model
4.3
Operational Aspects of the L-Interface
The structure of each L-interface adds one more dimension to the framework necessary to account for control, management and transport aspects of the services offered through the L-interface. As a general rule, each resource abstraction is comprised of data structures and methods, thereby introducing an object-oriented approach. In this context, we characterise data structures and methods according to the type of operations they have originated from or have been used (see also above). As a note, we generalize the operational aspects to have a flexible requirement to support control, management and transport (i.e., not all resources require these services). Finer granularity within each of these basic categories may be possible. For instance, configuration operations may be considered as a sub-category of management. In this manner, it is possible to map control, management and transport services on to the hierarchical abstraction model. Finally, by allowing a resource abstraction interface to reflect types operations in an explicit way (e.g., standard method invocation naming formats) it can facilitate the user or the designer of the Linterface in the development of services.
5
Generic Resource Abstractions
Fig. 5 depicts the interface inheritance hierarchy for a Differential Services IP router. As shown, Base Abstractions are the highest-level interfaces for the Transporter, Controller and the Capacity abstractions of the router. The Transporter which is primarily responsible for forwarding functional abstractions; the Controller interface abstraction which serves routing and traffic control abstractions; and finally, the Capacity abstraction which provides generalized abstractions for local computation,
QoS and traffic abstractions supporting, for example, bandwidth scheduling and static router configuration abstractions. As mentioned earlier, the Capacity branch of the router hierarchy is orthogonal to the other two branches. It is an attempt to quantitatively represent router resources and provide methods that exert partitioning and shaping of the resources. As an example, with the memory resource we can partition a portion of it and allocate it as a queue with a specific structure. An additional advantage of this approach is that it creates measurable, hence comparable, specifications for the router resources which, in turn, can be used to perform a number of operations like bandwidth management, admission control prior to actually committing router resources to any data transportation duties. Similar approaches have been proposed in [8], [10]. Marker
Classifier Forwarding Engine
Transporter
Forward Mapping
Connection Module
Connection Module Scheduler
Connection Module Queues
Line & Port Queues
Line & Port Line & Port Scheduler
IP Router
Controller
Route Controller
Forward Policy Database
Routing Tables
Traffic Controller
Traffic Policy Database(s)
Traffic Tables Flows Threads
Computation
Forward Paths
Memory Storage
Capacity
Disk Space
Transport Configuration Mapping
Configuration Space Name Space
Traffic Descriptor QoS Parameters Address Space
Meter Shaper
Differentiated Services Resource Abstractions
Dropper • DSCP • Scheduling PHB Configuration parameters • Queuing parameters • Customer TCA • AF/MF filters Configuration • Profiles • Treatment Monitoring Data • DSCP Code-point • PHB name Mapping • Description • Filter Traffic • Name Profiles • Qualitative service-level • Quantitative parameters • Traffic profile Traffic • In-profile treatment Treatment • Out-profile treatment
Fig. 5: Differentiated Services Abstraction Model
The Functional and Component resource hierarchy compose the generic resources for the router from the Base Abstractions. The functional abstractions of the router may have one or more service implementation scenarios (e.g., routing - RIP, OSPF, multicast) and may require rather specific component resource abstractions (e.g., queues structures, table, databases) implementations statically resident or created dynamically by way of a network service-specific implementation. These interfaces abstract the generic resources allowing the ability to manipulate, update, read or modify the implementation of the component resource.
6
Service-specific Abstractions
Service domains are realized and operate in any of these elements accessing, manipulating and consuming resources. It is a requirement therefore not to associate
component resources with a certain network service domain. The binding realization of network services with router component resources occurs within the Service Binding layer of the abstraction model. Static or dynamic component abstractions are influenced by way of service domain-specific abstractions. Algorithms, protocols, policies, code-pointers, etc, which are proprietary to the network service, are instantiated through the interfaces created by this layer of the abstraction model. An important aim of this layer is the abstraction of the context of the service being instantiated such that the proprietary nature of the service is hidden or at least mapped to what seems to be a general router resource abstraction. As an example, consider the Differentiated Services functional domain (illustrated in Fig. 5) and let us say that a generic resource is a memory module. We want to be able to reserve a portion of the memory for use by the Diffserv. In addition, we should be able to impose a structure on this portion according to the specifications of a Diffserv table, namely a table that has as its fields the packet ID tuple and the DS field. An amount of buffers is reserved and a specific structure (queue structure) also being imposed. This is complemented with a buffer management scheme. Finally, a portion of bandwidth is allocated and a code-point algorithm is invoked that implements the per-hop behavior. The purpose of the hierarchical abstraction model is to facilitate and guide the process, of using the L-interface as well as extending it as new services are required from the router resources. This can be achieved by starting from a generic abstraction of resources resulting in a number of generic interfaces which in turn can be used to define a service specific abstraction of resources thereby being represented as specialized interfaces, e.g. Differentiated Services interfaces.
7
Scenario – Differentiated Services
The Differentiated Services architectural model [12], [13] is positioned as a collection of service mechanisms, which allow network service providers to offer "differentiated" service levels to alternative customer traffic aggregates. This is done so by packet marking, traffic forwarding using Per Hop Behaviors (PHB), and conditioning of traffic through traffic conditioning mechanisms and/or policies [13]. The policies and router service mechanisms are appropriately realized and enforced at boundary routers or edge devices of different provider Diff-serv domains whereby Service Level Agreements (SLA) are deemed necessary. We illustrate in Fig. 5 an abstraction model for Diff-serv which maps accordingly to our proposed generalized resource abstraction model (presented earlier), separating core generic router resources from those that are specific to the realization of Diff-serv mechanisms and policies. As such, the service implementation or binding of Diff-Serv services to generic router resources support the service programming or provisioning, administration and operational aspects required of the L-interface in a hierarchical manner. This is necessary in that the user (e.g., programmer or administrator) of the L-interface may require operational services rather generically to manage or provision services over (i) entire functional service (e.g., traffic controller); (ii) a generic resource (e.g. classifier); (iii) a servicespecific resource (e.g., shaper); or (iv) an entire network service (e.g., Diff-serv) covering multiple service-specific resources. The hierarchical model may also apply
when a 'new' network service (e.g., routing service) is being deployed; requiring service binding to the existing network infrastructure through collective instantiation of service-specific 'resources' to generic router resources. Diff-serv building blocks are instantiated at the lowest tier of the resource abstraction hierarchy tree through the three major Base Abstractions, namely, i) PHB and traffic conditioning mechanisms are employed through the Transport; ii) PHB configuration, traffic conditioning configuration and monitoring are structured through the Controller; and finally, iii) traffic profiles, code-point mappings, traffic treatment policy and parameters are captured under the Capacity Base Abstraction. Operational services (e.g., install, enable or disable PHB configuration) are invoked through the Diff-serv L-interface abstractions through their methods. The object class definitions are structured in a manner that is consistent with our proposed L-interface abstraction model. We describe the following semantics of a generalized interface definition for resource abstractions and subsequently, an applied example of the Per Hop Behavior object and interface scheme associated with Differential Services: General Semantics - name/ID identifies the abstraction with a unique ID - type identifies the abstract data type - structures defines one or more structures supporting the local abstraction - status implementation status, either new (requiring instantiation) or resident - parent abstraction associates the local abstraction with a parent abstraction - peer abstraction defines peer abstractions as required within a hierarchical level -methods defines methods for operational, management or control of the abstraction Per-Hop-Behaviour (PHB) The per-hop behavior m echanism s are used to forward different traffic types with differing behavior. These are im plem ented via param eterization and policies that affect interfaces queues and schedulers on the router's egress Interface definition specification name/ID: PHB_configuration; type: database status: Include parent abstraction: traffic_policy_database; Include peer abstractions: LP_interface(); LP_queue(); LP_scheduler(); Structures: Struct{ code-point; *LPqueue_number struct scheduling {parameters}; struct queuing {parameters}; }; Struct{ *LPinterface_number; interfacestatus; EFstatus; Afstatus; MaxEFrate; MaxAFrate; }; Methods: InstallPHBdB(); EnablePHB(); DisablePHB(); ReaddB(), WritedB(), UpdatedB();
8
Conclusion
In this paper, we have presented an overview of the IEEE P1520 reference model. Fundamental requirements for open router interfaces were discussed. We have proposed an L-interface framework, introduced through a two-layer model consisting of generic resource and service-specific resource abstractions. In addition, we discussed a hierarchical model for resource inheritance and operational aspects supported by the methods, structures and semantics of individual resources. We also presented a simple scenario using Differentiated Services as an illustrated example of the use of the L-Interface.
9 1. 2.
References
IEEE Comm. Mag., Special Issue on Programmable Networks, Vol. 36, No 10, Oct., 1998. IEEE Network, Special Issue on Active and Programmable Networks, Vol. 12, No 3, May/June, 1998. 3. Calvert, K, et al, “Architectural Framework for Active Networks”, Version 0.9, Active Networks Working Group, August, 1998. http://www.dyncorp-is.com/darpa/meetings/anets98jul/anets-arch.html 4. Multiservice Switching Forum (MSF). http://www.msforum.org/ 5. Biswas, J., et al., “The IEEE P1520 Standards Initiative for Programmable Network Interfaces”, IEEE Communications, Special Issue on Programmable Networks, Vol. 36, No 10, October, 1998. http://www.ieee-pin.org/ 6. Buckley, W., “Virtual Switch Interface (VSI) Specification”, MSF Contribution Document, MSF98.002, November, 1998. 7. Newman, P., W. Edwards, R. Hinden, E. Hoffman, F. C. Liaw, T. and G.Minshall, “Ipsilon’s General Switch Management Protocol Specification Version 2.0”, RFC 2297, Internet Engineering Task Force, March 1998. 8. C. Adam, M.C. Chan, J-F. Huard, A.A. Lazar, and K-S. Lim, “Binding Interface Base Specification: Revision 2”, OPENSIG Draft, April 1997. http://comet.ctr.columbia.edu/xbind/documentation/ 9. DARPA Active Network Programs. http://www.darpa.mil/ito/research/anets/index.html 10. OPENSIG Working Group http://comet.columbia.edu/opensig/. 11. J.E van der Merwe, S. Rooney, I.M. Leslie, and S.A. Crosby, “The Tempest: A Practical Framework for Network Programmability”, IEEE Network, Vol. 12, No. 3, pp. 20-28, May/June 1998. http://www.cl.cam.ac.uk/Research/SRG/dcan/ 12. Blake, S., et al, “An Architecture for Differentiated Services”, RFC 2475, December 1998. 13. Bernet, Y., et al, “A Framework for Differentiated Services”, Internet Draft , October 1998 (Work in progress).Nichols, K., et al, “Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers”, RFC 2474, December 1998. 14. Campbell, A.T., Kounavis, M.E., Vicente, J., Villela, Miki, K. and H. De Meer, "A Survey of Programmable Networks", ACM SIGCOMM Computer Communication Review, April 1999.
