Copyright Notice In order to comply with copyright obligations regarding electronic information dissemination of IEEEcopyrighted material the following copyright notice has to be prominently displayed on the initial screen of the authors and/or their companies servers displaying this material. In this case it is a specific document, that is IEEE-copyrighted, and therefore the declaration appears as the front page of the applicable document, the full-text of which continue below.
“©2008 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works musr be obtained from the IEEE.”
Document commences on next page
A Methodology for Testbed Validation and Performance Assessment of Network/Service Management Systems Eugen Borcoci1, Abolghasem (Hamid) Asgari2, Silviu Ciochină1, Şerban Georgică Obreja1 1
University “Politehnica” Bucharest, Romania,
[email protected],
[email protected],
[email protected] 2 Thales Research and Technology (TRT), United Kingdom,
[email protected]
Abstract: Delivery of multimedia real time flows over multidomain IP networks require end-to-end Quality of Service guarantees. In order to manage and control the high-level services (Video on Demand, IPTV, etc.) as well as the network connectivity services across multiple domains in a coherent way, a distributed but integrated management system is proposed. Such a system has been defined, specified, and is currently implemented in the framework of the ENTHRONE European project. The system is being validated and assessed through several complex interconnected test-beds/pilots. This paper proposes a test methodology for validating the network service management functionalities on a multi-domain test-bed environment. Index Terms: End-to-end QoS management, pSLS, QoS negotiation, testbed, testing, scenarios.
I INTRODUCTION Delivery of multimedia flow services with guaranteed endto-end (E2E) quality of service (QoS) over a multi-domain network environment is still an active research area, especially when heterogeneous access networks exist. An integrated management system should exist, capable of managing the high level services with E2E QoS guarantees, while keeping independent administration and management of each network domain. This management system should also be capable of accommodating heterogeneous wire-line or wireless access networks as they are utilised by various communities. An integrated end-to-end QoS management system has been designed and is currently being implemented and validated in the framework of the ENTHRONE project (FP6 IST-507637 European project) [1], [2], [3], [4], [5]. The ENTHRONE Integrated Management Supervisor (EIMS) [3], [4] is the main management entity designed according to the MPEG-21 standard, sitting at the top of a heterogeneous network infrastructure. The EIMS offers a unified management framework in the audio-visual distribution chain. It assures E2E QoS provisioning by achieving the required “horizontal” coherence of management and control activities, based on Service Level Agreements/Specifications (SLA/SLS) concepts and an appropriate set of signalling protocols. The QoS approach includes the content adaptation, i.e. the adjustment of the application to the network and terminal capabilities and/or to compensate for the deficiencies of the network. Other partially similar management and
1-4244-2577-8/08/$20.00 ©2008 IEEE
control systems for single domain [TEQUILA] or multiple domains [AQUILA], [MESCAL] have been proposed. Given the limited space in this paper, the detailed differences between these ones and ENTHRONE are presented in [10], [11]. The ENTHRONE EIMS supports different business models and entities having their own resources and capabilities but cooperating to offer value-added services for end-users. The main business entities are: Services Providers (SP), Content Providers (CP)-owning Content Servers (CS), Network Providers (NP), Content Consumers (CC), Access Network Providers (ANP), Brokers/Resellers [1], [2], [3], [4]. The SP provides high level services to the end-users while the NPs manage their autonomous network domains. The ANPs manage the Access Networks. In real life, the same operator can play both roles of NP and ANP. Service Level Agreements and Specifications are used to express commitments between these business entities. Several testbeds are set-up in the project, in several countries that are interconnected in order to build an international pilot. Each testbed consists of a networking infrastructure (core IP domain/s, various access networks) and hosts, with the relevant software corresponding to business entities (SP, NP, ANP, CC, CP/CS) that are installed. The testbeds are aimed at several objectives to be fulfilled including development and validation through testing the software components/blocks to be integrated in the management and control system; validation of the management of connectivity services (or IP level network services) in both access and core IP part of the network; validation and deployment of the high level services. Both functional (i.e. correctness) and performance aspects are targeted in the test campaigns. Appropriate scenarios are defined for each defined test. This paper proposes and discusses a methodology to be applied for the ENTHRONE test campaigns and in particular gives examples on how this methodology is applied to the validation of network service management. The paper is organized as follows. Section II shortly presents the ENTHRONE network service management framework. Section III describes as an example, the Romanian Pilot set-up at University “Politehnica” of Bucharest. Section IV introduces the testing methodology. Section V define examples of sample
scenarios for functional and performance level testing and presents the sequence of actions to be performed as well as the metrics to give verdicts on test results. Section VI presents conclusions and future work. II. ENTHRONE NETWORK SERVICE MANAGEMENT The EIMS manages the services at two levels: high-level services and network level services (or IP connectivity services). An IP connectivity service is a service that provides reachability between edge routers/hosts in the IP address space. This service is independent of the networking technologies used and the network management systems used by different NPs in their domains. The EIMS offers to this network connectivity service to the applications and the entities of EIMS performing this function is called Network Service Manager (NSM). As the EIMS is distributed, NSM assigned at SP and NP domains cooperate with each other to realize the E2E chain. The NSM assigned at an NP should also cooperate with its domain network resource manager, [1], [3], [4], [5]. The ENTHRONE QoS architecture is scalable with respect to QoS related signalling overhead. The network connectivity service is first constructed at an overlay level. Based on forecasted information about future users and their traffic demands, the SP decides to construct in advance some logical multidomain QoS enabled aggregated pipes. This is done following pSLS subscription, which is performed via pSLS negotiations, between SP and first NP and then between successive NPs in a client-server fashion. Each request from a “client” entity is subject to Admission Control in the “server” entity given the fact that resources of the server are involved. The resulting unidirectional QoS enabled pipes are called pSLS links, [3], [5]. They could cross several IP core domains from the CSs up to various egress points (routers) where access networks of the anticipated CCs are connected. Later and in pSLS invocation phase, SP can decide to request that pSLS links be actually installed in the network (), instructing its NSMs to request the Network Resource Managers in each domain to execute the resource allocations for realising a QoS-enabled pipe between two points as specified in the pSLS. Then SP can individually “allocate” to CCs some part of pSLS-links (specified as cSLS-links) , for use by individual flows at CC requests. Thus, ENTHRONE solution avoids per flow inter-domain signalling. III ROMANIAN PILOT ISLAND This section describes a typical ENTHRONE Pilot island, “the Romanian Pilot Island (RPI)” located in Bucharest and shown in Figure 1. The network topology and technologies deployed as well as the EIMS entities that are present in the RPI are described below. Network Topology and Technology The topology is composed of three networking parts: 1) Multi-domain Core IP; three domains - linked via GEANT to other countries’ pilots
2) Access (Aggregation) - composed of wire-line technologies 3) Customer Premises Equipment (CPE) - composed of standalone hosts/terminals or LAN-based networks. Here, several CPE components may coexist. The terminals can be fixed or mobile with single mode or hybrid mode access. A number of transport technologies and tools are employed: as below: • •
LAN: WLAN-802.11b/g, Ethernet; Access Aggregation Networks (AAN): IP, IEEE 802.16d MAN, ADSL, DVB-T; • Core IP networks: LINUX and IP -based commercial routers, MPLS/IP DiffServ technologies and mechanisms; • Content Servers: TV and Multimedia Processors TVM), VoD servers; Streaming servers. The business entities specified in these scenarios are: CC, CP/CS, SP, and NPs. Several EIMS subsystems corresponding to business actors, each of them consisting of one or several functional entities are installed in RP Island as explained in the next sub-sections to offer support for defined tests and experiments. It should be noted that NSM sub-systems are used at both SP and NP domains. An EIMS-like manager capable of signalling with other network domain managers (NSM) is used at AAN. The infrastructure described above allows local experimentations of full ENTHRONE scenarios as well as global experimentations since it is interconnected to other countries’ pilots. Content Provider Manager and Server A Content Provider Manager manages content related negotiation and subscription with SP and provides the contents. It is installed on two hosts i.e., VoD and E-Learning servers providing Digital Item Description (DID) files and the associated audio video sources, i.e., TVM live encoding system. ENTHRONE software entities installed are as follows: ContentSRvMngr@CP - to manage content related negotiation with SP; CustSrvMngr@CP - to manage the CC requests for cSLS invocation. Note that CP and CS are currently implemented in ENTHRONE as a single TVM entity. Therefore, the notation CP/CS stands also for source (sTVM) if it is not specified otherwise. Service Provider’s EIMS Manager & SP Front End The SP’s EIMS Manger is installed on two hosts. This subsystem in RPI contains the following components: Customer Service Manager, Search Manager, E2EQoS Manager, DID Browser, Metadata Manager, Network Service Manager, Multicast Manager, and Service Monitoring. In addition, the SP includes an SP Front End for the interaction with the customers. A separate LAN connects the SP components installed on two computers enabling them communicate.
End-user Terminal Manager The CCs own Enthrone user terminals. The end-user terminal Manager software manages the communication between EIMS and the terminals, to display media packets received from content source or adaptation TVM, and to provide quality of service feedback to the EIMS. Network Provider Managers The main task of the EIMS@NP is to find, negotiate and establish a QoS enabled pipe from a Content Server to a Content Consumer, pipe established by a pSLS agreement. A Network Provider Managers (NetSrvMngr@NP) is installed in each of
ENTHRONE subsystems installed in RP Island
CPE: May be a single host or LAN or WLAN LAN/ WLAN
the three IP core domains of the RPI. The following components are installed at each IP domain: • Network Service Manager (NSM) • Inter-domain Resource Manager • Node monitoring and Network Monitoring • Intra-domain Network Resource Manager: specific blocks of NP for resource management and traffic engineering The assembly of these components provides the service and resource management for support of the network connectivity in the core network.
802.16d SS
EIMS
802.16d BS 802.16d SS
EIMS (NSM + Inter – NRM)
RMC@AAN
AN GW
LAN/ WLAN
802.d /SS
3 Core IP domains
802.16d BS
LAN
DSLAM
LAN/Hosts
Access Aggregation Network
DVB-T/IP receiver
DVB-T receiver terminal
DVB-T Transmitter
Data traffic
IP/DVB Multiplexer
Bandwith Manager
From TV Studio
IP Backbone GEANT)
Other Pilots
Signalling ( management and control)
Figure 1: The infrastructure of RP Island. AAN – Aggregation Access Network; RMC@AAN-– Resource Management and Controller at AAN; BS – 802.16d Base Station; SS – 802.16d Subscriber Station; AN-GW - Access Network Gateway; EIMS: (NSM – Network Service Manger; InterNRM Inter-domain Network Resource Manager); Intra-NRM - Intra-domain Network Resource Manager; CPE Customer Premises Equipments; DVB-T Digital Video Broadcast Terrestrial; DSLAM DSL Access Multiplexer
Access Network Managers Two types of Access networks exist in RPI: - Conventional Access Networks (LAN, WLAN) - Access Aggregation Network (AAN) where pSLS or cSLS links can be utilized. AAN constitutes an end hop between the core network and the local area networks. The RPI has IEEE 802.16d/WiMAX technology and DVB-T representing AANs. This pilot consists of two AN/AANs networks, both based mainly on WiMAX technology. One of them provides a hybrid access based on WiMAX and DVB-T. The IP data sent over DVB-T are extracted with an IP decapsulator and sent to the associated LAN. This link could be used for multicast streams while the WiMAX link is used mainly for unicast traffic. The 802.16/WiMAX technology will be used in ENTHRONE RPI in two ways: as AAN or simple AN. The following managers are necessary for the operation: - Network Service Manager @AANP (AANSM@AANP) having a similar role as NSM@NP except some functionalities related to traffic engineering. It manages the pSLSs through AAN. - Intra-domain Network Resource Manager@AANP (IntraNRM@AANP) Multicast Managers The RPI mainly demonstrates IP multicast based on PIMSM. In such a case, the appropriate functional blocks for managing PIM-SM in the last core domain [17], will be added to NetSrvMngr@NP in order to manage the pSLS link based tree subscription and invocation. IV TESTING METHODOLOGY Test Taxonomy and Scenarios The types of tests to be carried out in ENTHRONE pilot islands can be classified as belonging to one of the three categories defined in the general methodology [16]: • Integration/Validation Tests: to ensure that the functionality complies with the ENTHRONE functional architecture. The emphasis is on proving the functionality and validating the correct behaviour of components/sub-systems under test. • Algorithm/component level Performance Assessment Tests: to examine the efficiency of the algorithms or components, measure the computing resources consumed by the algorithm during its execution, etc. • System level Performance Assessment Tests: to determine whether the overall objectives of the ENTHRONE E2E integrated QoS management systems have been realized. Different aspects can be emphasized on: benefit/cost, scalability, reliability, stability, etc. The tests can target the functions and/or performance of the system at Management, Control and Data Plane separately or together. To perform the tests appropriate scenarios are defined to provide the necessary framework allowing to emphasize the
required properties (functional and/or performance). One scenario defines the test environment and a sequence of actions (inputs, internal actions and expected outputs) to perform. We distinguish between Operational (in terms of functionality), technical (in terms of correctness, scalability, stability, etc.), and usability (cost/benefit to provider and end-user) Scenarios relatively to a given subsystem to be tested; High level services Scenarios- display services like VoD, streaming, E-learning, etc., and benefits to be seen from user perspective. See Figure 2. Scenarios hierarchy
High level Scenarios Operational Scenarios
Performance
Validation/Integration (Functional correctness)
Figure 2: Hierarchical representation of tests and scenarios. Test structure The tests are structured based on their intended purposes i.e, the validation/integration, performance assessment tests at the algorithm/component and system levels. The structure of the test suite is hierarchical, and the suite is composed of test groups, which are in turn composed of elementary tests [ISO9646-1], [ISO-9646-2], [ISO-9646-7]. The reference of an elementary test can be defined below [16]: Test Suite Identifier/Standardization Organization/Protocol Identifier/Group ID/Test ID. For example: Suite1_0/UPB/NSM/Uni_pSLS_Sub/x For system level tests, the test purpose is defined in a detailed structure, loosely based on the format of Test Purposes for Informal Testing, as described in [ETSI ETR 266]. For Integration Tests, a simpler Test Purpose format is used. Methodology for Validation and Integration of ENTHRONE subsystems The objective is to verify the functional correctness of the signaling chains and/or internal behaviour of involved functional blocks. Here, the integration tests involve a “network” of interconnected subsystems to be tested. The type of validation is deterministic, i.e. we have 100% validation of functional correctness in scenarios for success or failure. The following generic methodology can be applied to validate a subsystem. As already shown, two main classes of tests are envisaged that is functional (correctness) and performance aspects.
The general steps of validation are the following:
correct “macro-MSCs” and correct content of statefull databases. The macro-MAC defines the main milestones for the subsystem behaviour as to give us the chance to state that it is functioning correctly.
I. Preparation for execution of tests: 1. Construct the subsystem environment and couple it to the subsystem to be validated. 2. Determine some constraints guidelines (what stimuli can be applied to the subsystem). 3. Establish the correctness and performance requirements. 4. Define the guiding scenario. 5. Define the correct behaviour (in terms of a desired Message Sequence Chart (MSC) and the performance thresholds
System level Sub-system's environment emulator
13
NetSrvMngr @NP1
1 12
NetSrvMngr@SP
CP/CS
11
10
Guiding constraints
Guiding scenario
Correctness and perf. requirements
& perf. event detection, measurement measured values Behaviour input/ Tracing output
Subsystem (to be validated)
Verdict OK/not_OK Behaviour trace(MSC)
Figure 3: Generic validation methodology of a Pilot subsystem. For performance related validation, specific metrics should be defined for Management and control plane (number of messages per set of actions, signaling delays, number of unnecessary/lost actions, etc.) and for Data plane (NQoS, pQoS related performance parameters, resource utilization degree, network load, etc.).
EQoS-pSLS
SP Admin
0.pSLSinv req
event detection
Execution environment
Verification,
II. System runs: 6. Apply stimulus messages at subsystem inputs (emulating the environment inputs to the system). 7. Guide the activity depending on subsystem reaction. 8. Collect the Message Sequence Chart (MSC) of the actual behaviour. 9. Collect status and quantitative information displayed at each functional entity debugging console. 10. Compare the MSC to the correct one and compare the performance values with thresholds. 11. Concludes a verdict/decision ok/not_ok based on 9 and 10. 12. Readjust if it is the case the system code to eliminate deadlocks, live locks, and undesired responses, adjust performance, etc. The metric used in the tests will specifically depend on the scenario. For functional validation, the metrics consist of identity of the actual MSCs obtained by running the subsystem, the SP
guiding action
EQoS-pSLS NetSrvMngr @NP2
2 9
7
8
3 6
NetSrvMngr @NP3
5
4
Intra-NRM @NP1
Intra-NRM @NP2
Intra-NRM @NP3
AS1
AS2
AS3
Figure 4: Basic pSLS invocation signalling chain (3 IP domains). V OPERATIONAL SCENARIO EXAMPLE: VALIDATION AND INTEGRATION OF NETWORK SERVICE MANAGEMENT FUNCTIONS
In this section, we give sample examples out of a complex suite of tests under Operational Scenario: QoS enabled Connectivity Network Services. Below it is shown a summary of local Pilot test Scenarios. All these tests are aimed at validating the correctness of the Network Service Manager (NSM) implementation behaviour relative to pSLS subscription, invocation and respectively cSLS subscription and invocation in a
multi-domain environment in unicast case. The tests are denoted by: SUITE1_1/UPB/NSM/Uni_pSLS_Sub/x SUITE1_1/UPB/NSM/Uni_pSLS_Inv/x SUITE1_1/UPB/NSM/Uni_cSLS_Sub/x SUITE1_1/UPB/NSM/Uni_cSLS_Inv/x EXAMPLE pSLS-link invocation setup - success scenario SUITE1_1/UPB/NSM/Uni_pSLS_Inv/1 The sequence of actions is summarized in Figure 4 and below, as a walkthrough list.
1. Admin@SP constructs of the pSLS XML file and sends it to ServProv@SP.
8. If AC is ok, then NetSrvMngr@NP1 books the resources and generates towards NetSrvMngr@NP2 an invocation request.
2. The ServProv@SP prepares the pSLS negotiation/ invocation object and requests a pSLS-link invocation from NetSrvMngr@SP.
9. NetSrvMngr@NP2 executes the same actions as in 5,6,7,8 and extends the signaling to NetSrvMngr@NP3.
3. NetSrvMngr@SP consults the pSLS subscription tables and checks the validity of data. It determines also the NP to communicate with.
10. NetSrvMngr@NP3 determines that the pSLS-link egress point belong to this domain (NP3) and makes local AC decision. If AC is ok, then it registers this pSLS.
4. NetSrvMngr@SP starts invocation signalling via EQoSpSLS protocol by sending a pSLS_i_req message NetSrvMngr@NP1.
11. NetSrvMngr@NP3 gives command to Intra-domain NetResMngr@NP3 to install resources in the network.
5. NetSrvMngr@NP1 determines whether pSLS-link requested ends in this domain. We assume a longer pipe in this scenario. 6. NetSrvMngr@NP1 compares the invocation parameters to the subscription ones and rejects the request if the amount of invoked parameters is greater than subscribed ones. 7. NetSrvMngr@NP1 splits the invocation request parameters of the pSLS in local part and remote part. It makes local Admission Control (AC) decision for the local segment and also AC for the inter-domain segment to NetSrvMngr@NP2. A d m in @ SP
N e tS rv M ngr @ SP
S rv P ro v @ SP
N et S rv M n gr @ N P1
SP
12. Intra-domain NetResMngr@NP3 returns positive answer to NetSrvMngr@NP3. 13. NetSrvMngr@NP3 NetSrvMngr@NP2.
positive
16. SrvProv@SP register the pSLS-link invocation in the pSLS repository 17. SrvProv@SP returns positive answer to Admin@SP The correct MSC obtained during experimentations is shown in Figure 5. N et S rv M ngr @ NP2
In t ra NRM @ N P1
In t ra NRM @ N P2
NP2 E Q o S -p S L S -S / I -N P
In t ra NRM @ NP3
N et S rv M n gr @ N P3
NP3 E Q o S -p S L S -S / I -N P
2 .2 .p S L S _ s _ req 3
4 ..p S L S _ i _ req 5 , 6 ,7
8 . p S L S _ s _ re q
In v o c a ti o n In i tia t io n 9 .In te rn a l N P a c ti o n s
L a s t IP d o m a in
10
11 12
1 3 .p S L S _ i _ rsp (a c c )
M1
p S L S _ i _ rs p (a c c ) . p S L S _ i _ rsp (a c c )
M2 1 5 p S L S _ i _ rsp (a c c )
M3
16 17
to
15. NetSrvMngr@SP returns positive answer to SrvProv@SP.
1 .2 .p S L S _ i _ req 2 .1
answer
14. Similar actions are repeated at NP2 and NP1
NP1 E Q o S -p S L S -S /I -N P
1 .1
returns
14
M4
Figure 5: Basic pSLS – invocation setup Scenario (success case).
VI CONCLUSIONS This paper described briefly the Enthrone project framework and proposed a testing methodology that is applied to test the Network Service Management of the ENTHRONE system in a large testbed/pilot composed of several IP network domains deploying other ENTHRONE designed entities. The methodology proved to provide an efficient and systematic way to validate/integrate and demonstrate the defined operational scenarios as well as high-level service scenarios managed by a complex distributed but integrated management system. Many other scenarios that will be experimented based on this methodology in the coming months are described in [16]. REFERENCES [1] [2] [3] [4] [5] [6]
Brétillon, P. (ed) et al.: ENTHRONE II Deliverable D01, “Overall System Architecture –Version 2”, February 2007, http://www.istenthrone.org. Sommer, P. (ed) et al.: ENTHRONE II Deliverable D02, “Pilot Architecture and Services Definition”, February 2007. Souto, P., Ransburg, M. (eds) et al.: ENTHRONE II Deliverable D03, “EIMS for ENTHRONE II”, February 2008. Sidibé, M., Mehaoua, A. (eds) et al.: ENTHRONE II Deliverable D06, “Service management and monitoring”, February 2008. Borcoci, E. (ed) et al.: ENTHRONE II Deliverable D18i, “Service Management and QoS provisioning”, intermediate version, July 2007. T.Engel, et al, “AQUILA: Adaptive Resource Control for QoS Using an IP-Based Layered Architecture”, IEEE Communications Magazine, January 2003, pp. 46-53. See also http://www-st.inf.tu-dresden.de/aquila/
[7]
[8]
[9] [10]
[11]
[12] [13] [14] [15] [16] [17]
P. Morand, et. al.,“Final specification of protocols and algorithms, for inter-domain SLS management and traffic, engineering for QoS-based IP service delivery” MESCAL IST Project Public Deliverable, D1.3, July 2005, www.mescal.org P. Morand, et.al., “Final specification of protocols and algorithms, for inter-domain SLS management and traffic, engineering for QoS-based IP service delivery”, MESCAL Deliverable, D1.3, July 2005, www.mescal.org T.Damilatis, et.al., Final Architecture, Protocol and Algorithm Specification, TEQUILA IST Project Public Deliverable, D3.4 Part B, Oct. 2002. T.Ahmed, A.Asgari, A.Mehaoua, E.Borcoci, L.Berti-Équille, G..Kormentzas “End-to-End QoS Provi-sioning Through an Integrated Management System for Multimedia Content Delivery” – submitted to Computer Communication Journal, May 2005. E.Borcoci, A.Asgari, N.Butler, T.Ahmed, A.Mehaoua, G.Kourmentzas, S.Eccles “ Service Management for End-to-End QoS Multimedia Content Delivery in Heterogeneous Environment”, AICT Conference, July 2005, Lisbon. ISO 9646-1;1994 Information Technology - Open Systems Interconnection - Conformance Testing Methodology and Framework - Part 1: General Concepts. ISO/IEC 9646-2:1995 Information technology -- Open Systems Interconnection -- Conformance testing methodology and framework -- Part 2: Abstract Test Suite specification. ISO/IEC 9646-7:1995 Information technology -- Open Systems Interconnection -- Conformance testing methodology and framework -- Part 7: Implementation Conformance Statements. ETSI ETR 266: Title: Methods for Testing and Specification (MTS); Test Purpose style guide. T. Ahmed (ed) et al., ENTHRONE II D27: Pilot and services integration and tests, February 2007. P. Souto (ed) et al., ENTHRONE II D21: Multicast provisioning and multicast service management
