hardware when monitoring network parameters and high level video parameters ... general purpose network monitoring tool, which in turn uses libpcap [3] as an ...
Distributed Video Quality Monitoring Nick Vercammen
Nicolas Staelens
Brecht Vermeulen
Piet Demeester
Ghent University - IBBT Department of Information Technology (INTEC) Gaston Crommenlaan 8, Bus 201, 9050 Ghent, Belgium Abstract
network can be detected. Measuring Quality of Experience combined with the measurement of network statistics allows operators to link quality losses to the causes at the network level. This paper describes an architecture to monitor video streams in IP networks in a distributed way. We also answer the question whether it is possible to measure network statistics and video application layer data for a full High Definition1 video stream in real-time with commodity hardware. The paper is organized as follows. In section two we describe some related work in the area of video monitoring in IP networks. Section three discusses the general monitoring architecture and goes into more detail on how packets are captured and processed in the network probe. The fourth section answers the question whether it is possible to perform real-time monitoring of HD video streams using commodity hardware. Finally we conclude the paper with some conclusions and future work.
Real-time Quality of Experience monitoring becomes an important field-of-interest for video service providers. Early detection of impairments caused by, for example, packet loss in the IP-based delivery network enables service providers to take the necessary actions in order to maximize end-users’ perception of quality. The influence of network impairments on visual quality depends on characteristics of the video stream such as the amount of motion and spatial detail. Therefore, streams should be monitored and parsed beyond the network layer. In this paper, we present a distributed video stream monitoring architecture and evaluate the real-time capability of parsing video streams up to the video layer. Our results indicate that it is possible to process a single video stream up to 60Mbit/s using commodity hardware when monitoring network parameters and high level video parameters such as frame type. A higher total bitrate can be achieved when monitoring multiple video streams encoded at lower bitrates. However, when detailed video parameters such as motion vectors and residual coefficients must be gathered, specialized hardware is needed.
1
2
Related Work
This section will describe some tools for network and multimedia monitoring. We will first start with tools for high level network monitoring and then describe systems that digg deeper into the network packets up until the video application layer. The tcpdump [4] tool is a well known and widely used general purpose network monitoring tool, which in turn uses libpcap [3] as an underlying library to capture packets from a network interface. Tcpdump captures packets, parses the ethernet, IP and transport layer headers and prints the data to a file or the console. By no means tcpdump keeps track of stream status or analyzes application layer data. Tools like rtpdump [8], rtpmon [2] and mmdump [5] are more geared towards multimedia monitoring. rtpdump decodes and prints RTP packet headers, rtpmon monitors RTP sessions and displays statistics based on the contents of
Introduction
In recent years video applications like video conferencing and live video streaming have gained interest and it is expected that their importance in IP networks will continue to grow. Due to the real-time character of these applications a connectionless transport protocol like UDP is typically used. Hence problems within the network can result in packet loss and late arrival of packets causing a degradation in quality of the received video stream. For video service providers it is a challenge to ensure that their service provides the minimum quality expected by the users. To ensure end-to-end quality a proactive approach is desired, by monitoring video streams at strategic demarcation points along the delivery chain, problems within the
1 resolution:1920x1080
1
RTCP packets. mmdump keeps track of multimedia sessions by capturing packets on fixed ports for control protocols like RTSP and H.323. By analyzing the contents of these packets it then dynamically changes the capture filter so that the multimedia data packets can be captured and statistics like the number of packets and the number of bytes per multimedia stream can be gathered. None of these tools parse the contained multimedia data or keep track of jitter and packet loss. The authors in [10] describe a full video delivery network monitoring suite. The perceived quality of a video is measured at the client side, by using a technology called Pseudo Subjective Quality Assessment (PSQA) [6]. At the video layer frame losses and mean frame loss bursts are monitored and used as input to the PSQA algorithm. It should be noted that extra information is inserted into the video stream when the video is streamed. This extra data together with some data from the video stream (e.g., average frame size, video bitrate, etc) are monitored by an adapted version of VLC [12] player. As there are currently, to our knowledge, only commercial tools available that enable real-time parsing of encoded video data, we have focused on the design of a system that is capable of monitoring video streams by means of an indepth inspection of the individual network packets. Besides monitoring the network layer, we also gather video application layer statistics by analyzing RTP headers and MPEG4/AVC data. The measured data will serve as input for a video quality metric currently under development.
3 3.1
Monitoring Architecture Monitoring platform
The video quality monitoring platform will monitor the entire network from the video head-end to the home network. The goal is to obtain information about the flow at several demarcation points through the network. By combining the information measured at each demarcation point, the monitoring platform should obtain a complete view on the quality of the video streams transmitted through the network. The monitoring architecture is illustrated in Figure 1 and consists of three main components. The first is the monitor probe which will perform the actual monitoring and can be queried to obtain the performed measurements. Different monitor probes will be distributed across the network. The second component is the management node which will collect all measurement data. The third and final component in the architecture is the measurement archive which will aggregate all monitored information and store it in a database.
Figure 1. Monitoring platform: monitor probes distributed accross the network, a central management node to gather the data and measurement archive to store the monitored data
Both the management node and the measurement archive are located on one central place. The next section provides both a functional and technical overview of the monitor probe, the other components will not be handled as they are out of the scope of this paper.
3.2
Monitor Probe
The monitor probe monitors the network at a specific demarcation point. It measures both network related parameters like packet loss, pack count, jitter, stream bit rate, etc and video related parameters like frame rate, resolution, GOP2 structure, motion vectors, etc. Monitor probes can be configured to perform different levels of packet inspection depending on their location, some will simply calculate network statistics (e.g. probes located at the DSLAM) while others perform deep packet inspection up to the video layer (e.g. probes located at the video head end and client side). The latter requires partly decoding of the video stream, which is very CPU intensive. Each monitor probe periodically exchanges its monitoring information with the management node. A monitor probe can also raise an alarm to the management node when certain monitored parameters exceed a threshold. Figure 2 shows an overview of the monitor probe consisting of four different layers: the packet capture layer, the media monitor plugin layer, the persistency layer and the user layer. We will discuss each layer bottum up. 3.2.1
Packet Capture Layer
The packet capture layer is responsible for capturing the network packets from the network interface and making 2 Group
Of Pictures
User Layer Web interface: php implementation
Persistency Layer MySQL database Monitor probe database communication
Media Monitor Plugin Layer RTP monitoring
H.264 monitoring
are lost within a GOP3 . These results will serve as input to a model to express the subjective quality of the video. In the most complex configuration the packet handler will parse the complete bitstream, by doing this we can determine the motion vectors and residual coefficients available within a frame. With this data, content classification can be made based on the scene complexity. Type of content is an important input for a reliable prediction of the subjective quality [7]. This configuration is however very CPU intensive because a bit by bit parsing of the captured data has to be performed.
Packet Capture Layer libpcap
Figure 2. Monitor probe software architecture
the data available to the application. In our implementation we use libpcap to capture the network packets. The monitor probe should be able to handle multiple high definition video streams, so we need to be able to reliably capture packets at high bit rates.
3.2.3
At a regular interval, currently one second, the measured data is retrieved from the monitor probe and stored in a database. This is done in a seperate thread to keep the packet capturing and handling process as light as possible. As a database, a MySQL server is used but the software is easily extensible to allow the use of other database systems. The database keeps track of the historical data, periodically old data will be erased to prevent storage issues. 3.2.4
3.2.2
Media Monitor Plugin Layer
Once the packets are captured from the underlying network they will be handled by the media processing core. Within this media processing core different monitoring components can be easily plugged. Each component has a different function and parses the packets at a different level. Currently two components are developed: the RTP packet handler and the H.264/AVC packet handler. The RTP packet handler measures certain parameters by parsing the RTP headers. This way lost packets and out of order packets can be determined, both will be stored as lost packets in correspondence with RFC3550 [9]. The video bitrate is determined by counting the number of bytes of the RTP payloads, interarrival jitter for video packets is calculated using the algorithm described in RFC3550. The H.264/AVC packet handler is responsible for parsing and analysing the AVC video data send as payload within RTP packets. This handler is able to parse a standard AVC compliant [1] bitstream and is configurable to analyse this bitstream into different levels of details. In the simplest configuration, the packet handler will determine the frame types that are received. This way we can keep track of the data that is lost within a GOP, by comparing sequential GOP’s with each other we can determine what frames
Persistency Layer
User Layer
A web interface allows operators to review the monitored data. The web interface is implemented in php and runs on an apache2 web server, the graphs are generated using flash.
4
Experimental Results
Figure 3. Network setup for evaluating the monitor probe
To evaluate the performance of the monitor probe a number of experiments were set up on the video testbed and executed with the help of our test management software described in [11]. Figure 3 presents a high level overview of the network setup used to execute the experiments. Video sequences are streamed from server to client, on the switch 3 In
case of a fixed GOP
80
80
60 Total CPU usage Kernel usage
40
Packets monitored 20
Percentage (%)
100
Percentage (%)
100
60 Total CPU usage Kernel usage
40
Packets monitored 20
0
0 30
40
50
60
70
80
Bitrate (Mbit/s)
(a) Default libpcap socket buffer
30
40
50
60
70
80
Bitrate (Mbit/s)
(b) Increased libpcap socket buffer (10000*1538)
Figure 4. Monitor probe performance for a single video stream at a constant bitrate using the default libpcap socket buffer size (a) and after increasing the buffer size (b).
all network data is mirrored to the monitor probe. The network can handle network streams of 100Mbit/s in both directions, this was validated by extensive throughput tests. The specifications of the machine used to run the monitor probe are shown in table 1. CPU Memory Network interface OS
AMD Athlon(tm) 64 Processor 3000+ (2000Mhz) 500MB 82541GI/PI Gigabit Ethernet Controller Linux 2.6.18
Table 1. Specifications of machine running the monitor probe
In a first test the monitor probe was configured to measure all video data, practically this means that all bytes of the video stream have to be read and analysed. Doing a test with a relatively simple video sequence, 750Kbit/s and resolution 320x240, quickly showed us that this task is too CPU intensive. The CPU is fully occupied in the parsing process, resulting in buffer overflows on the network socket and dropped network packets. So currently it is not possible to extract content specific information in real-time from a streamed video with our software solution and the available hardware. In a second test the monitor probe was configured to measure only network parameters and high level video parameters like frame type. The goal of this test was twofold: first we wanted to find out how much data the probe can
handle in this configuration and second we wanted to know what the bottlenecks are when monitoring video streams. In this test a 100s long HD (1080p) video sequence, encoded with x264 at a constant bitrate, was streamed and monitored. The video was encoded in different bitrates ranging from 30Mbit/s to 80Mbit/s. On the machine running the monitor probe, the CPU usage of the monitoring process was measured. Together with the data monitored by the monitor probe and the data from the streamer, the performance can be evaluated. In Figure 4 we see that the CPU usage of the monitor probe rises linear with the bitrate, hence the number of packets streamed. At a certain point the application is no longer able to monitor all packets. Some packets never reach the application because the socket buffer, allocated by libpcap, ran out of space and the packets are dropped. In Figure 4(a) we see that packet loss occurs for a bitrate of 50Mbit/s and more significant losses for 60Mbit/s and more. The CPU usage at 50Mbit/s indicates that the application is not using all CPU resources, still some packets are lost. When analysing the time it takes to handle a packet we notice that on average, it takes about 176µs, waiting for a packet and retrieving one from libpcap adds an extra overhead of 6µs. An average handle time per packet of 182µs would result in a monitoring ability of 5494 packets/s. For a stream of 50Mbit/s, 4440 packets are streamed per second so theoretically it should be possible to monitor them without losses. The cause of these losses is to be found in the variation of packet handling times. Some packets need deeper inspection, e.g. when a complete frame is received, and they will take longer to analyse. When a database update occurs, the packet handling thread will lock, because the software is run on a single core machine, causing an
90
100
90
80
80 80
60 60
50 40
40
30
Packets monitored Total CPU usage Bitrate
Percentage (%)
70
Bitrate (Mbit/s)
Percentate (%)
80
20
20
70 60
60
50 40
40
30
Packets monitored Total CPU usage Bitrate
20
20
10
10 0
0 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190
0 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190
0
Bitrate (Mbit/s)
100
Time (s)
Time (s)
(a) Default libpcap socket buffer
(b) Increased libpcap socket buffer (10000*1538)
Figure 5. Monitor probe performance for multiple video streams at constant bitrate using the default libpcap socket buffer size (a) and after increasing the buffer size (b).
increase in the packet handling time. We noticed that for some packets it takes 5 to 10ms to handle, at that point packets will fill the socket buffer space and eventually the buffer will overflow. The default socket buffer size allocated by libpcap on a linux machine is 2MB, or 1363 packets expressed in packets of 1538 bytes. To achieve our theoretical monitor speed, we adapted libpcap to allocate a buffer to hold 10000 packets of 1538 bytes. The result is shown in Figure 4(b), we are now able to monitor video streams of 60Mbit/s or 5340 packets/s. For this bitrate the CPU usage is above 95% and is fully occupied with capturing and parsing packets. The same test was repeated, but instead of streaming one stream at a time and increasing the bitrate of it, we started with one stream of 8Mbit/s and added a new one every 10 seconds. At 90 seconds 10 videos are streamed simultane, after 100 seconds the first video is finished and so on. It is shown in Figure 5(a) that starting from 50 seconds, 6 simultaneous streams (42Mbit/s), the application is starting to miss some packets in case of a standard libpcap socket buffer. Notice that once the buffer is full, it takes a long time to recover. It takes until 180s, when only one stream is left, to completely measure all packets again. In case of an adapted libpcap buffer (see Figure 5(b)), all packets can be monitored, even at bitrates as high as 80Mbit/s. The buffer is large enough and the peak of 70Mbit/s and 80Mbit/s short enough for the application to recover from these bitrates that it normally is not able to monitor. Knowing that video bitrates for AVC encoded sequences are never constant but have high peaks, a large socket buffer is important for reliable video monitoring.
5
Conclusion and future work
In this paper, we presented a system for real-time video monitoring in distributed locations within the network. This system is able to monitor both network parameters and video parameters. The depth of packet inspection can be configured to monitor low level video characteristics like motion vectors and residual coefficients or to monitor network parameters and high level video characteristics like frame type. Tests showed that it is currently not possible to perform the low level video monitoring with our software solution and commodity hardware. If video characteristics like the amount of motion and spatial complexity are needed for a video quality metric, a different approach is required. These characteristics could for example be inserted into the video stream in special packets. On the other hand tests showed that the monitor probe in its simplest configuration can handle single video streams with a bitrate up to 60Mbit/s. To achieve this bitrate the socket buffer size allocated by libpcap has to be increased (14.7MB). Higher bitrates can be achieved when monitoring multiple video streams, given that the peak periods are short enough. In case of buffer overflow, the application needs a long time to recover. We can conclude that given the nature of video bitrates, a large buffer is needed for reliably monitoring video. In monitoring video, the bottleneck is not the capturing of the packets but the processing of them. So if higher bitrates have to be achieved, these processing times will have to be reduced or the packet processing will have to be decoupled from the capturing process.
In future we will start using multicore machines to test the monitor probe. We will make advantage of the multiple cores by decoupling the capture process from the packet handling process. Looking at current CPU usage for the monitor probe we believe that a 50% gain can be made by decoupling them. We will also add some new measurements to the monitor probe like packet loss period, time between loss periods, etc.
6
Acknowledgements
The research activities that have been described in this paper were funded by Ghent University, the Interdisciplinary Institute for Broadband Technology (IBBT) and the Institute for the Promotion of Innovation by Science and Technology in Flanders (IWT). This paper is the result of research carried out as part of the Video Q-SAC project funded by the IBBT. Video Q-SAC is being carried out by a consortium of the industrial partners: Alcatel-Lucent, Telindus, Televic and fifthplay (Niko Group) in cooperation with the IBBT research groups: IBCN & MultimediaLab (UGent), SMIT (VUB) and IMEC.
References [1] ITU-T Recommendation H.264 : Advanced video coding for generic audiovisual services, November 2007. [2] D. Bacher, A. Swan, and L. Rowe. rtpmon - a thirdparty rtcp monitor. http://www.cs.columbia.edu/ ˜hgs/rtp/rtpmon.htm. [3] V. Jacobson, C. Leres, and S. McCanne. pcap - packet capture library. UNIX man page. [4] V. Jacobson, C. Leres, and S. McCanne. tcpdump - dump traffic on a network. UNIX man page. [5] J. V. D. Merwe, R. Caceres, Y. hua Chu, and C. Sreenan. mmdump: A tool for monitoring internet multimedia traffic. 2000. [6] S. Mohamed and G. Rubino. A study of real-time packet video quality using random neural networks. IEEE Transactions on Circuits and Systems for Video Technology, 12(12):1071–1083, December 2002. [7] A. Reibman and D. Poole. Characterizing packet-loss impairments in compressed video. In IEEE International Conference on Image Processing (ICIP), volume 5, October 2007. [8] H. Schulzrinne. rtpdump. http://www.cs. columbia.edu/˜hgs/rtp/rtpdump.html. [9] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson. RTP: A transport protocol for real-time applications (RFC3550), 2003. [10] D. Vera, P. Rodrguez-Bocca, and G. Rubino. Qoe monitoring platform for video delivery networks. In 7th IEEE International Workshop on IP Operations and Management (IPOM’07), November 2007.
[11] N. Vercammen, N. Staelens, A. Rombaut, B. Vermeulen, and P. Demeester. Extensive video quality evaluation: A scalable video testing platform. In Computer and Information Technology, 2008. ICCIT 2008. 11th International Conference on, pages 91–97, December 2008. [12] VLC Media Player. http://www.videolan.org.
