IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 8, NO. 5, OCTOBER 2006
1005
Efficient Streaming Packet Video Over Differentiated Services Networks Fei Zhang, James Macnicol, Member, IEEE, Mark R. Pickering, Member, IEEE, Michael R. Frater, Member, IEEE, and John F. Arnold, Senior Member, IEEE
Abstract—We investigate streaming video over Differentiated Services (Diffserv) networks that can provide a number of aggregated traffic classes with increasing quality guarantee. We propose a method to measure the loss impact of a video packet on the quality of the decoded video images. We show how the optimal Quality-of-Service (QoS) mapping from the video packets into a set of traffic classes depends on the loss rates of the classes and the pricing model, and we develop an algorithm to accurately find the optimal QoS mapping. The performance of our algorithm is evaluated through computer simulations and compares favorably to an existing algorithm. Index Terms—Differentiated services networks, error resilience, video compression.
I. INTRODUCTION N RECENT years, there has been a growing demand for streaming multimedia applications over the Internet. In particular, video communication over the Internet has attracted much research interest. Providing good image quality in real-time or near real-time video communications such as video-conferencing, video-telephony, and video-on-demand, is difficult with the current “best effort” Internet model. This same-service-to-all paradigm has become increasingly inadequate for Internet applications (including streaming video)that have diverse Quality-of-Service (QoS) requirements. In this effort, the Internet Engineering Task Force (IETF) has proposed two distinct approaches for service differentiation: the Integrated Services (Intserv) [1] and Differentiated Services (Diffserv) [2] architectures. However the Intserv architecture has received very limited acceptance among the network community due to its problem of nonscalability and nonmanageability [3]. On the other hand, the Diffserv approach is more recent, and its main goal is to provide a scalable and manageable network with service differentiation capability. Unlike the Intserv approach where each router tracks individual flows, the core of a Diffserv network only distinguishes between packets marked as belonging to different aggregate flows, also known
I
Manuscript received March 24, 2003; revised November 21, 2005. This work was supported in part by the Australian Research Council under Grant A00104347. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Chang Wen Chen. TF. Zhang, M. R. Pickering, M. R. Frater, and J. F. Arnold are with the School of Information Technology and Electrical Engineering, Australian Defence Force Academy, The University of New South Wales, Canberra, ACT 2600, Australia (e-mail:
[email protected];
[email protected];
[email protected],
[email protected]). J. Macnicol, deceased, was with the School of Information Technology and Electrical Engineering, Australian Defence Force Academy, The University of New South Wales, Canberra, ACT 2600, Australia. Digital Object Identifier 10.1109/TMM.2006.879865
as service classes. No absolute performance guarantees are provided in this type of network, only that aggregates with higher priority will receive preferential treatment to those of lower priority. In this work we only consider Diffserv Assured Forwarding [4] networks. One of the key questions that must be addressed in the development of Diffserv network architectures is how to efficiently utilize a given Diffserv network model for multimedia streaming. In this effort, a number of research groups have investigated video streaming over simulated Diffserv networks [5]–[13]. Our new contributions are as follows. • We propose a simple method to measure the loss impact of a video packet on the decoded video quality when the video sequence under consideration has periodic I-frames, taking into account error propagation effects in inter-coded frames. • We develop a new algorithm to find the solution to the QoS mapping problem, i.e., how to assign video packets to the available Diffserv classes. • We demonstrate through simulation that the new QoS mapping algorithm is more effective in minimizing the total loss impact of a video packet stream that passes through a Diffserv network, and therefore, can produce better video quality than the existing QoS mapping algorithm of Shin et al. [6]. The rest of this paper is organized as follows. In Section II, we discuss how to measure the loss impact of a video packet. In Section III, we derive the optimal QoS mapping algorithm. Simulation results are presented in Section IV and some concluding remarks are made in Section V. II. PACKET LOSS IMPACT ON VIDEO QUALITY In order to use a Diffserv network effectively, it is necessary to know the loss impact of every video packet, so that those packets with higher loss impact may be sent via an appropriate higher priority traffic class that suffers lower loss probability. (In this paper, we only consider the loss effect. We assume that delay and jitter effects can be absorbed by using a large play buffer in the decoder.) There are many methods available for estimating packet loss impact with various levels of computational complexity that could be substituted for the technique described below, see [14] for a recent overview. To measure the loss impact of a video packet quantitatively, we assume that the decoder is using a normative error concealment algorithm. For the first I-frame temporal concealment is not possible and spatial interpolation must be used. For subsequent frames we employ a temporal concealment scheme by using a neighboring MB’s motion vectors, if available, or zero
1520-9210/$20.00 © 2006 IEEE
1006
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 8, NO. 5, OCTOBER 2006
III. OPTIMAL QOS MAPPING ALGORITHM
motion vector if no reliable neighboring macroblock is available. Under this error concealment scheme, if a packet is lost in the th frame the initial error can be calculated as
Once each video packet has been assigned a loss impact (denoted as ), we consider how to send the packet stream through a relatively differentiated services network that consists of a set . Class of Diffserv classes numbered as experiences a packet loss rate and attracts a per-packet price . If the classes are ordered by increasing price therefore the losses in each class must be related by . We assume the sender has accurate knowledge of the network state e.g. by receiving feedback from the receiver. Under a given price constraint, there are many possible mappings from the packets to the Diffserv classes. From these mappings we want to find an optimal one that minimizes the total loss impact, and consequently maximize the received video quality. Here the total loss impact can be expressed mathematically as
(1)
(4)
is the where the sum is over all the corrupted pixels, and difference between the decoded signals with and without the is the difference error due to the packet loss. In other words, between the correctly decoded texture for frame and the output of the error concealment algorithm attempting to recover from the loss of the packet. when This initial error will propagate into the next frame corrupted pixels are used for motion-compensated prediction:
where denotes the Diffserv class to which the th packet is mapped. The price constraint is expressed as where is the per packet price budget. We note that this constrained optimization problem was also formulated in [6]–[8], where the optimal solution was called the optimal QoS mapping. However, they used a coarse grained method to solve the optimization problem. The packets were first classified into a number of categories with each assigned an average relative priority index (loss impact). The optimal mapping from the categories to Diffserv classes was then searched for. In the following we will show how to solve the optimization problem directly using a Lagrange multiplier approach. We first sort the packets into ascending order according to the loss impact. Then it can be shown that the optimal mapping packets are must be an ordered mapping such that the first mapped to the lowest priority Diffserv class, the next packets to the second lowest level Diffserv class, and so on. In such a scenario, the total loss impact can be expressed as
Fig. 1. Optimal QoS mapping algorithm based on recursive relation (9).
(2) is the number of times the corrupted pixel is referwhere enced by the th frame. This temporal error propagation will only be stopped by macroblock intra-refresh or an I-frame. We consider a compressed video sequence with periodic , then the total squared I-frames. If we assume that error distortion due to the packet loss in a frame, which is frames away from the immediately previous I-frame, would be , where is the distance between adjacent I-frames. Therefore, we define the loss impact of a given packet in the th frame as (3) This loss impact measurement shows that a packet loss in an I-frame will generally result in a greater (negative) impact on the decoded video quality, because the initial error has a long propagation effect. The present method is simpler and requires less computational effort than that proposed in [6], where three factors were taken into account in determining the relative priority index of a video packet. The three factors are the initial , the average motion vector size, and the number of error intra macroblocks contained in a packet, with weights of 0.7, 0.15, and 0.15 respectively. In Section IV, we will compare the performance of these two methods.
(5) And the price constraint becomes:
(6) Therefore, we are faced with an optimization problem with integer decision variables . The feasible solution is defined by the price constraint (6). To deal with the constraint we combine (5) and (6) to form the Lagrangian:
(7)
ZHANG et al.: EFFICIENT STREAMING PACKET VIDEO
1007
At the maximum value of the constrained optimization problem all partial derivatives of the Lagrangian with respect are zero: to the variables (8) Note that is the packet loss impact at the margin between . Eliminating the parameter , we obtain the classes and recursive relation: (9) must then be set so The number of packets in each class satisfies (9). (9) does not the corresponding loss impact , this must be set so that the total cost constraint compute (6) is met. Fig. 1 outlines a simple search procedure to find the that satisfies the constraint, hence maximaximum value of mizing the total sequence quality. This algorithm computes the loss impact at the margin, the corresponding packet index can be easily computed by inverting the loss function . Following from (9) and the fact that the packet loss impacts , there is the are sorted in ascending order (so following additional constraint on prices given the loss rates experienced
Fig. 2. Distribution of video packets among the four Diffserv classes versus the price constraint, for the “Foreman” sequence with new packet loss impact measurement. The results are obtained through three different methods: an exhaustive search (squares), the QoS mapping algorithm of the present paper (stars), and the categorization method (triangles) of [6]. The lines are for guidance only.
(10)
IV. SIMULATION RESULTS To evaluate the performance of the new packet loss impact estimation and QoS mapping algorithms a number of experiments were conducted using a standard MPEG-4 encoder and a decoder that had been modified to implement appropriate error concealment. Two standard sequences (CIF, 10 fps, 10 s duration) were tested: “Foreman” at 320 kbps and “Mother & Daughter” at 160 kbps with a video packet size of 500 bytes. These packetized bitstreams were then mapped to four Diffserv classes using the algorithm described in Section III and the technique described in [6]. As in [6], we use linearly increasing ( without loss of generality) prices . The number and loss rates of the form of available Diffserv classes is network-specific and may not be four as used here, these issues are outside the scope of this paper. The first feature of note is that the new iterative algorithm can provide very accurate solutions to the optimization problem associated with the QoS mapping. Figs. 2 and 3 show that the optimal solutions obtained through the new algorithm are virtually the same as those obtained through exhaustive search of all possible values for . In contrast, the categorization QoS mapping algorithm [6] provides coarse-grained solutions, which deviate significantly from the true solutions. These figures also show that as the price per packet constraint increases more packets are mapped into the highest Diffserv class. Since the new QoS mapping algorithm is very accurate, it can minimize the loss impact more effectively than the algorithm proposed in [6]. The normalized minimal loss impact (achieved by dividing the actual packet loss by the packet loss
Fig. 3. Same as in Fig. 2 but for “Foreman” sequence with “three factor” loss impact measurement [6].
when all packets are transmitted on the highest loss Diffserv class) achieved by the new QoS mapping algorithm decreases smoothly as the per packet price constraint increases. This is in contrast to Shin’s categorization optimization algorithm. Results are shown in Fig. 4(a) and 4(b) and Fig. 5(a) and 5(b). For a given loss impact measurement, the objective quality obtained with the new QoS mapping algorithm is better than with the Shin’s method, as shown in Fig. 4(c) and 4(d) and Fig. 5(c) and 5(d), where the PSNR values are statistical averages over 100 runs of different packet loss patterns. In Fig. 6, we plot the average PSNR value versus the frame number. For the “Foreman” sequence, the new QoS mapping algorithm is slightly better than with the categorization QoS mapping algorithm for both the new loss impact measurement and the three-factor method [6]. This is consistent with the fact that
1008
Fig. 4. Normalized minimal loss impact and the average PSNR achieved with the new QoS mapping algorithm (stars) and by the categorization algorithm (triangles) for “Foreman” sequence. (a) and (c) new loss impact measurement and (b) and (d) the “three-factor” measurement.
Fig. 5. Same as in Fig. 4, but for the “Mother & Daughter” sequence.
at the price constraint the all-frame averaged PSNR values, according to the data presented in Fig. 4(c) and 4(d) differ by 0.06 dB and 0.17 dB in Fig. 6(a) and 6(b) respectively. However, for the “Mother & Daughter” sequence, the new QoS mapping algorithm yields noticeably better results, and in this case the all-frame averaged PSNR values differ by 0.73 dB in Fig. 6(c) and 0.28 dB in Fig. 6(d). We observe that the new loss impact measurement performs very similarly to the three-factor method. Both of them can provide good protection to I-frames. (The peaks in Fig. 6 occur at the I-frames.) The reason is that both loss impact measurements usually yield higher loss impact for the I-frames than for the P-frames, resulting in the I-frames getting higher chances to be mapped into high Diffserv classes that suffer low loss rates. We also studied another Diffserv network model with three aggregated traffic classes, where the loss rate in the low,
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 8, NO. 5, OCTOBER 2006
Fig. 6. PSNR versus frame number. (a) “Foreman” sequence using the new loss impact measurement, (b) “Foreman” sequence using the “three-factor” loss impact measurement, (c) “Mother & Daughter” with new loss impact, and (d) “Mother & Daughter” with “three factor” loss impact. The solid lines are for new QoS mapping algorithm, the dashed lines are for the categorization algorithm of [6]. We also plot the PSNR of the nonerrored bit stream (dotted lines) for reference.
medium, and high priority classes is 10%, 2.5%, and 0%, respectively. We found that the simulation results were qualitatively the same as described above. In particular, the new optimal QoS mapping algorithm can minimize the total loss impact more effectively, and produce better objective video quality than the categorization optimal QoS mapping algorithm. Results for this case have been reported in [12]. A series of subjective tests was also undertaken to assess the performance of the new technique. These were based on a modified form of the Double Stimulus Continuous Quality Scale (DSCQS) method [15]. As the sequences to be measured are significantly degraded, it is not appropriate to use the original sequence (i.e., no compression or data loss) as a reference as is used in the standard form of this test. Instead, we use the new QoS mapping algorithm as the reference technique that is paired with the categorization QoS mapping and the two methods are then compared directly. Viewers were asked to score the relative subjective quality of each pair of sequences on a continuous scale divided into five divisions marked excellent, good, fair, poor and bad as a guide. The difference between the midpoint of adjoining division therefore corresponds to a 25% difference in the result. Three sequences were used for the tests, “Mother & Daughter” and “Foreman” as previously described and additionally the “Stefan” sequence coded at 640 kbps. Since we are dealing with random packet loss, there is no single sequence that uniquely corresponds to a given set of network conditions. For each test 20 different packet loss patterns were generated and the pattern whose corresponding decoded sequence has the closest PSNR value to the mean for the set of 20 was then used. Fig. 7 shows the results of the subjective tests that were conducted with 16 nonexpert viewers. The error bars show the 95% confidence levels for each test. The figure shows that
ZHANG et al.: EFFICIENT STREAMING PACKET VIDEO
1009
Fig. 7. Subjective test results. The vertical scale shows the difference between the subjective quality measured for the categorization QoS mapping algorithm and the new QoS mapping which is the reference condition. Positive values on this scale denote worse subjective quality than the reference Beneath each error bar is the average price per packet and the objective quality (PSNR) difference between the two techniques.
the new QoS mapping algorithm consistently outperforms the categorization method both in the subjective quality (by about half a division on the DSCQS scale), as well as in objective quality. V. CONCLUSION We have addressed two important issues in streaming video over differentiated services networks. First, the loss impact of a video packet has been analyzed and a simple method to measure the video quality degradation due to packet loss has been proposed. We have shown that packet prioritization based on the proposed loss impact measure achieves comparable performance to the method in [6] with lower computational complexity. We have also studied the optimal QoS mapping from a video packet sequence to a set of Diffserv classes. The results shown indicate that performance of the new algorithm is nearly identical to an exhaustive search (computationally infeasible for real applications) for the best allocation between classes. This is backed up by subjective test results that show the difference between the new algorithm and the method in [6] can be detected by nonexpert viewers. REFERENCES [1] R. Braden, D. Clark, and S. Shenker, “,” Integrated Services in the Internet Architecture: An Overview Jun. 1994, RFC 1633. [2] S. Blake, D. Black, M. Carlson, E. Davies, Z. Wang, and W. Weiss, An Architecture for Differentiated Services Dec. 1998, RFC 2475. [3] C. Dovrolis and P. Ramanathan, “A case for relative differentiated services and proportional differentiation model,” IEEE Network, vol. 13, pp. 2–10, Sep./Oct. 1999. [4] J. Heinanen, F. Baker, J. Weiss, and W. Wroclawski, Assured Forwarding PHB Group Jun. 1999, RFC 2597. [5] Y. T. How, D. Wu, B. Li, T. Hamada, I. Ahmad, and H. J. Chao, “A differentiated services architecture for multimedia streaming in the next generation internet,” Comput. Netw., no. 32, pp. 185–209, 2000.
[6] J. Shin, J. G. Kim, and C.-C. J. Kuo, “Quality-of-service mapping mechanism for packet video in differentiated services network,” IEEE Trans. Multimedia, vol. 3, no. 2, pp. 219–231, 2001. [7] J. G. Kim, J. Kim, J. Shin, and C.-C. J. Kuo, “Coordinated packet level protection employing corruption model for robust video transmission,” in Proc. VCIP, San Jose, CA, Jan. 2001, pp. 410–421. [8] J. Shin, J. G. Kim, J. W. Kim, and C.-C. J. Kuo, “Dynamic QoS mapping for streaming video in relative service differentiation networks,” Eur. Trans. Telecommun., vol. 12, no. 3, pp. 217–230, May/Jun. 2001. [9] H.-R. Shao, W. Zhu, and Y.-Q. Zhang, “User-aware object-based video transmission over the next generation internet,” Signal Process.: Image Commun., vol. 16, pp. 763–784, 2001. [10] T. Ahmed, G. Buridant, and A. Mehaoua, “Encapsulation and marking of MPEG-4 video over IP differentiated services,” in Proc. IEEE Symp. Computers and Communications, Tunisia, 2001, pp. 346–352. [11] H. Yu, D. Makrakis, and O. Barbosa, “Experimental evaluation of MPEG-2 video over differentiated services IP networks,” in Proc. IEEE Pacific Rim Conf. Communications, Computers and Signal Processing, 2001, pp. 369–472. [12] F. Zhang, M. R. Pickering, M. R. Frater, and J. F. Arnold, “Streaming MPEG-4 video over differentiated services networks,” in Proc. 1st Workshop on the Internet, Telecommunications and Signal Processing, Wollongong, Australia, Dec. 2002, pp. 100–105. [13] “Optimal quality-of-service mapping for streaming video over differentiated services networks,” in Proc. ICASSP, Hong Kong, Apr. 2003, vol. 5, pp. 744–747. [14] A. R. Reibman, V. A. Vaishampayan, and Y. Sermadevi, “Quality monitoring of video over a packet network,” IEEE Trans. Multimedia, vol. 6, no. 2, pp. 327–334, Apr. 2004. [15] Methodology for the Subjective Assessment of the Quality of Television Pictures 2000, ITU Rec. ITU-R BT.500-10. Fei Zhang was born in China in 1964. He received his B.Sc. degree in mathematics from the University of Science and Technology of China in 1985 and the Ph.D. degree in theoretical and mathematical physics from the University of Madrid, Spain, in 1992. During 1993-2002, he was a Research Fellow and Senior Research Fellow with the Australian National University, The National University of Singapore, and the University College Australian Defence Force Academy, The University of New South Wales, Canberra, Australia, and he published over 40 papers in the broad field of computational science. After working as a Software Engineer with CSIRO, Australia, for two-and-a-half years, he joined Geoscience Australia in July 2005 and is currently a System Design and Development Analyst for remote sensing applications.
1010
James Macnicol (M’03) received the B.E. degree in systems engineering from the Australian National University, Canberra, in 1998 and the Ph.D. degree from University College, University of New South Wales, Sydney, Australia, in 2003. Since 2002, he was a Research Associate with the School of Information Technology and Electrical Engineering, University College Australian Defence Force Academy, The University of New South Wales, Canberra, Australia. His research interests include video coding and associated issues relating to transmission of video over unreliable channels. He passed away in 2004.
Mark R. Pickering (M’96) received the B.E. degree from Capricornia Institute of Advanced Education, Rockhampton, Australia, in 1988, and the M. E. and Ph. D. degrees in electrical engineering from the University of New South Wales, Sydney, Australia, in 1991 and 1995, respectively. He is currently a Senior Lecturer in the School of Information Technology and Electrical Engineering, Australian Defence Force Academy, Canberra, Australia. His research interests include video and audio coding, data compression, information security, data networks and error-resilient data transmission.
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 8, NO. 5, OCTOBER 2006
Michael R. Frater (S’89–M’91) received the B.Sc. and B.E. degrees from the University of Sydney, Sydney, Australia, in 1986 and 1988 respectively, the Ph.D. degree from the Australian National University, Canberra, Australia, in 1991, and the M.H.Ed. degree from the University of New South Wales, Sydney, in 1996. Since 1991, he has been with the University of New South Wales at the Australian Defence Force academy, Canberra, where he is currently an Associate Professor. His research interests lie in the fields of audio-visual and multimedia communications and communication systems. Prof. Frater has served as an Associate Editor of the IEEE TRANSACTIONS ON IMAGE PROCESSING.
John F. Arnold (S’77–M’85–SM’96) received the B.E. and M.Eng.Sc. degrees from the University of Melbourne, Australia, in 1976 and 1979, respectively, and the Ph.D. degree from the University of New South Wales (UNSW), Sydney, Australia, in 1984. Since 1978, he has been with the School of Electrical Engineering, UNSW, initially at the Royal Military College, Duntroon, and more recently at the Australian Defence Force Academy, Canberra. He is currently a Professor of electrical engineering and Head of the School of Information Technology & Electrical Engineering. His research interests lie in the fields of video coding, error resilience of compressed digital video, and coding of remotely sensed data. He has published widely in these areas. Prof. Arnold is an Associate Editor of the IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY.
