a Evaluation Framework for More Realistic ... - Semantic Scholar

International Journal of Software Engineering and Its Applications Vol. 2, No. 2, April, 2008

MyEvalvid_RTP: a Evaluation Framework for More Realistic Simulations of Multimedia Transmission Chia-Yu Yu1, Chih-Heng Ke2, Reuy-Shin Chen2, Ce-Kuen Shieh1, Naveen Chilamkurti3 1 Institute of Computer and Communication Engineering, Department of Electrical Engineering National Cheng Kung University, Taiwan 2 Dept. of CSIE, National Kinmen Institute of Technology, Taiwan 3 Dept. of Computer Science and Computer Engineering La Trobe University, Melbourne, Australia, 3086 [email protected]; [email protected]; [email protected]; [email protected]; [email protected] Abstract Recently, multimedia is a more and more important Internet service. For multimedia, Quality of service (QoS) support is a crucial requirement. To meet these QoS requirements, researchers develop specific multimedia mechanism to enhance the performance of video transmission. When they evaluate the performance of their mechanism, most researchers use simulation tools to evaluate. However, when using these simulation tools, researchers usually acquire network-level performance metrics, such as throughput. They cannot evaluate video and audio delivered quality by comparing the original and distorted video. However, using network-level performance metrics can not evaluate the delivered quality correctly. To address this issue, some researchers proposed simulation tool-sets. These simulation tool-sets can evaluate the video delivered quality well. However, they cannot evaluate the audio delivered quality. Therefore, we propose a new simulation tool-set called as MyEvalvid_RTP to achieve more realistic simulations in this paper. By MyEvalvid_RTP, researchers can evaluate both the video delivered quality and the audio delivered quality.

1. Introduction Use of internet related applications has been growing rapidly in recent years. One of these applications is multimedia streaming which includes image and audio. For multimedia streaming, performance metrics such as delay, jitter and Peak Signal-toNoise Ratio (PSNR) [1] are important factors for any evaluation study. This means that multimedia streaming must maintain the specific QoS for user satisfaction. Therefore, many researchers enhance the performance of multimedia streaming by their proposed mechanisms for QoS. Furthermore, researchers often use simulation tool-sets, such as NS-2 [2], Qualnet [3] and OPNET [4], to evaluate their proposed mechanisms. Current simulation tool-sets for multimedia streaming use network-level metrics, such as throughput, delay and jitter, to evaluate the performance. However, in some cases these metrics are not able to represent the quality of multimedia precisely. Therefore, to

21


develop a better simulation tool-set for the quality evaluation of video and audio in multimedia networks is very important. We start with a brief introduction of multimedia streaming simulation. The multimedia streaming simulation is divided into three types: video traffic traces, real bit stream and video traffic model. a. Video-traffic traces The video-traffic traces represent certain characteristics of the real-video stream. In general, a trace file includes the frame number, frame type, and frame size to describe the characteristics of real-video traffic. The advantage of using traffic traces is that one doesn’t need to worry about copyright issues because it doesn’t contain the actual video information. However, the trace files do not contain the full information of the real video, so it is difficult to compare the quality with the original video. Therefore, for a simulation study using video traffic traces, only network-level metrics such as delay, throughput and jitter, can be obtained. However in the case of evaluating video transmission, network-level metrics may not be adequate to judge the quality received by an end user. Furthermore, it is hard to extensively study the effects of proposed network mechanisms on different characteristics of the same video because the encoding settings for the publicly available video-traffic traces are limited. Some sample video traffic traces which are in different formats such as H.264, MPEG, or MDC traces, can be found in [5]. b. Real bit stream The real bit-streams method uses the actual output of video encoding for video transmission evaluation. The advantage of using the real bit-streams is that we can get not only the network-level metrics such as bandwidth usage, frame rate and delay but also the application-level metrics such as PSNR and Mean Opinion Score (MOS) [6]. In addition, we can also visually rate the video delivered quality. However, the disadvantage of using the real bit-streams is that there are concerns about copyright issues. c. Video-traffic model A video model captures the characteristics of real-video bit streams using analytical methods. These methods are typically developed based on the statistical properties of a set of video-trace samples of real-video traffic. The advantage of using video-traffic model is that this can analyze the results more rapidly than other performance evaluation methods. In addition, the developed model can be used for mathematical analysis of networks. However, it lacks the possibility of visualizing a transmitted video. In the above three methods, when using real video to evaluate network performance, one must have basic knowledge of real network environment. This is because researchers must transmit real video in the real network and evaluate the performance. Therefore, when we want to evaluate the performance of a new protocol or architecture, we must wait until any related products become available. To address this issue, a series of simulation tools for multimedia streaming, such as NS-2 simulator, are developed. They can evaluate the network performance of a new protocol or architecture easily. Several papers are based on these toolsets [7] [8] [9]. However, these previous simulation tools can only evaluate the video delivered quality; they were unable to evaluate the audio delivered quality. In this paper, we propose a new simulation tool-set called as MyEvalvid_RTP. MyEvalvid_RTP provides a better simulation results for video and audio transmission. When using MyEvalvid_RTP, the researchers can evaluate not only the video delivered quality but also the audio delivered quality.

22


The remainder of this paper is organized as follows. Section 2 will describe the system architecture of MyEvalvid_RTP. In Section 3, we prove our new tool-set using some simulation experiments. Finally, we briefly summarize this paper.

2. The system architecture of MyEvalvid_RTP As previous simulation tool-set can evaluate only the video delivered quality, we developed new simulation tool-set which is able to evaluate both the video delivered quality and the audio delivered quality. In this tool-set, we combined Evalvid [10] with NS-2, VLC [11] and RTPtools [12] in the new simulation tool-set called as MyEvalvid_RTP for video and audio transmission. By MyEvalvid_RTP, researchers will be able to evaluate the video delivered quality and audio delivered quality for their proposed mechanisms. Figure 1 presents the system architecture of MyEvalvid_RTP.

Figure 1. The system architecture of MyEvalvid_RTP In MyEvalvid_RTP, we use VLC player which is publicly available and can be used as a media player and as a streaming server. It can be configured as a server to stream various audio and video formats (MPEG-1, MPEG-2, MPEG-4, DivX, and so on) in unicast or as a multicast traffic. It can also be a multimedia player for various audio and video formats. In addition, RTPdump can parse RTP packets and generate output files. RTPplay can play back RTP sessions obtained by RTPdump. Because of the characteristics of VLC, RTPdump and RTPplay, we combined Evalvid with these tools and NS-2 to obtain MyEvalvid_RTP. When using MyEvalvid_RTP, the researchers can use MPEG-format video file which contains image and audio to evaluate the proposed networking mechanism. We describe the simulation process shortly in the following. When evaluating the performance of video transmission by MyEvalvid_RTP, we first stream MPEG video file which contains image and audio using VLC and extract the information of the video file to obtain the traffic trace file by RTPdump and other programs. The format of traffic trace file is shown in the table 1. In the traffic trace file, the first column

23


is sending time. It means the time that packet is sent out to the receiver. The second column is the packet payload size. It means the packet size which does not contain packet header. Table 1. The traffic trace file format RTP Packet Payload Inter sending Time Size 0.000000 1328 0.042937 1328 0.083010 1328 0.123996 1328 0.166965 1328 0.201452 1328 0.201971 1328 0.202627 1328 After obtaining the traffic trace file, we evaluate the performance of the proposed mechanism by using the traffic trace file as an input to NS-2 environment. After finishing simulation, two trace files are generated. One is sender trace file and the other is receiver trace file. Table 2 and Table 3 shows the sender trace file and the receiver trace file respectively. Table 2. The sender trace file format Packet sending Packet id Packet size time sequence 0.000000 id 0 udp 1356 0.042937 id 1 udp 1356 0.083010 id 2 udp 1356 0.123996 id 3 udp 1356 0.166965 id 4 udp 1356 0.201452 id 5 udp 1356 0.201971 id 6 udp 1356 0.202627 id 7 udp 1356 Table 3. The receiver trace file format Packet receiving Packet id Packet size time sequence 0.022769 id 0 udp 1356 0.065706 id 1 udp 1356 0.105939 id 2 udp 1356 0.146625 id 3 udp 1356 0.189754 id 4 udp 1356 0.224601 id 5 udp 1356 0.236115 id 6 udp 1356 0.247949 id 7 udp 1356 In the sender trace file, the first column is sending time. Therefore, it is the same as the first column in traffic trace file. The second column is the id sequence. It means the sequence

24


number of every packet. And the last column is packet size. It means the packet size which contains packet header. Therefore, it is larger than the second column in the traffic trace file. In the receiver trace file, the only difference with the sender trace file is the first column. The first column in receiver trace file is the receiving time. It means the time that packet is received in the receiver. When obtaining these trace files, we use Evaluate Trace (ET) program to analyze sender trace file and receiver trace file to obtain network-level performance metrics (throughput and delay and so on). In addition, one distorted MPEG video file is generated through RTPplay and VLC tools. Finally, we can evaluate the end-to-end video and audio delivered quality by comparing the original and distorted MPEG video file. A detailed reference to the simulation process could be found on our website [13].

3. Performance evaluation In this section, in order to prove the correctness of MyEvalvid_RTP, we evaluate the performance of video transmission over two different network environments by MyEvalvid_RTP. The first environment is wired network, and the other is wireless network. We present the simulation results and discuss them separately in the following subsections.

Figure 2. Parking lot topology 3.1. Wired network scenario

In the subsection, we show the simulation results acquired by MyEvalvid_RTP in wired network. In this scenario, we use parking lot topology [14] as wired network scenario which is presented in Figure 2. In this scenario, each link is bidirectional between router and router or router and node. The bandwidth of each link between node and router is 10 Mb and the delay is 1 ms. In addition, the bandwidth setting for each link between routers is different to achieve bottleneck. The bandwidth settings are shown in table 4. In this scenario, there is one video traffic flow between Video Server and Video Receiver. In the following simulation results, the performance of this video traffic will be evaluated. In addition, five nodes will transmit CBR traffic to CBR sinks. We consider CBR traffic as background traffic. In the following results, we will observe the effects when we increase the number of CBR traffic for each CBR source node. The source and destination node of each CBR traffic flow is shown in table 5.

25


Table 4. The bandwidth settings between routers Link Bandwidth (Mbps) Router A—Router B 1 Router B—Router C 2 Router C—Router D 5 Router D—Router E 8 Table 5. The source node and destination node of each CBR traffic flow CBR traffic Source Destination node flow node CBR1 Video Video Receiver Server CBR2 S_node2 D_node2 CBR3 S_node3 D_node3 CBR4 S_node4 D_node4 CBR5 S_node5 D_node5

Firstly, we will evaluate the performance of video traffic when every source node has 2, 4, 6, 8, 10 CBR traffic flows. Here, we present the throughput and delay evaluated by MyEvalvid_RTP for wired network. We present throughput for different number of CBR traffic flows for every source node in figure 3. From figure 3, we can observe that the throughput for the observed video streaming is higher when there are fewer CBR traffic flows for every source node. This is because when transmitting video traffic between video server and video receiver, traffic must go through each link between routers. However, these links between routers are shared by video traffic and other CBR background traffic. When there are more CBR traffic flows for every source node, video traffic will contend the transmission medium with more CBR background traffic flows. Therefore, more CBR traffic flows will lower the throughput of the video traffic. 100

2 CBR flows 4 CBR flows 6 CBR flows 8 CBR flows 10 CBR flows

90

Throughput (Kbps)

80 70 60 50 40 30 20 10 0 0

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17

Time (Second)

26


Figure 3. Throughput comparison of parking lot scenario

Delay (Seconds)

Next, we show the delay for different number of CBR traffic flows for every source node in figure 4. From figure 4, we can observe slightly different delay obviously when there is different number of CBR traffic flows in every source node. When more CBR traffic flows are in every source node, the delay will be slightly higher than when there are fewer CBR traffic flows. This is still because video traffic flow must contend with other CBR background traffic flows. Through MyEvalvid_RTP, researchers can easily obtain network-level performance metrics, such as throughput and delay. In addition, researchers also can compare the video and audio delivered quality between original MPEG video and distorted MPEG video. A frame of MPEG video with and without distortion is showed in figure 5. These sample MPEG video and distorted MPEG video can be downloaded from our website [13]. From figure 5, we can observe the following results. When there are more CBR traffic flows in every source nodes, the video delivered quality will be deteriorated more seriously. Here, the audio delivered quality within distorted MPEG file can not be described by text. Therefore, if researchers are interested in evaluating the audio delivered quality for distorted MPEG file which is acquired above, you can download MPEG files from our website and evaluate its performance using this scenario.

2.0 1.9 1.8 1.7 1.6 1.5 1.4 1.3 1.2 1.1 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0

2 CBR flows 4 CBR flows 6 CBR flows 8 CBR flows 10 CBR flows

0

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17

Time (Second)

Figure 4. Delay comparison of parking lot scenario

27


(a)

(b)

(c) (d) Figure 5. The visual quality of parking lot scenario (a) original MPEG video (b) distorted MPEG video for 4 CBR traffic flows (c) distorted MPEG video for 6 CBR traffic flows (d) distorted MPEG video for 8 CBR traffic flows 3.2. Wireless network scenario In this scenario we use wireless network as our simulation setup. Here, we use Ad Hoc network as wireless network scenario. Figure 6 shows the Ad Hoc network scenario. We use ten mobile nodes for our wireless setup. They are distributed at an area of 300×300 meters randomly. Some network settings are shown in table 6. In this scenario, the video traffic is transmitted from n9 to n0. In addition, there are background CBR traffic between every pair of nodes, from the ith node to the (i+1)th node. Here, we will observe effects with varying number of mobile nodes.

Table 6. The network settings for Ad Hoc network scenario Network Settings Node number Routing Protocol Bandwidth between every pair of node Topology area Position of nodes

28

Value 6, 8, 10, 12, 14 DSDV 1Mbps

300×300 Distributed randomly


Figure 6. Ad Hoc network scenario (The number in brackets is the coordinate of node) Similar to previous scenario, we also present the throughput and delay evaluated by using MyEvalvid_RTP for wireless network. First of all, we present the throughput for different number of nodes in Figure 7. 75 70

6 nodes 8 nodes 10 nodes 12 nodes 14 nodes

65

Throughput (Kbps)

60 55 50 45 40 35 30 25 20 15 10 5 0 0

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20

Time (Second)

Figure 7. Throughput comparison of Ad Hoc network From figure 7, we can observe that the throughput for this network is higher when the number of nodes is fewer. This is because these nodes of Ad Hoc network share the transmission medium. Nodes can not transmit any traffic when other nodes are transmitting their network traffic. Therefore, as more nodes exist in the network, more CBR background traffics exist in our network scenario. This will result in the case that the probability that other nodes are transmitting CBR background traffic is higher when video server is transmitting video traffic to video receiver. The video server must wait until the transmission medium is idle. This will lower the throughput of video traffic. Figure 8 shows delays for different number of nodes in Ad Hoc network.

29


6.0

6 nodes 8 nodes 10 nodes 12 nodes 14 nodes

5.5 5.0

Delay (Seconds)

4.5 4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 1

2

3

4

5

6

7

8

9

10

11

12

13

14

Time (Second)

Figure 8. Delay comparison of Ad Hoc network From figure 8, we can observe the different delays when different numbers of nodes are in the Ad Hoc network. It is similar to throughput above. When more nodes are in the network, the delay will be higher than when there are fewer nodes. From figure 9, we can observe the following results. When there are more nodes in Ad Hoc network, the video delivered quality will be deteriorated more seriously. Audio delivered quality can not still be described by text. It is the same as the wired scenario. The researchers who are interested in audio delivered quality within MPEG files can download MPEG files from our website to evaluate the performance. From these two simulations, we can know that MyEvalvid_RTP can evaluate the performance of multimedia streaming correctly. This will help the researchers which evaluate the performance of proposed networking mechanisms about multimedia streaming. Therefore, MyEvalvid_RTP is a useful simulation tool-set.

(a)

(c)

(b)

(d)

Figure 9. The visual quality of Ad Hoc network (a) Original MPEG video (b) distorted MPEG video for 6 nodes (c) distorted MPEG video for 8 nodes (d) distorted MPEG video for 10 nodes

30


4. Conclusion In this paper, we combine Evalvid with NS-2, VLC and RTPtools to obtain a new simulation tool-set MyEvalvid_RTP. By using MyEvalvid_RTP to evaluate the performance of video transmission, the researchers can get the network-level performance metrics, such as throughput and delay, easily. In addition, the researchers can also obtain both the video delivered quality and the audio delivered quality at the same time. The researchers can see the video quality and hear the audio quality by VLC or other MPEG-compatible player. Therefore, we can compare the video and audio quality difference of the original MPEG video and distorted MPEG video easily. The characteristic of MyEvalvid_RTP can help researchers be able to do more realistic simulations and get more realistic results. In brief, our proposed QoS assessment framework would be a good choice for researchers who want to verify their designs in multimedia communication, such as network protocols or QoS control schemes. [References] [1] Chih-Heng Ke, Cheng-Han Lin, Ce-Kuen Shieh, Wen-Shyang Hwang, “A Novel Realistic Simulation Tool for Video Transmission over Wireless Network”, The IEEE International Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing (SUTC2006), June 5-7, 2006, Taichung, Taiwan, IEEE Computer Society Press [2] Network Simulator ns-2, http://www.isi.edu/nsnam/ns/. [3] Qualnet, http://www.scalable-networks.com/ [4] OPNET, http://www.opnet.com.tw/eng/ [5] Video traffic trace files, http://trace.eas.asu.edu/ [6] T. BETCHAKU, N. SATO, H. MURAKAMI, “Subjective Evaluation Methods of Facsimile Image Quality”, in Proc. IEEE Int. Conf. Communications (ICC’93), vol. 2, Geneva, Switzerland, May 23-26, 1993, pp. 966-970. [7] C. H. Ke, C. K. Shieh, W. S. Hwang, A. Ziviani, "A Two Markers System for Improved MPEG Video Delivery in a DiffServ Network", IEEE Communications Letters, IEEE Press, ISSN: 1089-7798, vol. 9, no. 4, April 2005, pp. 381-383. [8] J. Naoum-Sawaya, B. Ghaddar, S. Khawam, H. Safa, H. Artail, Z. Dawy, "Adaptive Approach for QoS Support in IEEE 802.11e Wireless LAN," IEEE International Conference on Wireless and Mobile Computing , Networking and Communications (WiMob 2005), Montreal, Canada, August 2005 [9] A. Lo, G. Heijenk, I. Niemegeers, "Performance Evaluation of MPEG-4 Video Streaming over UMTS Networks using an Integrated Tool Environment", International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS 2005), Philadelphia, PA, USA, July 24-28, 2005. [10] J. Klaue, B. Rathke, and A. Wolisz, "EvalVid – A Framework for Video Transmission and Quality Evaluation", In Proc. of the 13th International Conference on Modelling Techniques and Tools for Computer Performance Evaluation, Urbana, Illinois, USA, September 2003. [11] VLC, http://www.videolan.org/vlc/ [12] RTPtools, http://www.cs.columbia.edu/IRT/software/rtptools/ [13] MyEvalvid_RTP, http://140.116.72.80/~yufrank/YCY/myevalvid_rtp.htm [14] T. Bonald, A. Prouti`ere, J. Roberts, J. Virtamo. “Computational aspects of balanced fairness”. In Proceedings of the 18th International Teletraffic Congress (ITC-18), September 2003, pp. 801–810.

31


AUTHORS Chia-Yu Yu is currently a Ph.D. student in the Institute of Computer and Communication Engineering, Department of Electrical Engineering, National Cheng Kung University. He received his MS, and BS degrees from the Computer Science and Information Engineering Department of National Cheng Kung University, Tainan, Taiwan. His current research areas include multimedia communications and wireless networks.

ChihHeng Ke received his B.S. and Ph.D degrees in Electrical Engineering from National Cheng-Kung University, in 1999 and 2007. He is an assistant professor of Computer Science and Information Engineering, National Kinmen Institute of Technology, Kinmen, Taiwan. His current research interests include multimedia communications, wireless network, and QoS network.

Ruey-Shin Chen is currently an assistant professor in Computer Science and Information Engineering, National Kinmen Institute of Technology, Kinmen, Taiwan. His current research focus includes electronic commerce, computer network management, and statistics and analysis.

Ce-Kuen Shieh is currently a professor in the Department of Electrical Engineering, National Cheng Kung University. He received his Ph.D., MS, and BS degrees from the Electrical Engineering Department of National Cheng Kung University, Tainan, Taiwan. His current research areas include distributed and parallel processing systems, computer networking, and operating systems.

Naveen Chilamkurti is currently a lecturer teaching in the Department of Computer Science and Computer Engineering, La Trobe University, Australia. He is a Senior Member of IEEE and has published about 40 conference papers and two journal papers. He has a PhD from La Trobe University. His current research areas include wireless Multimedia, Multicast Congestion Control, Multicast Security, Reservation Protocols, TCP/IP Congestion Control, HighSpeed Networks, IPv6 Security and Authentication.

32