Extracting Client-Side Streaming QoS Information from ... - CiteSeerX

Extracting Client-Side Streaming QoS Information from Server Logs Naomi Terada

Eiji Kawai

Hideki Sunahara

Graduate School of Information Science Nara Institute of Science and Technology 8916-5 Takayama, Ikoma, Nara 630-0192, Japan

Abstract

erators rely on their intuition without quantitative foundation to estimate the client-side service quality. The other problem is that some log entries which are generally thought to indicate the service status are misleading. In a Windows Media Server log file, there is an entry named c-quality (“client quality”). This parameter indicates the ratio of the packets received by a client to those sent by a server, expressed in percentages. If the value of this parameter is large enough, the service quality is believed to be high. However, we observed in our preliminary experiments that a Windows Media Server recorded the client quality at “100%” even when some dropped frames occurred on a client. Generally, such a high rating of the client quality gives a wrong impression that the end-user received an entirely satisfactory service. Another approach to obtain precise information about the service quality is packet monitoring. By packet monitoring, we can obtain highly accurate information about the communication between a server and a client. However, for streaming service, UDP is usually used as a data transmission protocol. In contrast to TCP, monitoring UDP packets at the point near the server host does not help the service quality management very much because there is no feedback information from the client (no ACK information is exchanged in UDP communications). That means packet monitoring on the server-side is not effective to know the service status on the client. Additionally, we can also monitor the service control session, e.g., RTSP, between the client and the server, in which streaming service state information is exchanged. However, the format and the precise semantics of the control data are often proprietary and unknown to the public. To make matters worse, the control session exchanges only the application status information. As we mentioned earlier, the application status does not directly express the service quality

Effective management of the quality of multimedia streaming service is a challenging issue. Traditionally, many server operators rely on their intuition for the service quality management because the quantitative semantics of the server log data has not been explored. In this paper, we propose a new approach to quantitative estimation of the client-side service quality based on the server log data. The experimental results show fundamental guidelines for effective service quality management.

1

Introduction

Quality management of network application service is one of the chief concerns of service providers. Particularly in the case of streaming service, effective service quality management is a key factor for the success of service commercialization, e.g., pay-per-view streaming service. As the first step to successful service quality management, it is highly important for service providers to know the service state precisely. For that purpose, they often analyze server log files. However, in streaming service, there are following two major problems in that approach. One is lack of information that directly expresses the actual service quality experienced by the end users. A server log file records several service quality parameters including both server-side and client-side software program statistics. However, the correlation between those parameter values and the actual service quality perceived by the end users is unknown to the server operators. Therefore, many server opThis paper is originally published in the proceedings of IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PacRim 2005), Victoria, B.C., Canada, August 24-26, 2005.

1

2.2

experienced by the end-users. The data buffer implemented in a multimedia client software is another factor that makes the situation more complicated. A client software keeps a certain amount of the received data in the data buffer to avoid unexpected pauses during the playback. With this mechanism, network packet loss can be recovered by retransmission under a certain real-time restriction, and therefore it does not immediately mean service quality degradation. In this paper, we propose a new approach to quantitative estimation of the client-side service quality based on the server log data. By evaluating the client-side service quality in various network environments such as small/large network delays, low/high packet loss ratios, etc., we extract the quantitative and generalized guidelines for the quality management of streaming service. Applying our approach, service providers acquire a method to evaluate the service satisfaction level of each end user.

2 2.1

Service State on a Client

Apparently, direct observation of client software behavior is the strongest method for assessing the quality of service. The parameters we can directly observe in a Windows Media Player through our eyes include the frame drops and the playback pauses, which have a close correlation with the service quality experienced by the end-users. In our preliminary experiments, we found that when we perceived frame drops visually, the playback frame rate, which we can obtain from the player statistics in real-time during the playback, declined. In this study, we therefore monitor the player statistics every second by a special external program and detect the declines of the playback frame rate. Thus, we define the frame rate decline time as the total seconds when the decline of the frame rate is detected. The frame rate decline time can be obtained only by observing the player statistics and not by a server log data analysis. Herein, there are two points we have to note about the playback frame rate. One is the unstableness of the playback frame rate for initial several seconds of the playback. Typically, it shows a small spike and gradually decreases to the normal playback frame rate. These seconds are counted in the frame rate decline time by our definition. The other point is the small jitters of the playback frame rate. Although these jitters do not degrade the quality of the streaming service, some of them may be counted in the frame rate decline time. In our experiments, we observed about eight seconds of the frame rate decline time when no transmission delay and no packet loss were configured. Also, the playback pauses are explicit evidence for the degradation of the service quality. A playback pause can be detected by the occurrence of a rebuffering. The buffering frequency and the total buffering time are reported from a client to a server, and recorded in the server log file. In addition to those parameters, some other status parameters on the client are reported to the server via a control session with RTSP, which includes client received packets, client recovered packets, and client lost packets as we mentioned in the previous subsection.

State of Streaming Service Service State Information in a Server Log

Traditionally, the quality management of network service has been based on server log data analyses. A server log contains various information about client service statistics. For example, a Windows Media Server records a wide variety of service performance indices in a log file [1]. Among them, the number of received, recovered, and lost packets (c-pkts-received, c-pkts-recovered-resent, and cpkts-lost-client), the buffering time and frequency (ctotalbuffertime and c-buffercount) are essential metrics. A data packet is recorded as received when it was successfully received by the client without retransmission. A recovered data packet means that it was received by the client after retransmission. A lost packet is one that never reached the client by the playback time. The buffering time indicates the total amount of time that a client required in bufferings. The buffering frequency is the number of bufferings in a service stream. When a rebuffering occurs, the playback pauses for several seconds and more. In general, more frequent rebufferings result in longer buffering time. Larger network delay and higher packet loss ratio also worsen the buffering time. Thus, those parameter values recorded in the server log file are important to understand the client-side service quality.

2.3

An Approach to Extract ClientSide Service Quality from a Server Log File

To obtain precise information about streaming service quality on each client, we have to solve the following two problems. One is the difficulty of obtain2

Server Log Received Packets Recovered Packets Lost Packets Frequency of Rebuffering Duration of Rebuffering

Estimate

6HUYHU :LQGRZV0HGLD 6HUYHU

(QFRGHU :LQGRZV0HGLD (QFRGHU

Service Quality on Clients Duration of Frame Dropping

+''

Influence

Influence

7HVW0RYLH)LOH

5RXWHU3DFNHW)LOWHU 1,671HW

Network Status Network Delay Delay Jitter Packet Loss Rate

$XGLHQFH

Figure 1: Service quality estimation from a server log. &OLHQW :LQGRZV0HGLD3OD\HU

ing the QoS information directly from each client in addition to the parameters recorded in a server log file. To enable this, we need a special module integrated in the player software or the operating system. This approach is impractical because it requires each end-user to install the module and agree on the module execution. Since the module may cause a service fault, user support is also required. The other problem is the difficulty of wide-area network monitoring. As we already mentioned, it is impossible to obtain precise information about the data reachability of UDP communications by network monitoring near the server. In addition, it is almost impossible for server operators to grasp a full picture of service quality status on distributed wide-area networks. In this study, we propose an approach to estimate the client-side service quality from the server log data, which gets around the above two problems. We collect various service quality data through experiments changing network condition parameters such as transmission delay and packet loss ratio and reveal the correlation between the data recorded in the server log file and the service quality observed on the client. The analysis framework is depicted in Figure 1.

3 3.1

Figure 2: Experimental environment.

Table 1: Software versions and encode bit-rates. Player Video codec Audio codec Encode bit-rate

Windows Media Player 9 Windows Media Video 9 Windows Media Audio 9 340kbps, 148kbps, 58kbps

WMT is a software suite for streaming service, which includes a Windows Media Encoder (WME), a Windows Media Server (WMS), and a Windows Media Player (WMP). The WME, WMS, and WMP were connected via 100Mbps fast ethernet. The installed operating systems were Windows XP for the encoder and the client, Windows 2003 Server for the server, and Linux Fedora Core 3 for the PC router. The router that connected the server and the client implemented a network emulation mechanism with NIST Net [3]. In the experiments, we picked up various network parameters such as the network delay, the delay jitter, and the packet loss ratio. For each network environment configuration, the server delivered a 180-second streaming live video, and we observed the service quality on the client and collected the log data on the server. The versions of the software used in the experiments and the encode bit-rates are shown in Table 1. In each test, the video data was encoded in multi-bit-rate, which allowed the server to deliver the video data at the best bit rate considering the network condition.

Experiments Network Environment

The target application of this study is real-time streaming service over the Internet. Figure 2 depicts the experimental network environment and the service systems. In this environment, we adopt Windows Media Technology (WMT) [2] as a streaming service platform. However, our approach can be applied to any server-client streaming distribution systems. 3

Group A

Table 2: Service quality classification. Quality Group A Group B Group C

3.2

High Middle Low

Dropping Frames None Observed Observed

# of Received Packets 8000 7000 6000 5000 4000 3000 2000 1000

Rebufferings None None Observed

0

Criteria for Estimating Service Quality on Clients

Unexpected frame drops and playback pauses have an adverse influence on the streaming service quality. We therefore classify the streaming service quality into the three groups as shown in Table 2. The group A is the highest quality group with neither frame drops nor rebufferings. The group B is the middle quality group. Some dropping frames are observed but no rebufferings occur. The group C is the lowest quality group with rebufferings, which usually causes a large number of frame drops. In this classification, the rebufferings are easy to detect because the number of bufferings in the service is recorded in the server log. However, the detection of the frame drops is more complicated. In this study, we obtained the frame rate decline time, of which definition we described in subsection 2.2. In our experiments, we found that the frame rate decline time of a good quality streaming playback stayed under 20 seconds. Thus, we judged the streaming service quality was good, i.e., in the group A, when the frame rate decline time was under 20 seconds. This threshold value should be changed if some experimental conditions are different from those in this study. For example, a longer streaming playback can produce a larger frame rate decline time. We should adjust the threshold value when we evaluate real-world streaming service.

3.3

Group B

Group A Group B Group C

Group C

50 0 50 100 150 100 150200 200 250300 300 250 350400 450 350 # of Lost Packets # of Recovered Packets

Figure 3: Correlation between the service quality and the service statistics in the server log data. packets. In this situation, it is natural to think that all the data packets dropped in the network were recovered by retransmission and the client received all the streaming data because the server log reported no lost packets for the group B. However, the service quality categorized in the group B received a slight damage of frame drops. We guess one of the possible causes of this misleading situation is that the statistics report was generated by the network layer modules in the player software and not by the display layer modules such as the decoder. Even when a data packet is received by the network layer module in time, it may not be used for the playback because of the delay to transfer the data from the network module to the display module, e.g., the decoder. Anyway, the conclusion here is that the server operator should be sensitive about the number of recovered packets. The results in the group C are different from those in the group B in a larger number of lost packets. This means that some retransmissions were failed. In this situation, most users cannot tolerate the poor service quality. A point we have to notice here is that even a small number of lost packets can trigger a rebuffering.

Experimental Results

4

The goal of this study is to reveal the correlation between the client-side service quality and the server log data. Figure 3 depicts each result of the streaming distribution tests in a point with the axes of the number of received, recovered, and lost packets. We marked each point to classify the service quality level and circled the three groups. As depicted in this graph, the points in the group A are concentrated at the point with no recovered packets and no lost packets. Then, the group B stays on the line with no lost packets but some recovered

Related Work

Effective management of client-side service quality is a hot research topic. Dalal et al. [4] developed a client-side service quality management framework integrated into a client software. Although this is a straightforward approach to obtain the service state information from the client, an extra installation cost is required of the end-users. Contrastively, our approach is to extract the client-side service state information from the server-side data. 4

References

The approach of Wang et al. [5] is basically similar to ours. They picked up the time and frequency of re-buffering during the service and the number of lost packets as service quality indices. In this study, we conducted more precise experiments examining various other parameters such as the number of recovered packets. The experimental results showed that not only the number of lost packets but the number of recovered packets contains important implication of service degradation on the client. In addition, we defined the frame rate decline time as a service quality index, which allowed more precise service quality assessment.

5

Conclusions Work

and

[1] H. Koyun. Logging Model for Windows Media Services 9 Series, March 2003. http:// www.microsoft.com/windows/windowsmedia/ howto/articles/LoggingModel.aspx [2] Windows Media Technology. http://www. microsoft.com/windows/windowsmedia/ [3] NIST Net. nistnet/

http://www-x.antd.nist.gov/

[4] A. C. Dalal and E. Perry. A New Architecture for Measuring and Assessing Streaming Media Quality. In Proceedings of the Third Workshop on Passive and Active Measurement Workshop (PAM 2003), pages pp. 223–231, La Jolla, CA, April 2003.

Future

[5] Z. Wang, S. Banerjee, and S. Jamin. Studying Streaming Video Quality: From An Application In this paper, we proposed an approach to estimate Point of View. In Proceedings of the Eleventh the client-side service quality of multimedia streamACM International Conference on Multimedia, ing applications based on the server log data. We pages 327–330, Berkeley, CA, November 2003. conducted service experiments with various network ACM Press. configurations and revealed the quantitative correlations between the service quality perceived by the end-users and the client application status recorded in the server log file. Based on our experimental results, the server operators can estimate the client-side service quality by analyzing the server log data. The server operators have to pay careful attention to the number of recovered packets. Even when the number of lost packets is almost negligible, the service can be unsatisfactory if the number of recovered packets is large. Our future work includes more fine-grained parameter configurations in the experiments and more detailed analyses of the results. Especially, the influence of the content characteristics, e.g., playback length, encoding format, and degree of motion, to our conclusions should be explored. We also have to increase the accuracy of the frame rate decline time by removing the influence of the initial spike and the jitters of the playback frame rate.

6

Acknowledgement

We would like to thank Koji Ogawa for his offering the WMP statistics collector program. 5