Correlating User Perception and Measurable Network Properties: Experimenting with QoE P´al Varga, Gergely K´ un, G´abor Sey, Istv´an Moldov´an, and P´eter Gelencs´er 1
Budapest University of Technology and Economics, Department of Telecommunications and Media Informatics, Magyar Tud´ osok krt. 2., H-1117 Budapest, Hungary (pvarga,kun,moldovan,gelencser)@tmit.bme.hu, 2 AITIA International Inc., Czetz J. u. 48-50., H-1037 Budapest, Hungary
[email protected]
Abstract. User perception of a networking service is usually very different from the operators’ understanding of service usability. Quality of Experience (QoE) metrics are supposed to describe the service from the end-users’ point of view – although QoE is hard to measure for mass services. Collection and analysis of QoS and SLS (Service Level Specification) properties of networking services are daily tasks of the operators. These metrics, however often provide misleading description of user satisfaction. Our ultimate aim is to find methods and metrics determining QoE by passive measurements on an aggregated network link. In this paper we describe our experimental results on correlating the severity of a network bottleneck and the experienced service quality. During our measurements we have loaded the network with various kinds of service requests and made notes on the perceived quality. We have also captured packet level traffic, and derived metrics based on packet interarrival times, packet size information and packet loss information. This paper briefly presents some of our analysis results.
1
Causes of Quality of Experience Degradation
Quality of Experience is measured – or more appropriately: perceived – at the end terminal, by the user. QoE shows similarities to Quality of Service (QoS), it does, however, differ from QoS by various means. QoE is subjective in nature, furthermore, service availability and user access capabilities get more focus in QoE than in QoS. There can be several scenarios where the experienced service quality becomes less than satisfactory. The roughest QoE degradation is the unavailability of a service. This actually happens more regularly than one might expect. Examples of temporal and/or regular service-unavailability include an unsuccessful paying procedure at a webshop, a broken download, or a ”webpage unreachable” message. These cases are very hard to measure from a central site, since continuous application monitoring and data processing is not always feasible. Another type of QoE violation is tied to network QoS, hence it can be correlated with some
2
kind of QoS metrics. Network related QoE-degradation can manifest itself as extended response time, decreased download speed, less enjoyable audio and video streaming. This is usually caused by one or more network bottlenecks, where packets get congested, queued, then delayed or dropped. There are only a few QoE-QoS measurements published at the moment. An example of these is [1], where the authors describe the effects of end-to-end delay of live video streaming on QoE. In this paper we describe our results on correlating the severity of a network bottleneck and the experienced service quality. There can be several link properties applied to characterize the presence of bottlenecks in packet data networks [2]. In the following section we describe our experimental results on correlating perceived QoE and properties of some bottleneck metrics.
2
Measurement scenario, methods and results
Our aim is to correlate the end-user experience with measurable properties of the traffic. The metrics that correlate better with the end-user experience will be more effective QoE measures than the ones that do not show direct correlation. To reach this aim we have set up a configurable DSL-like environment in a segment of our university department, where the department staff was the suffering object of our QoE testing. Altogether about 50 end-user machines were involved in the test. We have degraded the usual 100 Mbps Internet access to 1 Mbps DSL-like lines. Moreover, we have introduced an artificial bottleneck for the department segment. Later this bottleneck was changed from 7 Mbps to 2.5 Mbps in six steps. To obtain more subjective results, the downgrade was not linear; sometimes we have upgraded the service - pretending as if it was going to be normal. We have allowed each of these bottleneck scenarios to last 30 minutes. During the tests we have captured the traffic at two measurement points, and made notes on our experiences as network application users. The measurement scenario is depicted by Figure 1.
...
SWITCH
Hub
- Bridge “DSL MUX”
Hub
- Bridge “Aggregate Bottleneck”
SWITCH
Internet
packet capture
Fig. 1. Measurement architecture with the relations of download volumes
3
We have extensively used various networking applications and services during the test. Applications included audio and video streaming, web-browsing, ftp and peer-to-peer downloads, Internet-telephony (Skype), database access and many more. Table 1 shows our experiences in function of the bottleneck-severity.
Available Bandwidth
Video stream
Audio Skype stream
P2P traf- Web fic browsing
7.0 Mbps
perfect
perfect
perfect
33-95 kBps good
5.0 Mbps
occasional squares in picture
perfect
perfect
33-90 kBps good
4.5 Mbps
playing pauses at key-frame changes perfect
perfect
33-88 kBps slow
4.0 Mbps
sec-long pauses at key-frame changes perfect
perfect
12-14 kBps slow
3.5 Mbps
bad; squares and pauses are regular short pauses
scratching 3-10 kBps
unbearably slow
2.5 Mbps
very bad; not enjoyable
scratching 3 kBps
extremely bad
longer pauses
Table 1. Perceived quality of network services at different bottleneck scenarios
To correlate QoS with QoE, we have derived packet-level properties from the collected traces. As a basic analysis, these included throughput, packet retransmission (TCP packet loss). As a more advanced analysis, we have calculated the delay factor of flows [2] in order to understand what the users may experience as application delay. Furthermore, we have derived the metric called PIT kurtosis (the 4th central moment of Packet Interarrival Time distribution) [3]. This measure indicates the severity of queuing in the traffic path. Figure 2 introduces our results by presenting the most interesting of them, thus from the seven scenarios only two could be seen: the 7 and the 4.5 Mbps cases. When the available bandwidth (ABW) was 7 Mbps the network was not overloaded: all the applications worked properly, while at 4.5 Mbps the network load caused service degradations in case of some applications (see Table 1). Figure 2.a presents four average metrics calculated for the two scenarios. We have found (like other authors [4]) that the calculation of an average packet loss ratio cannot reflect the severity of the congestions. Likewise the throughput metric: the measured amount of traffic was approximately the same in the two cases. In spite of these the average delay factor and much rather the kurtosis of PIT PDF shows a significant difference between them. Figure 2.b depicts the calculated delay factor values for 3 minutes long intervals for both scenarios. At 7 Mbps there is no severe congestion, however at 4.5 Mbps, there is. In the second part of the 4.5 Mbps scenario the delay factor diminished, mainly due to the decrease of user-generated traffic. Certainly users gave up trying to access network services after a number of unsuccessful requests. This also explains that the difference of average delay factors is smaller than the PIT PDF kurtosis values’ for the 7 and 4.5 Mbps cases (see Figure 2.a). Figure 2.c and d present the number of resent packets in function of time, calculated for each 10 ms. Considering the two graphs there is no significant
4 10
7 Mbps
20
7 Mbps ABW 4.5 Mbps ABW
8
4.5 Mbps
Delay Factor
16
6
4
12 8 4
2
0 0
3 delay factor
PIT PDF kurtosis
PLR [%]
a. Average values of metrics
21
30 Time [min]
39
48
57
b. Delay Factor values for 3 min intervals
5
5
4
4
rtx packets/10ms
rtx packets/10ms
12
Thr [Mbps]
3 2 1
3 2 1
0
0
0
8000
16000
24000
32000
Time slices
c. Number of retransmission at 7 Mbps
0
8000
16000
24000
32000
Time slices
d. Number of retransmission at 4.5 Mbps
Fig. 2. Measurement results for two available bandwidth scenarios
difference between them. TCP packet loss type metrics and retransmission counts may suggest some network misbehavior only in very special cases, but in general these measures do not provide really useful information, even if it is analyzed for short time periods.
3
Summary
This paper investigated network performance metrics and aimed to correlate these with the end-users’ Quality of Experience. Our standpoint is that the metrics used for detecting bottlenecks are able to reflect QoE as well. In order to prove our ideas we have carried out a number of measurement scenarios in real network environment with different bottleneck conditions, some of which were presented in the paper. Presently we are working on further analysis of traffic behavior in bottleneck network situations, and in QoS aware networks.
References 1. Rahrer,T., Fiandra R., Wright,S., Triple-play Services Quality of Experience (QoE) Requirements and Mechanisms, DSL Forum Working Text WT-126, February 21, (2006) 2. Varga, P., K´ un, G., Fodor, P., B´ır´ o, J., Satoh, D., Ishibashi, K.: An advanced technique on bottleneck detection. In IFIP WG6.3 Workshop, EUNICE 2003, (2003) 3. Varga, P., K´ un, G.: Utilizing Higher Order Statistics of Packet Interarrival Times for Bottleneck Detection. In IFIP/IEEE E2EMON05, (2005) 4. Vacirca, F., Ziegler, T., Hasenleithner, E.: An Algorithm to Detect TCP Spurious Timeouts and its Application to Operational UMTS/GPRS Networks. Journal of Computer Networks, Elsevier, (2006)
