A Framework for Consumer Devices Reliability Metrics Estimation Branislav Kordic, Ivan Kastelan, Milos Pilipovic, Darko Lulic
Abstract—This paper presents a framework for consumer devices reliability metrics estimation utilizing tools and systems based on statistical usage testing and black box testing methodology. The framework aims to provide modularity and adaptability due to fact that most often a testing system is composed of different tools and subsystems. The proposed solution is evaluated by means of particular case study using commercially available and industry proven statistical usage testing tools and black-box testing system. In the case study some of basic features of an Android based Set-Top-Box device are verified. Finally, lessons learned and brief overview of available test systems and tools are given. Index Terms—Reliability metrics estimation, statistical usage testing, black-box testing, automated test systems, test tools.
(reliability, mean-time-between-failure and etc.) utilizing tools and systems based on statistical usage testing and black box testing methodology. The framework is composed of hardware and software parts which provide to us to model device’s usage (i.e. behavior) model, to generate and execute test cases, and to calculate reliability of device under test. The framework aims to provide modularity and adaptability due to fact that most often a testing system is composed of different tools and subsystems. The rest of the paper is organized as follows: Section II gives a framework structure overview. In Section III through a case study framework implementation and validation are described. Lessons learned and brief overview of testing tools is given in Section IV and Section V, respectively. Conclusions are given in Section VI.
I. INTRODUCTION
II. FRAMEWORK OVERVIEW
Modern multimedia and smart devices accommodates dozens applications able to communicate with each other, share and stream video and audio content, and to control remote systems over a network as well. Usually an end-user in its home has more than one multimedia and handheld device such as smart Digital TV (DTV) or Set-Top-Box (STB), tablets, consoles, and mobile phones which are networked using a router. In this kind of a setup, reliability and robustness of the device plays a major role. Measuring consumer device’s reliability metrics is not an easy task for several reasons: (i) significant numbers of application are supported and all applications are not equally used by user, therefore not equally important, what makes this task more difficult, (ii) a device is composed from various software and hardware components developed by third-party, and (iii) due to constant pressure to reduce time-to-market. Of course, demanded reliability of consumer devices is not same as for some real-time critical systems but anyhow certain level need to be provided. In this paper we present our proposal of a framework for consumer devices testing and reliability metrics estimation
The framework structure is presented in Fig. 1 and it consists of the following: - Device under test (DUT) usage data collector is used to collect data about DUT usage, which functionalities are commonly used, which buttons are frequently pressed, and etc. This part of the framework provides information what should be included in a device usage model and thus improves quality and relevance of generated test cases and test campaign in general. - Usage model creator is a tool used to make a DUT usage models. The usage model(s) is used to generate test cases for a target automatic test execution system (i.e. test bed). Afterwards, using obtained test results against the usage model(s) reliability metrics estimation is calculated. - Test case generator uses a usage model to create test cases automatically. Depending on a tool test cases format may be different – test cases can be generated in a script language or in some kind of meta-language. Also, different test case generation strategies such are state, transition or inputs (stimuli) coverage may be supported. - Test case adaptor is used to adjust generated test cases to target test bed. It is necessary if test case generator doesn’t support target test bed, or if generated test cases are in meta-language form. - Automated test execution system at least needs to provide two functionalities: automated test execution (includes DUT control) and test campaign report generation. Black-box testing methodology is preferred because in general an internal structure of DUT is unknown.
Branislav Kordic is with the Faculty of Technical Sciences, University of Novi Sad, Trg Dositeja Obradovica 6, 21000 Novi Sad, Serbia (e-mail:
[email protected]). Ivan Kastelan is with the Faculty of Technical Sciences, University of Novi Sad, Trg Dositeja Obradovica 6, 21000 Novi Sad, Serbia (e-mail:
[email protected]). Milos Pilipovic is with the RT-RK Institute for Computer Based Systems, Narodnog fronta 23a, 21000 Novi Sad, Serbia (e-mail:
[email protected]). Darko Lulic is with the RT-RK Institute for Computer Based Systems, Narodnog fronta 23a, 21000 Novi Sad, Serbia (e-mail:
[email protected]).
DUT_Functio nal_Spe cification
DUT Usage Data Collector DUT_Usa ge_Stats.xml
Usage Model Creator Ope rational_Pro file.file
Test Case Generator Test_Suite_Descrip tion.file
Test Case Adaptor Test_Suite.py
Automated Test Execution System Test_Results.html
Test Results Parser Test_Results.file
Relability Estimation Module DUT_Relabil ity_Report.file
Fig. 1. The framework structure
- Test result parser is considered as a simple tool used to extract test results from a comprehensive test campaign report. Processed results (number of passed, failed and inconclusive tests along with the number of test steps executed) are inputs to the next component. - Reliability estimation module encapsulates mathematical methods and models for DUT reliability metrics estimation such as reliability (R), mean-time-betweenfailure (MTBF), and mean-time-to-failure (MTTF). Mathematical models have to support reliability metrics estimation for different DUT testing phases. III. CASE STUDY In the case study the framework is implemented using following components: a) ALL4TECH MaTeLo tool [1] b) RTRK BBT system [2] c) Third-party and custom made tools A. MaTelo tool MaTeLo tool (Markov Test Logic) is commercial tool developed within Eurospean IST project [3]. It comprising two tools important for the framework implementation: Usage model editor and Testor. Usage model editor is GUI tool for creating usage model(s). Usage models are modeled as Markov process with a macro states support, which sometimes considerably simplifies usage model and states organization. In
later steps, usage models are used by Testor tool to create abstract test cases. Testor has twofold role: (1) generate abstract test cases utilizing usage models and (2) implements logic for reliability metric estimation – R and MTTF. In order to calculate R and MTTF to Testor usage models and test results report must be provided. Test results report has to provide results for every executed test step. In the framework MaTeLo Usage model editor and MaTelo Testor tools are used as Usage model editor, Test case generator and Reliability estimation module. B. BBT system RTRK Black-Box-Testing (BBT) system is a part of RTRK BBT ecosystem which includes an automatic test execution system, test tools and services. It provide comprehensive, feature-rich, and modular automatic test case execution framework along with support for third-party components and subsystems integration. For the implementation RT-Executor tool and RT-AV100 and RT-IR001 devices are used. RTExecutor software tool drives whole testing system: automatic test campaign execution, DUTs control and test results gathering. Inputs to RT-Executor are configuration files (defines a list of control devices – remote controller, etc.) and a set of test suites written in Python language. RT-AV100 and RT-IR001 devices provide real-time audio and video content capturing, and infra-red command transmitter-receiver, respectively. RT-AV100 capture output from a DUT as a picture and send it to RT-Executor for further processing. To check is a DUT in an expected state RT-Executor asserts captured picture with a referent picture. RT-IR001 is used to send infra-red commands to DUT. In the implementation RTRK BBT system is used as Automated test execution framework. C. Third-party tools Testor tool generates abstract test cases which cannot be executed within BBT system hence it must be adapted to suit particular execution environment. Test case adaptor is a simple custom made C++ based command-line tool developed to translate abstract test cases to Python scripts. BBT system generates test reports with all data gathered during testing process, but BBT system data report format doesn’t match Testor expected format. To be able to extract all data needed by Testor another custom made command-line tool was developed - Test result parser. Indeed, Testor supports that test results can be entered through a web browser but that is rather time consuming task. Inputs to the test result parser are an abstract test cases and test results reports generated by Testor and BBT system, respectively. In our solution DUT usage date collector is composed from RT-Executor tool and RT-IR001 device. The device is configured to a receiver mode what enable us to capture infrared commands sent to a DUT. In the case study we collected data about frequently pressed buttons on a remote controller. Using collected data we managed to calculate probability of pressed buttons as well to estimate what commonly used functionalities may be (live TV, zapping, EPG browsing, etc.).
D. Testing and reliability estimation validation As a proof of concept for proposed framework an Android based STB device is set-up for testing and reliability estimation. Based on collected data about STB usage we modeled several use case scenarios: first installation procedure, zapping, and STB menu navigation. The STB usage model is created in same manner as is described in [4] and [5]. For sake of simplicity we didn’t create too complicated models (i.e. use cases) because they require more test equipment such as signal streamers and modulators. In Fig. 2 part of STB usage model is shown. By using MaTeLo Testor tool we generated three test suites up to 400 test cases (the academic license is used). Generated test suites contain abstract test cases in xml format so with test adaptor they are converted to py scripts. After the conversion the test suites are executed within BBT system. STB testing results are given in Table I. In the testing process we executed test cases until first failed test step. Not executed test cases are marked as inconclusive. When a test campaign is finished a test report is generated. In BBT system the test report is generated in html format. Test results parser as inputs takes the html report and xml abstract test cases files, and as output produces new xml file with test cases steps and corresponding results for each test step. As mentioned earlier, this could be done manually, but for a larger set of test cases it is very time consuming. Finally, the test results are imported to Testor for R and MTTF estimation. The estimated results are given in Table II. MaTeLo tool calculate R from equation (1).
TABLE I STB TESTING RESULTS
Num. of test suites Num. of test cases (TCs) Num. of PASS TCs Num. of FAIL TCs Num. of INCONCLUSIVE TCs
3 1200 840 1 359
TABLE II STB RELIABILITY METRICS ESTIMATION RESULTS – R AND MTTF
k [day] 0.5 1 15 25 35
R 0.953548 0.909254 0.240035 0.092710 0.035808
MTTF [day] 20.039429 10.019714 0.667981 0.400789 0.286278
This calculation is done according to the number of activations (k) during a time period (t) and usage probability (pu) (Sayre or MaTeLo). MTTF is calculated in relation to the reliability and it evaluates time to the first defect detection. Usage probability value, pu, represents the probability to have a software execution without any defect [6].
R (t ) = ( pu ) kt
(1)
A time period and k values are very important parameters and they have to be carefully defined. In fact they define the frequency of device usage, i.e. a number of system inputs per time unit that cause state transition within usage model. To emphasize, in order to get meaningful results for estimations a several test campaign must be conducted as well as valid parameters must be defined because inappropriate values of a time period and k parameters can lead to too pessimistic or too optimistic estimations. In Table II is shown how different values of k influences R and MTTF – it is likely that a defect be revealed when a system is frequently used. II. LESSONS LEARNED
Fig. 2. Part of STB usage model
In the presented framework Automated test execution system and Reliability estimation module are essential to achieve valid and relevant results. Obtained test results must be valid, reliable, and have to provide fine granularity regarding test case execution. A test execution system must track test cases execution in step-bystep manner and need to check is a DUT set in a correct state after each step (need to determine is a test step pass, failed, or inconclusive), otherwise the results will not be well correspond to a usage model what can lead to incorrectly estimated values. Also, important and a tricky task, which sometimes need to be done, is to restore DUT to its last known or/and last stable state. Mainly problems are caused by hidden states which can’t be modeled or not included in a usage model for any reason. For instance, in software component it is impossible to model
memory leaks as a state, or if system supports default settings than those can be omitted in order to simplify a usage model. These issues are equally addressed to tools and test execution systems. It’s most likely that automated test execution system or framework is not supported by modeling and test case generation tools thus custom test cases translation tool have to be developed. It is recommended to choose tools which are able to generate abstract test cases because they in many ways facilitate test cases to a particular test framework adjustment. III. BRIEF TEST TOOLS AND SYSTEMS OVERVIEW In the paper as a tool for statistical usage testing MaTeLo tool is used. MaTeLo tool is commercial tool with the academia license support. The users are not only limited to MaTeLo tool. JUMBL is another interesting open-source tool which utilizes Markov process too [7]. It’s aimed for automated, model based and statistical usage testing. JUMBL tool is Java class library supported with a set of command-line tools and utilities. In comparison to MaTeLo tool JUMBL is modest but certainly very useful tool. Anyhow, if available tools may not satisfy user’s requirements a new tool may be developed using some of existing frameworks [8]. We choose MaTeLo tool because it’s developed to satisfy industry needs, contains GUI based editors and tools, and support vast number of features (able to generate abstract test cases, supports different testing strategies, UML support, and etc.). Testing frameworks are dominantly very expensive commercial tools with minor or not at all contribution to opensource community due to fact that they rely on proprietary software and dedicated hardware. On the market are available several test frameworks which supports black-box-, grey-box-, and white-box- testing methodology. As a replacement for RTRK BBT system, with focus on multimedia devices, S3 test system may be used [9], and for testing more general systems NI Automated test solution may be a choice [10]. IV. CONCLUSION In this paper we presented a framework proposal aimed for consumer devices reliability metrics estimation (reliability, mean-time-to-failure, and etc.). The framework relies on existing automated black-box testing systems and tools based on statistical usage (or model based) testing methodology. The goals are to compose modular and viable system for devices reliability metric estimation using open-source and commercially available tools and systems suitable for broad class of multimedia consumer devices such are tablets, mobile phones, and smart DTVs and STBs. The proposed framework is validated by means of particular case study. In the case study ALL4TECH MaTeLo tool and RTRK BBT system for automated test execution are
used. MaTeLo tool(s) are used as usage model editor, automatic test case generator, and as reliability estimation module. RTRK BBT system is used as automatic test execution framework to execute test cases and generate test campaign report. Further, as lack of compatibility of MaTeLo tool and BBT system, in order to band together two parts of the system, custom third-party tools – test case adaptor and test results parser – are developed As proof of concept within the framework a STB device reliability and MTTF is successfully estimated. Finally, we conclude with lessons learned during implementation as well as brief overview of available tools and systems which may be used as substation for MaTeLo tool and RTRK BBT system. Our on-going work is oriented towards automated web testing frameworks and its integration in order to be able to do estimation of multimedia web-based application and services especially for Android operating system. V. ACKNOWLEDGMENT This work was partially supported by the Ministry of Education, Science and Technological Development of the Republic of Serbia, under grand no. TR32014. REFERENCES [1] http://www.all4tec.net/MaTeLo/homematelo.html [2] http://bbt.rt-rk.com/overview/ [3] A. Guiotto, B. Acquaroli, A. Martelli, “MaTeLo: automated testing suite for software validation”, Proceedings of DASIA 2003, Jun. 2003, Prague, Czech Republic. [4] V. Marinkovic, B. Kordic, M. Popovic, V. Pekovic, “A method for creating the operational profile of TV/STB device to be used for statistical testing”, 2013 IEEE EUROCON, July 1-4 2013, Zagreb, pp. 93-97 [5] T. Teckan, V. Zlokolica, N. Teslic, M. Gunduzalp, “User-driven automatic test-case generation for DTV/STB reliable functional verification”, IEEE Transactions on Consumer Electronics, vol. 58, May 2012, pp 587 – 595 [6] All4tec–MaTeLo User Manual.pdf, http://www.all4tec.net/wiki/ [7] S. Prowell, “JUMBL: A tool for model-based statistical testing,” System Sciences, 2003. Proceedings of the 36th Annual Hawaii International Conference on System Sciences, vol. 00, no. C, pp. 1–9, 2003. [8] M. Popovic; I. Velikic, "A generic model-based test case generator", 12th IEEE Int. Conf. and Workshops on the Engineering of Computer-Based Systems (ECBS '05), Greenbelt, Maryland, Apr 4-7 2005. pp 221 228 [9] http://www.s3group.com/tv/ [10] http://www.ni.com/automatedtest/
