2015 IEEE World Congress on Services
Automated Mobile Testing as a Service (AM-TaaS) Isabel Karina Villanes
Erick Alexandre Bezerra Costa
Arilo Claudio Dias-Neto
Federal University of Amazonas Manaus, Amazonas, Brazil
[email protected]
Federal University of Amazonas Manaus, Amazonas, Brazil
[email protected]
Federal University of Amazonas Manaus, Amazonas, Brazil
[email protected]
characteristics inherent to cloud computing and mobile test. For instance, some issues are identified in [9]: lack of standards, mobiles test environments and well-defined test models and coverage criteria to address distinct needs in mobile testing.
Abstract—Due high popularization of cloud services and the use of a wide range of mobile devices with different environments and platforms, a new model to offer software test service emerged, called Testing as a Service (TaaS). It uses cloud infrastructure to provide on-demand testing services on cloud for end users at any time. Nowadays there are many studies about different services about TaaS. Based on this scenario, we propose a framework, called Automated Mobile Testing as a Service (AM-TaaS), which offers automated test for mobile applications based on AQuA’s test criteria. We performed an experiment emulating nine Android mobile devices with different characteristics in order to test the criterion “OTA Install” (automated installation of mobile applications in a device). We emulated devices under virtual machines and cloud infrastructure. Experimental results show 100% of the emulated devices could be tested using automated test cases designed in the framework for this test criterion.
In order to facilitate and improve these issues, this paper presents a framework called “Automated Mobile Testing as a Service” (AM-TaaS), which offers automated tests for mobile apps based on AQuA’s test criteria. This framework aims to help developers and testers that require to evaluate the quality of their applications in various mobile devices in a short deadline, offering a friendly web interface with three simple steps: Upload .apk file, define mobile app characteristics and choose test cases and mobile devices to be emulated. A Proof of Concept (PoC) was conducted with five mobile apps in nine different emulated mobile devices, aiming to evaluate the feasibility of the proposed framework. Initially, we selected one test criterion proposed by AQuA for the Android platform (Install OTA). The results show that a mobile app was tested in nine different mobile devices with 18 test cases in about 14 minutes (considering the cloud infrastructure configured in the framework is still using a provisory server). This result is an overall average of the 5 app that were tested in the proof of concept conducted in this work.
Keywords—Cloud Computing; TaaS; Automated Test.
I.INTRODUCTION During the last decade, mobile phones have evolved into powerful, dynamic, connected and smart devices able to deliver all kind of services at our fingertips. As soon as this trend arises, mobile applications (called mobile apps) begin to be distributed with them; mobile apps demand is higher than ever. While mobile apps were originally offered for basic tasks, such as weather information, games (e.g.: the famous game Snake), email, nowadays it is more sophisticated like GPS mapping and location-based services, banking, video streaming and mobile medical apps.
This paper is organized as follows. Section II introduces reference background about Testing as a Service (TaaS), Mobile Testing and Mobile Testing as a Service. Section III specifies our proposed framework, Automated Mobile Testing as a Service (AM-TaaS) and describes the process proposed to use. Section IV reports the Proof of Concept (PoC) conducted with real mobile apps, emulating nine different Android devices from different brands. Section V reviews the related work. Finally, Section VI highlights some conclusions and future works.
Nowadays, mobile users demand more choice, functionality and opportunities to customize their phones. Mobile apps are time consuming, expensive and difficult because there are hundreds of different mobile devices, with different operating systems, browsers, screen sizes, various natives APIs, software features and hardware components[1][2][3]. This makes us think about mobile apps quality and how can affect users, because they will not be comfortable with a mobile app that crashes or loses important data. Therefore, complexity of mobile apps are making more difficult and critical to develop and test this new software category.
II.BACKGROUND A. Testing As A Service (TaaS) During the last years, a new service model, known as Testing as a Service (TaaS) or Cloud Testing, has gained enormous popularity due to its scalability. TaaS is a hot topic issue at the moment [10][6].
Thus, the use of cloud computing resources for software testing could help and improve this issue. Testing as a Service (TaaS) is becoming a hot research topic that is being used on different software platforms for Regression Test [4], Web Security [5], Unit Testing [6][7] and Load Web Testing [8]. However, testing in mobile apps faces challenges and 978-1-4673-7275-6/15 $31.00 © 2015 IEEE DOI 10.1109/SERVICES.2015.20
According to [11], TaaS is automated software testing as a cloud-based service. In [6], the authors define TaaS as a new service model to provide testing capabilities to end users. This 79
In order to support testing in mobile apps, AQuA1 (App Quality Alliance), a non-profit group led by members of main electronics industries such as AT&T, LGE, Motorola, Nokia, Oracle, Samsung and Sony Mobile, provides test criteria for different mobile platforms aiming to contribute for the quality of mobile apps. These criteria cover tests for several characteristics/features, such: install and launch, memory use, connectivity, messaging and calls, event handling, user interface, performance, battery usage, privacy and user permissions, amongst others. Some of these test criteria could be automated to reduce the effort for testing. These criteria help developers to ensure software quality in mobile apps.
service promotes a cloud-based testing architecture to provide online testing services following a pay-per-use business model [12]. In addition, it can be used for test environment or test tools, because all test activities and management are brought to the cloud [13]. TaaS combines two ideas: (1) offering software testing as a competitive, easily accessible web service, and (2) doing fully automated testing in the cloud, to harness vast, elastic resources toward making automated testing practical for real software [11]. The cloud computing is divided into three types of layers: IaaS, PaaS and SaaS [14].
Based on the relevance of mobile apps in our current scenario, the contribution provided by TaaS, as infrastructure to support automated testing, and the test criteria proposed by AQuA for the evaluation of mobile apps, we propose a framework to perform automated testing for mobile apps using the cloud infrastructure. We called this framework Automated Mobile Testing as a Service (AM-TaaS). On section V describes works proposed in the technical literature that deal with mobile testing using the cloud infrastructure.
• Infrastructure as a Service (IaaS): clients buy the infrastructure, as servers and storage, and the model is pay-as-you-go on demand; • Platform as a Service (PaaS): it provides a platform where the software can be developed and deployed allowing the customer to manage, run and develop the application; • Software as a Service (SaaS): it is deployed over the Internet, managed by a third-party vendor and its interface can be accessed on the client’s side.
C. Mobile Testing as a Service The growing in the use of mobile apps in the last years raised the idea of Mobile Testing as a Service (Mobile TaaS), defined as “a provider of an on-demand testing services for mobile apps to support software validation and quality engineering processes” [9]. This service is based on Mobile Cloud Computing (MCC), which is the intersection between both mobile and cloud computing[18], and as a new technology to achieve rich experience of a variety of mobile services at low cost. A general architecture of MCC[19] can be show in Fig. 1.
TaaS is executed as part of all three of these layers as it can be used to test the infrastructure, platform or software, or all combined. According to [15], TaaS is important due to the following reasons: • Cost reduction by leveraging computing resources in clouds. • Scalable test environments with virtualization. • On-demand testing service in 365/7/24. • Model Pay-as-you-test. • Quality certification by third parties.
Mobile TaaS uses the cloud infrastructure to offer a model “pay-as-you-test” to ensure the resource sharing and cost reduction. Nevertheless, in [20] the authors claim in mobile app testing, there are various unique factors that challenge the testing process. Such as device diversity (screen size, screen orientation, free memory, chipset, architecture), operating system, network and runtime environment. The use of mobile devices and apps are evolving exponentially. A research published by [21] explains that cloud services will evolve 17.3% until 2018 and it requires more attention about test mobile apps.
B. Mobile Testing The term Mobile Testing refers to different types of testing, such as native mobile app testing, mobile web app testing, and mobile device testing [16]. Mobile app testing refers to “testing activities for native and web applications on mobile devices using well-defined software testing methods and tools to ensure quality in functions, behaviors, performance, and quality of service, as well as features, such as mobility, usability, interoperationability, connectivity, security, and privacy” [16].
A systematic review published by Priyanka et al. [22] found two papers related to mobile commercial tools on cloud testing in a set of 38 scientific papers regarding cloud-based testing. Moreover, Gao et al. [16] present several TaaS tools for Android Platform, that are summarized in Table I. AM-TAAS (proposed in this work) is also included on this table in order to be compared with other tools.
Different mobile test infrastructures are defined, such as emulation-based, device-based, crowd-based testing and cloud testing. According to [17], mobile app testing has some similarities to website testing as both involve validation in many environments (smartphones and browsers, respectively). However, mobile testing has specific testing characteristics like different platforms, operating systems, screen sizes, memory and processing capacity, models by vendors and communication by different networks. Thus, testing all combination from these characteristics could be unfeasible.
Some researches [9], [17], [20], [23], [24] propose mobile infrastructure, nevertheless, it is not approached test automation for mobile apps. Unlike our framework, we include test automation based on AQuA’s test criteria, which were validated by experts in the field. 1
80
http://www.appqualityalliance.org
etc. The proposed framework aims to help developers with this task. The proposed framework is called Automated Mobile Testing as a Service (AM-TaaS). It has three important characteristics: (1) use of virtualization, (2) emulation-based mobile devices, (3) use of cloud infrastructure. Android was chosen to be the initial target mobile platform for the experiments because it is the most popular and widespread mobile OS, according to several statistics agencies. A. AM-TaaS Architecture and Implementation The framework’s architecture and implementation are illustrated on Fig. 2 and Fig. 3, respectively, where users (developers or testers) would use cloud test resources to validate their App file. Fig. 1
Mobile Cloud Computing Architecture
AM-TaaS comprises six components: Cloud Controller, Hypervisor, Virtual Machine, Test Automation Manager, Mobile Emulator Manager and Front End. They will be described in the next subsections.
NEOLOAD
SEETEST
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Open Source
X
X
X
X
Emulation-Based Test
X
X
X
X
X
X
X
X
Device-Based Test Testing For Native Apps
X
Testing For Web Apps GUI-Based Testing
X X
X
X
MOBILECLOUD
MONKEYTALK
MONKEYRUNNER
APPIUM
ATTRIBUTES
AM-TAAS
TABLE I. MOBILE TAAS TOOLS.
X
Load Testing
X X
Automated Criterion Test
X
X
X
X
X
The local infrastructure for private cloud environment is implemented over OpenStack [25], a cloud operating system that controls large pools of compute, storage and networking resources, all managed through a dashboard that gives administrators control while empowering their users to provision resources through a web interface.
X
Performance Testing
Supported Script Language
1) Cloud Controller The Cloud Controller handles authentication, reporting, and resources. It has administrative interface for the cloud management GUI to managing virtual machine, cloud-based resources and for end-users access to services.
X
X
In the next section, we explain our proposed Framework, called AM-TaaS, which aims to help developers and testers that require evaluating the quality of their mobile apps in various mobile devices in a short deadline.
Fig. 2. AM-TaaS Architecture.
III.AUTOMATED MOBILE TESTING AS A SERVICE Test automation has a great impact on mobile app quality, ensuring that app will work according to the characteristics defined in the development process, reducing human interference during tests execution. In this scenario, achieve mobile apps quality has not been an easy task, due mobile app tests require app execution in a large number of possible combinations of mobile devices, operating systems (SO) and different characteristics about screen size, memory capacity,
Fig. 3. (a) Implementation of AM-TaaS, (b) Components in Guest VM.
81
2) Hypervisor For virtualization environment, we could use a Hypervisor like KVM [26], VMWare [27] or XenServer [28]. The main task to be performed by this component is to run and control several virtual machines and manage the resources status. 3) Virtual Machine The VM runs Ubuntu 14.04. It allocates subcomponents Mobile Emulator Manager and Mobile Test Automation Manager. •
•
Mobile Emulator Manager: since the proposal is based on Android platform, the framework uses Android SDK Manager for mobile device emulation. It can run several different versions of Android Operating System using the resources of the host machine and mimics all the typical hardware and software features of a typical mobile device.
Fig. 4. Service Process of AM-TaaS.
Test Automation Manager: the goal of this component is to accommodate all automated tests criteria and provide emulated devices in order to run test cases with the MonkeyRunner tool. MonkeyRunner is the standard tool of Android SDK that provides automated execution of GUI-based tests written in Python [29]. Thus, a set of automatic test scripts to evaluate different test criteria proposed by AQuA would be available in this component to be executed according to the characteristics of the mobile app under test.
Step 3, the user needs to indicate which mobile device models will be emulated for tests execution.
•
Step 6, finally, in the last step, the framework sends a report to the user with the results obtained during the tests execution.
A. Set up of Environment For this study, mobile devices were selected based on smartphone sales [31], the most used Android versions in the market [32] (Fig. 5), and mobile devices classification [3].
Step 1, starts when the user accesses the service through web interface (using the MTControol tool). User needs to select a local app file and upload it. The format of app file must be Android application package (.APK), since in the current version, only Android apps are supported. In this moment, the user needs to upload the application file (.apk) to be tested and some information about its dependencies, such as the dependency of another mobile app in the device.
•
Step 5, automatic test scripts, located on the Test Automation Manager component, are executed on emulate devices according to the characteristics of the mobile app under test, defined in the steps #1 and #2, and the mobile devices models selected in the step #3.
A Proof of Concept (PoC) is a realization of a certain method to test a design, idea or assumption, demonstrating its feasibility. Therefore, in order to show the effectiveness of AM-TaaS, we designed a PoC experiment. It was set up in a controlled environment.
B. Service Process of AM-TaaS Fig. 4 depicts the service process to be followed by AMTaaS, which are:
Step 2, the framework’s user indicates types of testing to be executed on the mobile application under test.
•
IV.PROOF OF CONCEPT (POC)
In AM-TaaS, we are using the MTControol tool [30] to implement this component. MTControol is a tool that supports the management of testing for mobile apps based on AQuA’s guidelines. It has features to characterize mobile apps and recommend test cases based on test criteria published by AQuA for different platforms, including Android.
•
Step 4, it is an internal process where the component Mobile Emulator Manager starts the emulated devices.
In the next section, we present a proof of concept using the AM-TaaS framework with some real mobile apps. The goal is to evaluate the feasibility of the proposed framework in test mobile apps using the cloud infrastructure, initially, for one test criterion published by AQuA for Android platform.
4) Front End The access to front end could be done by any machine to launch OpenStack using web interface (with any browser Chrome, Firefox, IE, Safari).
•
•
This classification defines three categories for mobile devices, however, we needed to redefine it aiming to attend new mobile characteristics. Thus, they were grouped and classified based on this information and the new categories are show in Table II. The environment was implemented with a VM (Ubuntu 14.04, 8GB Ram, 50GB HDD) and nine emulation-based mobile devices. Table III lists the mobile devices characteristics. We can observe they have different versions of Android S.O., size screen, RAM capacity and models.
82
Fig. 5. Most Popular Android Versions.
TABLE II. MOBILE DEVICES CLASSIFICATION. GROUP
PRIORITY
GROUP 1
C
GROUP 2
B
GROUP 3
A
DESCRIPTION Small devices with a small CPU, RAM (
