International Conference on Computer & Information Science (ICCIS)
2012
A Novel Approach of Automation Testing on Mobile Devices
Leckraj Nagowah and Gayeree Sowamber Computer Science & Engineering Department University of Mauritius Reduit, Mauritius
[email protected] ,
[email protected] Abstract-Mobile phones and mobile applications have now become
an
integral
part
of
our
everyday
life.
Mobile
application testing plays a pivotal role in making the mobile applications
more
reliable
and
defect
free.
Existing
test
compromise the reliability of the test since emulators are not the actual devices [2] and may not reflect the actual results if the same tests are run on the mobile device itself. To enable automation testing directly on mobile devices,
automation tools have been tailored to perform mobile test automation through mobile emulators. Other tools require the
we
mobile
Framework
device
connected
to
where a
the
application
computer so
is
installed,
that the tests
to
be
can be run.
Obviously, the results obtained from emulators often differ from those obtained on actual mobile devices. To our best
introduce
MobTAF,
that
a
enables
Mobile
mobile
Test
Automation
applications
to
be
automatically tested on actual devices and not on emulators. The rest of the paper is structured as follows: Section II
knowledge only a few solutions for creating automated tests of
gives an overview of test automation frameworks, test
mobile applications exist and their functionality is very limited.
automation tools performing mobile application testing and
In this paper, we introduce the idea of implementing a mobile test automation framework- MobTAF which performs mobile test automation on the mobile device where connection to a computer is not required. The framework moves the testing
also presents some related works being carried out in the field. The section also presents a critical analysis of the existing
works.
Some
design
issues
for
MobTAF are
infrastructure to the phone, to support test situations that
discussed in Section Ill. Section IV gives the system
cannot easily be replicated with an emulator. A prototype
architecture of
application
MobTAF. Details about the implementation of a prototype
was
then
automation framework,
implemented MobTAF.
to
test
The results
the
mobile
prove
that
MobTAF framework is an efficient way of performing mobile application tests directly on the mobile devices.
Keywords - mobile test automation framework; mobile application testing; mobile device test automation; mobile testing; software testing I.
provides an overview of the tests carried out on MobTAF and also presents some discussions on the framework. Section VII contains an evaluation of MobTAF against the commonly known automation tools and fmally Section VIII concludes the paper and also presents some future works. II.
As a result of the tremendous leap in mobile technologies, The biggest challenge is not only to develop them but also to test these applications to ensure their stability and robustness. Testers generally have two options when it comes to mobile application testing, either to test using real or
using
emulated
devices.
Running
tests
on
emulated devices are much easier compared to real devices. A tester can easily switch to different device types by
framework,
based on MobTAF are given in Section V. Section VI
INTRODUCTION
mobile applications are getting more and more complex.
devices
our mobile test automation
This
section
LITERATURE REVIEW
provides
an
overview
of
eXlstmg
test
automation frameworks, commercial test automation tools and a survey of studies have been carried out in the field of automation testing on mobile devices. A.
Test Automation Frameworks A number of test automation frameworks [3] already exist
among which there are: Test
Script
Modularity
Framework:
It
requires
the
simply loading the appropriate device profile. Moreover,
creation of small independent scripts that represent modules,
emulators run on powerful PCs which have been designed
sections, and functions of the application-under-test. These
with
application
testing
in
mind.
However,
the
big
disadvantage of emulated devices is that they lack the
small scripts are then used in a hierarchical fashion to construct larger tests that make up a particular test case.
peculiarities and faults that only real devices can provide.
Test Library Architecture Framework: It is very similar to
Although real mobile devices have limited processing and
the Test Script Modularity framework and offers the same
their
advantages, but it divides the application-under-test into
applications on mobile devices [1 ]. Test automation on
procedures and functions instead of scripts. This framework
mobile devices has just been emerging in the last few years.
requires the creation of library files that represent modules,
There are existing test automations frameworks, tools and
sections, and functions of the application-under-test. These
techniques that are used to run automated tests on the
library files are then called directly from the test case script.
storage
capabilities,
testers
should
also
test
mobile applications. However it has been observed that
Data-Driven Testing Framework: It is a framework where
these automated tests are mostly carried out on mobile
test input and output values are read from data files and are
emulators.
loaded into variables in captured or manually coded scripts.
But
running
the
tests
on
978-1 -4673-1 938-6/1 2/$31 .00 ©20 1 2 IEEE
[924 ]
emulators
may
2012
International Conference on Computer & Information Science (ICCIS)
whose
and output verification values. Navigation through the
client/server architecture where the PC acting as the client
program, reading of the data files, and logging of test status
sends test execution requirements to the device which acts
and information are all coded in the test script.
as the server.
The Hybrid Test Automation Framework: It is the most commonly implemented
framework which is simply
a
results
are
being
It
In this framework, variables are used for both input values
reported.
is
based
on
a
Fig. 1 depicts the design of Hermes Framework showing the interactions of the key components of the client server
combination of the above frameworks. Test modularity can
architecture. The components of the framework are briefly
be implemented by nesting test scripts, and the library files
described in Table 1.
can be used to implement functions, procedures, or methods.
The Hermes Framework has proved to be very effective
This framework allows data driven scripts to use the
in
libraries and utilities that are accompanied by a keyword
technical knowledge is required to write the test cases in
driven architecture.
XML.
B.
by
Test Automation Tools A test automation tool automates the usual steps that are
involved in a test. There are different types of tools that are suitable for various applications. These tools make the testing process more quick and efficient. Some popular test automating
tools
HP
are
QuickTest
Professional,
detecting
the
maximum
number
of
defects.
However
The testing capabilities of Hermes are also limited functionalities
of
the
J2ME
API
where
some
properties such as color and table width ofGUI components cannot be tested. The application behaves differently when run on different types of emulators such as GUI layout, image presentation, component size. Mobile subsystem
Test
Application
Complete and Selenium amongst others.
Test Agent
HP QuickTest Professional [4] also known as QTP is the
most advanced automated functional and regression testing solutions that support all major software applications and environments. The tool captures, verifies and replays user interactions Through
automatically partnerships
for
with
PC
based
applications.
cutting-edge
PC subsysten1
innovative
companies such as DeviceAnywhere, HP is extending its industry-leading functional testing capabilities to mobile applications. Test
Complete
[5]
is
a comprehensive solution
for
automating testing on a variety of Windows Mobile-based devices.
With
See Test
plug-in
for
TestComplete,
TestComplete can be used for test automation of real mobile devices connected via USB to a computer [6]. Selenium [7] is a portable software testing framework for web
applications.
The
Selenium-IDE
(Integrated
Development Environment) is the tool to develop Selenium
Figure 1. TABLE 1.
test cases. Selenium does not currently support mobile XML Test Script
a
mobile test
automation system
developed by Conflair, automates the tests on a PC using common test automation tools like QuickTest Pro. While the tests are executed on the device itself, they need to be controlled from a PC. The mobile device is connected to the PC through WiFi, cable or even a cellular network. The system supports different platforms and the same scripts can
Generator
GUI Visual Modeler Test Script Execution Automation
be used for a number of devices. However, the system also
Test Report
requires a connection to a PC, thereby limiting its use for
Generator
testing mobile applications on the go. C. Related Works A number of studies have been carried out in the field of automation testing on mobile devices. The Hermes Test Framework [9] has been devised to write tests which are to be run in a distributed environment
KEy COMPONENTS OF HERMES FRAMEWORK CLIENT: PC System
application testing except through the use of web drivers for mobile browsers. ™ RealMobile [8],
Hermes Framework
The test script is represented as an XML document. The test generator generates the XML test script by providing a visual editor to specify the test cases. It generates a visual model of the mobile application's GUI which will in turn help to specifY the test cases. It is located within the PC and processes the XML test script and sends test execution requests via the local communication module to a mobile device. Responses returned from the device to the Test Script Execution Automation and results are presented by the Test Report Generator. SERVER: Mobile System
Communication Module
It uses the Bluetooth technology to support remote communication.
Application
The application under test.
Test Agent
It processes the incoming test execution requests. It simulates a virtual user sending inputs to the application and send responses to the client pc.
[925 ]
2012
International Conference on Computer & Information Science (ICCIS)
II
Although some other studies [1 0, 1 1 ] have been carried out on mobile application test automation, we could not find a thorough evaluation of their proposed systems.
n
It has also been observed that automation testing has always been performed with the mobile device connected to
II
the pc. The test scripts reside on the local pc and requests are
sent
to
the
mobile
for
test
execution
Via
a
communication module. None of the frameworks perform
Figure 2.
with the pc to run the test on the device, other run their tests on emulators installed on the PCs. To our knowledge, we could not find a mobile test automation framework that runs directly on mobile devices.
II
Data Layer
I
II�
Reporting Layer
the automation testing of the mobile applications solely on the mobile device. While some frameworks communicate
II�
Abstract Layer
Interaction between the layers
Abstract Layer: The components in the layer will interact with both reporting and data layer to run the test. Reporting
Layer:
This
layer
will
consist
of
the
components which will be responsible for generating test reports. Data Layer: This layer will consist of all the data that will
Ill. DESIGN ISSUES From an analysis of existing frameworks, tools and works, a set of design issues for a mobile test automation framework are presented in this section. A mobile test automation framework should support the
be stored in the database. Fig. 3 gives a detailed overview of the interactions of components in the framework. A. Object Repository
automation testing directly on mobile devices where no
The Object Spy which forms part of the object repository
connection to a personal computer is required. This will
identifies the objects present in the application's Graphical
enable mobile applications to be tested on the go in real life
User Interface and retrieves its corresponding properties.
situations. The test cases and test data should be saved on
Once retrieved, the details are sent to the Object repository
the mobile device and therefore can be executed anywhere
which then displays all the objects details in the application under test. Objects can be textboxes, labels and buttons. The
and at anytime on the device. The framework shall consist of an Object Repository which is responsible for retrieving and displaying all the object details of application under test. The framework shall contain a Test Data Creator which creates test data to test the application. The test data will be saved
to
a
database
which
is
integrated
within
the
framework. The Test Executor shall run the test data against the
object details can be the object class, label, and size. B.
Test Executor Once the Test data has been created and saved in the
database, the Test Executor is then invoked to run the automated test against the mobile application with the test data. If multiple test data is to be run, the test executor reads each row of test data and test the application.
application under test.
MOBILE DEVICE
The Test Report Generator shall use the test execution details and generate a test report.
APPLICATION UNDER TEST
The Test Report Exporter shall export the test report to an
MobTAF
external data file which can be viewed on the mobile itself. The file can also be exported to and viewed on a computer. The framework shall allow for both manual testing and
Data
automated testing. Although
a
hybrid
test
automation
framework
has
numerous benefits, the framework should be at least a Data Driven Testing Framework. IV. SYSTEM ARCHITECTURE
Test Executor
Our Mobile Test Automation Framework, MobTAF, has been designed based on a data driven test automation framework. It comprises of three distinct layers namely: Abstract Layer, Data Layer, and Reporting Layer. Each layer consists of specific components that work together to perform the automated test. Fig. 2 shows the
Test Report
Test Report
Generator
Exporter
interaction between the three layers. Figure 3.
[926 ]
I
�-----------------------------
Architecture of MobTAF
2012
International Conference on Computer & Information Science (ICCIS)
A. RMS Database
C. Test Report Generator Report
MySQL database was initially considered since it allows
Generator is invoked to generate a test report based on the
running queries to perform various manipulation of data
execution details received by the Test Executor. A Test
stored in the database. It was observed that MYSQL could
Once
test
execution
is
complete,
the
Test
Report is displayed in the form of a test execution grid
not communicate with the packages of MobTAF. Finally the
specifying which test data row has passed or failed. The
RMS database has been chosen as an alternative which is
result is also saved in the database.
mostly suited for mobile applications. The RMS is both an implementation and API for persistent storage on Java ME
D. Test Report Exporter Test Reports can also be exported to an external data file. The Test Report Exporter retrieves the test report from the
devices, such as smart phones. B. Java Packages
database and exports it as an external CSV data file. The data
MobTAF has been implemented with both inbuilt and new
file is saved on the mobile device itself. The data file can be
java packages. The J2ME environment introduces a number
read either on the mobile device itself or can be exported to a
of Java packages which are ahnost exclusively placed into
desktop computer via Bluetooth.
the package javax.microedition.
Fig.
4
component
provides
a
within
detailed
MobTAF
overview works
on
how
together
each
when
an
automated test is run.
Retrieve ObjEds A
(J)jed I
J
Display objects details
01 Spy 10 0 5
p P L I
fumtest with
C A
T
data
I 0
J �I
Test
N
Exe::uted
U N
D E R
1
T
I
T...t
I
Executor
0
·1
(2)
(i)
the
MobTAF
framework.
The
details
I
�
TABLE I!.
to create d.h
en. .... r
Packa2e Name
Description
MobTAF.AppMenu
It acts as a container for the application under test in MobTAF. It contains all the classes relating to the storage of data in the database. It contains the class to create test data It contains the class to generate test reports.
I
MobTAF. TestDataCreator MobTAF.TestReportGenerator
Test d.h
I DB '''"-
0) � .
VI. TESTING AND DISCUSSIONS To the test the MobTAF framework, a simple J2ME
Test
Login application was designed consisting of two textboxes
Report retrieved
namely
T...tReport &POrEr
1
MOBTAF PACKAGES
MobTAF.Database
I
s aved
I
S
MobTAF.
saved
Test Report
T...tReport I �
I
some of the java packages used for the implementation of
Send request
IT.".,."
Test Exe::uti:m
T
V.
of
within the framework to communicate. Table II describes
1
(J)ject
Rep�
Test Data
E
Figure 4.
implementation
packages contain the classes which allow the components MobTAF
r--
New java packages have been built which are specific for the
...
I
&poredT...t Report
username
and
password.
MobTAF
correct
is displayed, else if an incorrect combination is entered, a
J
SYSTEM IMPLEMENTATION the
the
login failure page is displayed. Fig. 5 shows the main interface of the login screen as viewed on the mobile device.
Enter Usemame
Interaction between MobTAF components
Although the design of
When
username and password are entered, a login successful page
-- ----� I ,-::.,.:=--
framework is
q
Enter PaS5lo'Ord
platform-independent, a prototype has been implemented and tested on the J2ME - Java 2 Platform Micro Edition platform which is the Java Platform for Symbian based smart phones. The following development tools have used to develop the prototype based on MobTAF: •
Net Beans IDE 6.9.1 with Mobility pack
(integrated Java Wireless Toolkit) •
Bluetooth software
•
Symbian OS S60 Mobile Phone
Figure 5.
Login Application
The MobTAF framework have been tested on Nokia 663 0 and Nokia N70 mobile phones which are 3G smart phones using the Series 60 platform with the Symbian operating system. Test
scenarios
have
then
been
devised
to
test
the
framework and for each of the above test scenarios, some
[927 ]
2012
International Conference on Computer & Information Science (ICCIS)
test cases have then been devised. These test cases were
MobTAF contains a Test Data Creator which allows a
successfully tested with the J2ME prototype showing that an
user to create his own test data set to test a mobile
implementation of the MobTAF framework is really possible
application.
and a mobile test automation framework can be implemented
The test data creator can create two types of test data:
solely on a mobile device. Table III shows the test scenarios
Single Test Data Set and Multiple Test Data Set. Fig. 7
and Table IV lists all the test cases for each test scenario.
shows the two types of test data set to test the login screen.
The Object Repository within MobTAF retrieves the
The framework contains a Test Executor which runs a test
object details from the application under test and displays
data set against the application under test and displayed the
them to the user. The objects can be labels, textboxes and
results in a test execution grid.
buttons.
Fig. 8 shows the test execution grid displaying the test
Fig. 6 shows part of the object repository for the Login
results of each test run.
screen. E.g. the Object is a textbox whose label is Username
MobTAF can export the test results to a test report in the
and it belongs to the class textfield and it can contain up to
CSV format which is of small size, hence not affecting the storage constraint issue of mobile devices. The exported test
1 5 characters. TABLE Ill.
MOBTAF TEST SCENARIOS
MobTAF Test Scenarios Scenario No TSI TS2 TS3 TS4 TS5 TS6 TS7 TS8 TS9 TSIO
Test Scenario Description
Install MobTAF on mobile and launch login application Perform Manual Testing on Login Application View Object Repository Perform Automated Testing on Login Application Create new test data Run automated test with new test data Export Test Report to external data file. Run automated test with existing test data
report can be viewed on both the mobile device and a Pc. The CSV file format has been chosen as the external data file since CSV files can be read on any mobile device and desktop PCs as compared to XML format which might not be readable on certain mobile devices. Fig. 9 shows the test report for the
test viewed in CSV format.
Verify the interfaces MobTAF Verify the loading of screens and display of results TABLE IV.
Scenario
Test
No
Case
TSI
TCl TC2
MOBTAF TEST CASES
Test Case Description
Install the MobTAF lnitializer to the mobile deyice. Open the Login Application on the Mobile Deyice
TS2
Figure 7.
lITestOata Grid Create Oata Export Ooto [Run Test [)elete
TS3
Single and Multiple Test Data Set
rw
RemCNelll Bock Select
[once
Figure 8.
Test Execution Grid
,Poth (l!lI:IWIIIQ flleMlTle)
"Oate";'Usemame";"PCiSS\IIOrd";" Result" 'Jat Rug .:02: 19 GMT +Il2: !!!l1Q11";"T ester"T est"fiIIl" "Sot Rug 27 221)2: 19 GMT +02: 00 201 1";"rilmln·�d�2"Poss"
'UT estRelXlrt.csv Stotus. [oncel Exit
TSIO
Select
Conce
Figure 9.
Figure 6.
[928 ]
Object Repository
Test Report Exporter
2012
International Conference on Computer & Information Science (ICCIS)
VII. EvALUATlON
QTP and Test Complete have got lower scores in terms of
MobTAF has been evaluated against the existing test automation tools Quick Test Professional - QTP and Test Complete. Automation Tests have been run on a login application
using
Quick
Test
Professional
and
Test
Complete. Test Data was created in both tools to run the test. At the end of the test execution, test reports have been generated. The two tools have been compared with MobTAF and they have being evaluated based on the following evaluation criteria: 1.
Mobile Test Automation: The level o f mobile test
2.
Ease of Use: The user friendliness of the tool
response time. QTP is a tool which is very big in size and takes a lot of time to be installed. It is supported by several add-ins. There were performance issues in terms of loading of screens and generation of test reports. However, they have more advanced reporting features when compared to MobTAF. In general, MobTAF has obtained an overall good rating in terms of all the evaluation criteria with respect to the other tools and has proved to be an effective test automation tool for testing mobile applications directly on the mobile device.
automation each tool can perform.
VIII.
interface. 3.
Data Driven test: The ability to perform tests with multiple set of test data.
4.
Test Data management: The ability to manage and restore test data.
5.
Reporting: The ability to generate reports based on
6.
Response time: Time taken to create test data,
test execution results. execute tests. These criteria were rated based on a score of 1 to 5 where 1 means poor and 5 means excellent. Fig. 1 0 shows a histogram for the evaluation of the automation tools a ainst the different evaluation criteria. ., ,------
CONCLUSIONS AND FUTURE WORKS
The lessons learnt from this work have shown that a mobile test automation framework can be devised so that it runs entirely on the mobile device without requiring any additional device or
emulator.
A new test automation
framework MobTAF has been proposed which performs automation testing of mobile applications directly on the mobile devices. The results obtained hence reflect real life scenarios rather than being dependent on emulators residing on personal computers. MobTAF therefore brings a new dimension to testing of mobile applications as it moves the testing infrastructure to the phone, to support test situations that cannot easily be replicated with an emulator. MobTAF has demonstrated that both manual and automated tests can be performed on mobile devices. It contains an object repository which identifies and displays objects details in
4
the application under test. The Test Data creator allows user
3
can then be run against the application under test and the
to create both single and multiple test data set. The test data test results are displayed in the test execution grid. Both the test data and test results are stored in a database which is integrated within MobTAF. The test results can be exported to a test report in CSV format. The results obtained from the
()
Mobile I �,t
AlJr()li1�rioL1
Oil,e uf u,�
LJataLJnven
lest i.Jata
Hepurtmg
Ite,ponse tirnp.
test
clearly
indicate
that
frameworks
developed on the mobile device,
can
be
entirely
thereby requiring no
connection to desktop computers. In the future, research will be focused on determining the gains in completeness, efficiency and/or fault detection
Figure 10. Evaluation of tools MobTAF
obtained
a higher score
for
Mobile
Test
Automation since QTP and TestComplete are mostly suited for test automation on desktop applications. To perform mobile
test
automation,
specific
plug-ins
have
to
be
installed. Moreover, the tools require that the mobile device is connected to the PC or simply carry out the testing on emulators. Another observation has been made on the ease of use of these three tools. QTP and Test Complete interfaces are quite complex to use while MobTAF is very user friendly and simple to use. All the three tools supported data driven tests, i.e. tests
effectiveness that are provided by MobTAF in comparison to other frameworks. The current limitation of the prototype is that it runs only on a J2ME environment. Another version of MobTAF can be developed on the Android or iOS platform or the implementation can be based on standards such as W3C and BONDI so that they are platform independent and can hence be run on other mobile devices such as PDAs and handheld devices.
MobTAF has been
tested on a very simple login application. It can be enhanced to be able to test games and other complex application s with large number of non-trivial components. Testing web-based applications might also be considered.
can be run with different set of test data.
[929 ]
2012
International Conference on Computer & Information Science (ICCIS)
REFERENCES [1]
Keynote, White Paper
on "Testing Strategies and Tactics for mobile
Applications".
[2] [3]
L. Williamson, "The economics of mobile application testing", DeveloperWorks,2011. M.
Kelly,
"Choosing
a
test
automation
framework",
IBM, IBM
DeveloperWorks,2003. [4]
HP QuickTest Professional, "Test Automation for Mobile Devices",
HP QuickTest Professional software and DeviceAnywhere solutions, Data sheet,July 2009 [5]
Automated QA Corporation, "Getting Started with TestComplete
[6]
Experitest,
8.0". "Plug-in
for
TestComplete",
Available
at:
http://experitest. com/pi ug -insqtptestcompleteiplug -in-for testcompletei [Online: Accessed on 12 Apri1201l]. [7] [8] [9]
1.0.,2011. - Mobile Test Automation", Available at: http://www.confiair.com [Online: Accessed on 15 ApriI201l]. Selenium, "Selenium Documentation", Release
Confiair Inc, "RealMobileTM
S. She, S. Sivapalan and I. Warren. "Hermes: A Tool for Testing
Proceeding of the 20th Australian Software Engineering Conference,2009. [10] D. Weiss and M. Zduniak, "Automated integration tests for mobile applications in Java 2 Micro", Proceedings of the 10 th international conference on Business information systems Edition, 2007. Mobile Device Applications",
[Il] L.
Zhifimg,
L. Bin and G. Xiaopeng, "Test automation on mobile
device", Proceedings of the 5th Workshop on Automation of Software
Test, AST 2010, pp. 1-7, ACM,New York,2010.
[930 ]
