2015 IEEE Symposium on Service-Oriented System Engineering
Testing location-based function services for mobile applications Oum-EI-Kheir Aktouf Grenoble University LCIS Lab. France
Tao Zhang Northwest Polytechnical University, China
[email protected]
Oum-El-Kheir.Aktouf @grenoble-inp.fr
Tadahiro Uehara
San Jose State University San Jose, USA
[email protected]
Software Innovation Laboratories Fujitsu Laboratories LTD. Japan uehara.tadahiro @jp.fujitsu.com
or GPS technologies. These location-based mobile services bring new issues and challenges to mobile app test engineers due to the following reasons: - Lack of existing research work addressing locationbased test models, adequate validation coverage criteria, as well as testing methods.
Abstract— Mobile testing is becoming an urgent and important research subject due to the explosive increase of mobile app downloads and mobile users. Now, location-based function services in mobile apps not only enhance mobile user experience, but also bring new challenges and issues in software testing. This paper focuses on location-based testing issues for mobile apps, and proposes a new testing model and method to address these needs. An application example for a popular location-based mobile app (Waze) and its case study results are reported based on SJSU Engineering Dpt. students’ mobile testing project experience.
-
Index Terms— Mobile app, Mobile computing, Location-based test model, Software testing.
Lack of existing test automation tools supporting location-based mobile app function testing and QoS requirement validation.
This paper focuses on location-based mobile app testing issues and challenges by proposing a new test model and validation method. One location-based mobile app [www.waze.com] is used to demonstrate the usage of the proposed approach. Our case study results are included to show its effectiveness and value in validating location-based functional services for mobile apps.
I. INTRODUCTION The recent explosive increase of mobile app downloads and mobile users brings very strong business opportunities and demands on mobile testing and automation. According to ABI Research, it is predicted that revenue from such tools, which today are primarily manual but are becomingly increasingly automated, will rise from $200 million in 2012 to $800 million by 2015 https://www.abiresearch.com/press/ 200-million-mobile-application-testing-market-boos). As indicated in [1] [2], most research on mobile application testing has focused on solutions to special technical problems in the following areas: - White-box testing of mobile apps, - Black-box and GUI testing of mobile apps, - Wireless network testing, - Validation of mobile app quality-of-service (QoS) requirements, - Mobile usability testing, and - Mobile test automation frameworks.
The structure of this paper is organized as follows. Section 2 discusses the basic concepts of location-based service testing, and proposes a new location-based service test model for mobile apps. Section 3 shows our application example and case study on mobile app (Waze). Section 4 reviews the related work, and Section 5 presents our final conclusion and discusses the future research. II. LOCATION-BASED SERVICE TESTING AND TEST MODEL Location-based testing approaches are rather new and motivated by the strong increase in the use of mobile applications with location-aware capabilities. This section introduces our testing model and explains how this model can help in enhancing the testing procedure of locationbased mobile apps.
However, many of today’s mobile apps provide users with dynamic location-based functional services using map-based
*Corresponding author 978-1-4799-8356-8/15 $31.00 © 2015 IEEE DOI 10.1109/SOSE.2015.17
Jerry Gao*
308
while generating test cases or deriving test strategies. The emergence of mobile applications has added real complexity to the testing processes and requires the adoption of new models or at least a smart adaptation of existing models to take into account mobility. One specific concern of mobile applications is the location features and corresponding services.
A. Location-Based Software Service Testing Location-based Services are commonly defined as “a general class of computer program-level services that use location data to control features” [www.wikipedia.org] or as “services offered through a mobile phone that take into account the device’s geographical location” [www.technopedia.com].
Our issue concerns the testing of mobile app component services given a user/app location. Consequently, we will work on a component-based model endowed with location features. Indeed, the proposed model aims to cover the location aspects of a given mobile app. To this end, this model introduces concepts like GeoObjects, Active Mobile Objects and Passive Mobile Objects that are basic mobile app component types as it is explained below.
Following these definitions, we can define locationbased testing of mobile apps as being “a method for assessing the functionality of alocation-based mobile software”. Mobile apps offer many services to their users. Function services of location-based mobile apps largely depend on the current user location and context. Due to the mobility feature, many context data have to be processed in real time as context may change rapidly. Indeed, as said in the introduction, the number of mobile app users is increasing dramatically and it becomes urgent to provide reliable function services in order to keep user confidence. There exist large variety of mobile apps, and the ones using location-based services are often concerned with real-time notifications and context changes. Location-based testing is mainly used to avoid faulty notification, lack of notification and to ensure service availability and correctness with regards to changing user context.
Furthermore, testing the location features of a locationbased app requires a strategy that covers the important aspects of the service. These aspects and their descriptions are given in Table 1 below. To take into account different types of objects and services, we propose a graph-based test model. This graph model, called G, is as follows: G=(N, E, S, Lc), where -
This is motivated by the inherent complexity of locationbased mobile apps. This complexity is due to real-time changing context data, huge quantities of processed data, multiplicity of possible scenarios that cannot be all provided, etc. Consequently, testing of location-based mobile apps is a very costly process that needs efficient approaches to provide effective application reliability/testing cost tradeoff.
-
B. Location-Based Software Testing Model The basic idea underlying location-based testing is to consider the user actual location to test aspects of the mobile app. In this work, aspects of interest concern function services provided or requested by different mobile app components. When it comes to modelling mobile systems or application for testing purposes, most used approaches rely on three basic models: - State-based models, mainly used to better understand the behavior of an application, - Semantic spanning trees, for testing and analysis of mobile environments (underlying OS, mobile platform, Internet connectivity, browser, etc.) especially in configurable component-based applications. - Component-based models, which help in understanding the individual behavior of application components as well as interactions among them.
-
-
Lc is the location context map where objects of interest are displayed to the app user. N is the set of nodes in Lc. N is composed of three different sets of nodes, namely N=NF∪NAM∪NPM, where NF is the set of fix (geo) nodes (such as gas stations, schools, etc.), NAM is the set of active mobile nodes (i.e. app users) and NPM is the set of passive mobile nodes (these are objects subject to reports from active mobile nodes, such as accident, traffic jam, road closure, etc.). We consider M as being the total number of nodes in N. E ⊆ N×N is a set of couples (ni, nj) where ni, nj ∈ N, i,j=1,..,M. Each couple (or edge) (ni, nj) in E indicates a service execution from nj to ni (the concept of service will be refined below). S is the service set, S={S1, ..., SM}. Each Si is associated to a node ni, i=1..M ; Each Si={Si1, Si2, ... Sixi}, where xi is the number of functions associated to node ni, i=1..M.
This testing model allows partitioning the map view context into relevant components regarding the testing process. Hence, testing activities may focus on key component functions in order to validate the mobile app functionalities. This allows cost reduction and better function test coverage, as it will be shown in our case study.
Using such models supports application analysis and helps to focus on the toughest and most important features
309
TABLE I. BASIC COMPONENT TYPES IN A MOBILE APP. Aspect GeoObjects Active Objects Passive Objects Service Functions Map Navigation GPS Location Search and Other functions
D. Black-box testing Although many existing mobile apps are open source, our work concerns testing of valid functioning of services associated to app nodes. This is why, we have given priority to functional test methods that seems to be more appropriate for our study. Classic Black-box testing approaches such as Equivalence Partitioning Method and Decision-Table Based Method have been used in our work to obtain experimental results (cf. Section III)
Description Permanent objects on the map (Gas Stations, POI) Users of the app Objects created by the user (Police, accident, jam) The functions associated with the GeoObjects The digital map provided by the app server Capability of the app to guide user to destination Capability of the app to use the built-in GPS Search function of the app to find locations.
Mobile Active Users
C. Node relationships What we mean by relationships here are different possible interactions between the app components. In general, possible interactions are related to service functions that are available among nodes. For example, we may have these interactions: - NAM Æ NF: an active mobile node detects a fix node. - NAM Æ NPM: an active mobile node detects a passive mobile node. - NAM Æ NAM: an active mobile node detects an active mobile node.
Middle school
Mobile Passive User
Park
Police Dpt.
Gas station
Car wash
a. Example of a simple context map
To illustrate how our graph-model may be derived from a context map, let us consider the following example of a simple context map on Fig. 1.a., showing 5 geo objects (circled in red), 2 active mobile objects (circled in blue) and 1 passive mobile object (circled in green). Fig.1.b. shows corresponding graph model that indicates component relationships, i.e. function services among nodes. These relationships are temporal as they depend on the occurrence time of the corresponding operations. Furthermore, in a mobile app, nodes may appear and disappear. This means that the graph model is dynamic.
b. A corresponding graph-model showing actual relationships between components
For studying dynamic graphs, one possible approach is an event-based approach, which consists of dividing the observation time into window frames and analyzing events occurring within each frame to determine the system state (the graph state). So, a dynamic graph is a discrete sequence of static graphs (then classic approaches may be applied ...). This issue is currently investigated by our research team and will be subject of a future paper.
Fig. 1. Example of a context map and its corresponding graph model E. Coverage criteria and Test Procedure The objective of the test procedure is to test core service functions of different application component types. Once these service functions have been determined, we designed relevant test cases using black-box testing as said above and determined fault coverage. Results presented in this paper are obtained using manual testing. Testing automation issues are under studies and will be published in future work.
The graph G is then refined to represent a snapshot of Lc at a given time t. Let’s call the refined graph Gt. Then we have Gt=(N, St, Et) where St is the set of services issued at time t by nodes of G (set N) and Et is the set of services (relationships) joining nodes in St. Such a refinement makes it possible to consider context events implicitly in our test procedure.
310
F. Sample examples This section depicts the application of the proposed testing model to some sample examples of the Waze mobile app that will be described more thoroughly in section III.
state that users can view the geo objects and wazers at the same time.
1) Sample example One This example corresponds to the following context map (Fig. 2).
Fig. 3. Sample example 2 of Waze app. 3) Sample example Three
Fig. 2. Sample example 1 of Waze app. For this location context, we are interested in mobile active nodes (users or wazers). There are 4 objects of this type. So the set N1 is: N1 = { Waze Grown Up1, Waze Grown Up1, Waze Warrior, Waze Royalty} Each of these nodes is capable of emitting and asking for the following services indicated in the set S1: S1 = {Points, Rank, Joined, Speed, Map chat, Message} Fig. 4. Sample example 3 of Waze app.
2) Sample example Two This example corresponds to the context map of Fig. 3, that contains only Geo objects.
For this example (Fig. 4), we have the following sets of nodes and relations. Set UAM of Active Mobile Objects: UAM = {Wazers, Guest Wazers}
For this map context, set N2 of nodes and S2 of services are as follows: N2 = {Gas Station, Buildings, Streets, Freeways, Parks, Pools, Schools} S2 = {preview location, send location, add to favorites, set as start point, show on map} If we consider simultaneously both sample example 1 and sample example 2, we can say that possible relations between set N1 and set N2 are: R12 = (N1, N2) R12 = relations between Mobile Active Objects and Geo objects. These relations between wazers and geo objects
Set S of services: S = {Map Chat, Message, Points, Rank, Joined, Speed} For this example, it is possible to determine set R of possible relations that may be established: 1. between Active Mobile Objects and Geo Objects, 2. or between Active Mobile Objects and Wazer status.
311
A. Waze Mobile App The Waze Mobile App is a communatory GPS. Its users share real time traffic information and alerts on events on the road (accident, police control, etc.). Fig. 6 below shows main components of the ecosystem of Waze app.
4) Sample example Four For this example (Fig. 5), we have the following sets of nodes and relations. Set UpM of Passive Mobile Objects: UpM = {Police, Vehicle stopped, Accident, Rain/Snow, Traffic Jam, Construction, Road Closure, Road Goodies, Camera} Set S of services: S = {Like, Comment, Distance, Posted by, Posted at}
Fig. 6. The ecosystem of Waze app A. Building the Location-based Test Model for Waze Waze is a social mapping app that features real-time traffic data provided by users. Using Waze, a user can report Police, Gas stations, accidents, traffic jams, etc. Obviously, Waze provides many location-based services for users. As shown in Fig. 7, we recall that the scope of mobile testing includes the following nine types of testing activities and focuses: Geo objects, Mobile active objects, Mobile passive objects, Location-based services, other services, Navigation, Map, and GPS. More details are described in Table 1 above. Fig. 5. Sample example 4 of Waze app.
In this case, relations to be determined are: - Relations between Passive Objects and Geo Objects, - and Relation between Passive Objects and Active Mobile Objects. To make a proof of concept of the proposed testing model and approach for testing location-based mobile apps, we asked two groups of engineers to test the mobile app Waze. One group used the proposed approach, and the other one used a classic testing approach. Then we compared the test cases and test results from both groups. More details are described in the following section.
Fig. 7. The test Scope for Waze B. Designing Test Cases for Location-based Service of Waze Using the proposed approach, we first identify the main location contexts for Waze, and then find out all Nodes, Services and Relations in every context as shown above in proposed sample examples. Then we design test cases and evaluate the obtained testing complexity. Fig. 8 shows the considered case study location context. We determine nodes, services, and relations for this location context as follows.
III. CASE STUDY This section reports one case study for a very popular mobile location app. It concerns the Waze App which is a community-based GPS that besides allowing the users to find their routes, also allows them to exchange context information in real time.
The set of nodes:
312
The effectiveness of any testing approach is considered as the ability to detect faults using this approach. Thus, we compared the number of faults detected during testing by both groups. Obtained results are depicted in Table IV.
N = {Police, Vehicle stopped, Accident, Rain/Snow, Traffic Jam, Construction, Road Closure, Road Goodies, Camera} The set of services: S = {Like, Comment, Distance, Posted by, Posted at}
For clarity purpose, we give here one example of location-related fault detection, which has been performed using this approach. This case is shown on Fig. 9. In this test case, we wanted Waze to provide a navigation service from a remote location based on the internal context map. However, in this case, Waze failed to provide a route from the remote location.
An example set of relations: R = {A user report one police, A user report one traffic jam} The testing complexity for this location context is given in Table II. TABLE II. TEST COMPLEXITY FOR THE LOCATION CONTEXT No of Node Types
No of Nodes
No of Services
No of Relations
2
10
5
2
The experimental results show that the proposed method will help engineers designing test cases for testing locationservices of mobile apps, which is more effective to detect location-related faults.
Fig. 9. An example of location-related fault IV. RELATED WORK Location-based services are rather new in the landscape of computing applications, for a very good reason as general public equipment that allow to use GPS information have been available only recently. Consequently, development of location-based application has attracted a great attention from researchers and developers such in [3] [4].
Fig. 8. Sample location context for Waze case study
We designed 290 test cases for Waze, which included 82 location-based test cases in 9 different location contexts.
However, these new applications raise serious security and privacy concerns as personal information, i.e. the current user position for example, are conveyed. Consequently, many researchers have considered the security and privacy issues. New solutions have been presented, for example in [5] which presents a service architecture for solving privacy problem and in [6] which studied the security problem regarding the location cheating attacks. Like these design and security issues, location-based testing is a hot topic that is currently intensively investigated. For example, the work in [7] addressed the lack of well-defined test models to handle distinct needs in mobile testing by developing semantic trees to modeling and analysis of mobile test environment (mobile platforms, web browsers, mobile technologies, etc.). The work of Ke Zhai et al. [8] has studied the specific problem of regression testing in mobile applications, knowing that is it hard to find the same service
C. Analyzing and Comparing Testing Results Table III below shows results obtained by both testing groups of engineers, the one using the proposed model and approach (Group 1), and the one using their own testing expertise (Group 2). TABLE III. THE TEST CASES FOR LOCATION-BASED SERVICES OF WAZE Waze Test Target Geo Objects Mobile Passive Objects Mobile active Objects Map Navigation GPS Geo Services Location -based Search Services All test case
Number of Test Cases Group 1 Group 2 8 5 18 8 14 10 4 5 7 6 3 4 12 8 16 9 82
55
313
compositions to replay the same scenario for regression testing. The authors propose an approach based on black-box test prioritization techniques that integrate service selection to avoid reusing faulty services. This work also proposes interesting location-based metrics to evaluate testing performances. Other work presented in [9] handles the testing problem by developing test automation techniques and tools.
V. CONCLUSION AND FUTURE WORK With the fast increase of the number of mobile app downloads and deployments, mobile testing for locationbased function services become an urgent and challenge task for engineers due to the lack of cost-effective testing methods and tools. This paper addresses location- based mobile testing issues and needs and propose a new test model, test coverage criteria. Meanwhile, our application example on Waze mobile app is reported. The related case study results demonstrated that this is a promising and useful approach. For the future research direction, we plan to start the research and development of new test automation methods and tools support location-based function service testing for mobile apps.
Mobile apps are running on devices that are equipped with a large variety of sensors. Consequently, they can be made aware of many context events, such as user events, external environment events, device related events, etc. [10]. This feature has been used to develop event-based testing approaches that take into account relevant event-based scenarios [10]. This is an interesting feature to consider while testing mobile apps.
Table IV. Test cases and test results for Waze app case study Coverage Teat Case Set
Total Test Case
LocationRelated Test Case
Group 1
290
82
6/6
46/46
8/8
18
7
Group 2
226
55
6/6
27/46
3/8
15
3
LocationRelated Object
Acknowledgment
LocationBased Service
Relation Coverage
All Faults
LocationRelated Faults
[6] W. He, X. Liu and M. Ren, "Location Cheating: A Security Challenge to Location-based Social Network Services," in 31st International Conference on Distributed Computing Systems, Minneapolis, Minnesota, USA, 2011. [7] C. Tao and J. Gao, "Modeling Mobile Application Test Platform and Environment: Testing Criteria and Complexity Analysis," in JAMAICA 2014 Proceedings of the 2014 Workshop on Joining AcadeMiA and Industry Contributions to Test Automation and Model-Based Testing , San José, USA, 2014. [8] K. Zhai, B. Jiang, W. K. Chan and T. H. Tse, "Taking Advantage of Service Selection: A Study on the Testing of Location-Based Web Services through Test Case Prioritization," in IEEE International Conference on Web Services, Miami, Florida, USA, 2010. [9] S. She, S. Sivapalan and I. Warren, "Hermes: A Tool for Testing Mobile Device Applications," in 2009 Australian Software Engineering Conference, Queensland, australia, 2009. [10] D. Amalfitano, A. R. Fasolino, P. Tramontana and N. Amatucci, "Considering Context Events in Event-Based Testing of Mobile Applications," in IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops, Luxembourg, 2013.
This research project was funded by Fujitsu Research Lab. on Mobile SaaS Testing from 2013-2015. The authors wish to thank students of “Testing Class” at Eng. Dpt. of SJSU for their experimental work on the model. References [1] J. Gao, X. Bai, W.-T. Tsai and T. Uehara, "Mobile Application Testing: A Tutorial," IEEE Computer, vol. 47, no. 2, pp. 4655, February 2014. [2] J. Gao, W.-T. Tsai, R. Paul, X. Bai and T. Uehara, "Mobile Testing-as-a-Service (MTaaS) - Infrastructures, Issues, Solutions and Needs," in IEEE 15th International Symposium on High-Assurance Systems Engineering, Miami, Florida, USA, 2014. [3] A. Krevl and M. Ciglaric, "A Framework for Developing Distributed Location Based Applications," in Parallel and Distributed Processing Symposium, 2006 (IPDPS 2006), Rhodes Island, 2006. [4] S. Bjørnestad, B. Tessem and L. Nyre, "Design and Evaluation of a Location-based Mobile," in 4th IFIP International Conference on New Technologies, Mobility and Security (NTMS), 2011. [5] O. Jorns, O. Jung and G. Quirchmayr, "A Platform for the Development of Location-Based Mobile Applications with Privacy Protection," in 3rd International Conference on Communication Systems Software and Middleware and Workshops, 2008. COMSWARE 2008, Bangalore, India, 2008.
314
