Available online at www.sciencedirect.com Available online at www.sciencedirect.com
ScienceDirect ScienceDirect
Procedia Computer Science 00 (2017) 000–000 Available online at www.sciencedirect.com Procedia Computer Science 00 (2017) 000–000
ScienceDirect
www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia
Procedia Computer Science 110 (2017) 117–124
The 14th International Conference on Mobile Systems and Pervasive Computing The 14th International Conference on Mobile Systems and Pervasive Computing (MobiSPC 2017) (MobiSPC 2017)
Automation Support for Mobile App Quality Assurance – Automation Support for Mobile App Quality Assurance – A Tool Landscape A Tool Landscape Susanne Brauna,a, *, Frank Elberzhageraa, Konstantin Hollaa Susanne Braun *, Frank Elberzhager , Konstantin Holl Fraunhofer IESE, Fraunhofer-Platz 1, 67663 Kaiserslatuern, Germany Fraunhofer IESE, Fraunhofer-Platz 1, 67663 Kaiserslatuern, Germany
Abstract Abstract Competitive pressure in app stores, as well as direct and transparent feedback of app store reviews have resulted in an Competitive pressure app stores,app as quality well asand direct transparent appreduced store reviews have resulted in an increased demand for in outstanding userand experience. Atfeedback the same of time, time-to-market, decreased increased demand for outstanding appassurance, quality and user experience. At the same time, reduced time-to-market, decreased budgets and time available for quality and careful user experience design have to be considered. In response, an budgets and time available forapp quality assurance, careful user experience tools designhas have to bearound considered. In response, an enormous market for mobile quality and userand experience measurement grown the mobile app store enormous market for mobile app quality andagile userdevelopment experience measurement tools has grown around mobileand appreadystore ecosystems. Developers following lean and approaches continuously produce newthe features ecosystems. Developers following leansettings, and agile development approaches continuously produce features andlimited. readyto-ship software increments. In those budgets for evaluation and familiarization into newnew tools are very to-ship software increments. In those settings, budgets for evaluation and familiarization into and newmore tools than are very limited. Currently there are alone more than 28 tools and frameworks for functional test automation 16 different Currently thereavailable. are aloneFor more 28 software tools anddeveloping frameworks for functional test automation and more than device clouds mostthan of the companies, it is impossible to evaluate and test all16 of different them. In device clouds For most of the developing companies, is impossible evaluate andlandscape test all offor them. In this paper, we available. present a classification in software order to help navigation throughitthe mobile apptoquality tools easier this paper,and wemore present a classification selection targeted evaluationinoforder tools.to help navigation through the mobile app quality tools landscape for easier selection and more targeted evaluation of tools. © 2017 The Authors. Published by Elsevier B.V. © 2017 The Authors. Published by Elsevier B.V. © 2017 The under Authors. Published by B.V. Program Chairs. Peer-review responsibility of Elsevier the Conference Conference Peer-review under responsibility of the Program Chairs. Peer-review under responsibility of the Conference Program Chairs. Keywords: Mobile Applications, Quality, Quality Assurance, User Experience Keywords: Mobile Applications, Quality, Quality Assurance, User Experience
* Corresponding author. Tel.: +49 631 6800-2138; fax: +49 631 6800-9 2138. E-mail address:
[email protected]. * Corresponding author. Tel.: +49 631 6800-2138; fax: +49 631 6800-9 2138. E-mail address:
[email protected]. 1877-0509 © 2017 The Authors. Published by Elsevier B.V. Peer-review©under the Conference Program 1877-0509 2017responsibility The Authors. of Published by Elsevier B.V. Chairs. Peer-review under responsibility of the Conference Program Chairs.
1877-0509 © 2017 The Authors. Published by Elsevier B.V. Peer-review under responsibility of the Conference Program Chairs. 10.1016/j.procs.2017.06.129
118 2
Susanne Braun et al. / Procedia Computer Science 110 (2017) 117–124 Susanne Braun, Frank Elberzhager, Konstantin Holl / Procedia Computer Science 00 (2017) 000–000
1. Introduction The market for mobile devices is growing rapidly and new mobile applications are being developed and shipped continuously. For instance, the number of mobile applications available in the Apple App Store rose from 800 applications in the year 2008 to 1.5 million applications in the year 201527. Until 2020, the mobile application market will grow from 70 billion US dollars in 2015 to 189 billion US dollars according to current market research forecasts28. The high pervasiveness of mobile devices and their constant presence have led to mobile application development happening in almost all domains. Of course, also quality assurance has to cope with the new needs of such app development. There exist some inherent characteristics of the mobile app development: For example, there are technological peculiarities of mobile applications, such as context awareness and limited resources, and peculiarities of their development process, such as the consideration of minimal quality assurance overhead and short development time. Missing mobile-specific testing methods and time and effort constraints are challenges that companies developing mobile application have to cope with nowadays29. A trend to address such challenges is to use automation support during app development. Tools can be used for several purposes, respectively in many development steps, for example during development, during quality assurance, or during the deployment process. In all of these areas, new tools emerged during the last years, or existing tools were further developed to better address the mobile development. However, in this “zoo of tools”, it is difficult to keep the overview. However, this is important to know in order to be able to decide whether the existing tool landscape is still up to date, and to select new tools that address new challenges. In this publication, we want to share some insights into the tool support for mobile app quality assurance. Especially for testing activities, tools are usually used in a broad way for different activities. We present a classification and give a number of selected tools, which are most common. Of course, a tremendous number of tools exist and new tools also emerge. However, our classification presents a schema how also future tools can be sorted. For practitioners, such a tool map gives a good overview whether all relevant parts in the development and quality assurance process are considered already for automation, and provide a baseline for a discussion whether new tools might be considered in the future. Besides comparisons of tools, as for example given by Gartner36 or Forrester37, we considered a much broader set of tools and tool vendors, and provide a structured classification of the tools. However, the mentioned studies might be helpful when one tool vendor needs to be analyzed in more detail. Section 2 presents our tool map, starting with an overview and followed by a description of every category together with the tools. We also discuss some lessons learned. Section 3 concludes our article. 2. The Mobile App Quality Tools Landscape Map 2.1. Overview Based on existing tool comparison such as given by Gartner or Forrester as mentioned above, a search via google, and own experience from the testing field, we identified several commercial and open source quality assurance tools focusing on mobile apps. During our research, we identified five major clusters of tool support that cover different aspects of how today’s mobile app quality assurance is mainly supported: • Functional Test Automation: Functional test automation tools are at the center of the map with more than 28 tools. The large amount of tool support underlines the importance of test automation in mobile specific app development projects, which are most often performed in an agile manner. Agile projects have always been stressed by agile pioneers like Mike Cohn and concepts like the agile testing pyramid1.
Susanne Braun et al. / Procedia Computer Science 110 (2017) 117–124 Susanne Braun, Frank Elberzhager, Konstantin Holl / Procedia Computer Science 00 (2017) 000–000
3
Device Clouds: With the ever growing zoo of highly different mobile devices with various form factors, platforms, operating system versions and hardware equipment, device clouds are gaining higher importance. One major reason is that the costs for acquiring and maintaining a device farm that covers the characteristics of the majority of devices that are currently in the wild is constantly increasing. Accordingly, we identified more than 16 different device cloud providers. • Analytics & User Experience Engineering (UX Engineering): Another tool category that is increasingly becoming popular, are tools for analytics and UX measurement. There are at least 14 different tools on the market that target mobile analytics and mobile UX engineering. • Performance & Load Testing: Performance, as well as continuous availability also significantly contributes to an outstanding UX of mobile applications. Accordingly, we identified 14 tools alone for performance and load testing. • Acceptance Testing & Test Coverage: We identified four more tools for acceptance testing and test coverage. In total, we consider currently 76 tools in our map, of which 27 tools were either open source or free to use (see Figure 2 at the end of this Section). However, some of the tool vendors targeted several of our five categories. Our tool map provides a reasonable overview on the existing tool landscape and supports quality managers and test engineers to support their testing strategies and to provide ideas for their test environments. Figure 1 depicts the structure of our mobile app quality tools map showing the five major categories with an impression of the size (i.e., reflecting the number of tools). •
Analytics
Analytics & UX Measurement
Countly appsee
Corporate / Local
Hosted
Device Clouds
Flurry
Device Clouds
Google Cloud Test Lab
Experitest SeeTestCloud Online
Mobile Labs deviceConnect
HPE Mobile Center
testdroid
TestPlant eggCloud
Firebase Test Lab
SauceLabs Automated Testing Platform
Perfecto Continuous Quality Lab
Keynote Device Anywhere
SOASTA TouchTest
Experitest SeeTestCloud Onsite
AWS Device Farm
Xamarin Test Cloud
Borland Silk Mobile Testing
SSTS pCloudy
Google Analytics
Fabric Piwik
Firebase Analytics
Crashlytics
Functional Test Automation
Selendroid
Appium
Mobile Labs Trust
Jamo Automator
ios-driver & Selenium
Espresso
Calabash
Squish
Jamo M-eux test
XCTest UI
UI Automator
Experitest SeeTestAutomation
SOASTA TouchTest
Keynote Mobile Testing
KIF
Spoon
HPE UFT (QTP) & LeanFT
SmartBear TestComplete Mobile
Ranorex
Robotium
Progress Telerik Test Studio Mobile
SauceLabs Automated Testing Platform
SSTS OpKey
Monkeyrunner
TestPlant eggPlant Mobile
IBM Rational Test Workbench
Tricentis Tosca Mobile+
UX Measurement Perfecto WindTunnel Dynatrace UX Management
Performance & Load
Acra
Test Coverage
Test Coverage & Acceptance Testing
mixpanel
iOS
Cross Platform
Test Coverage & Acceptance Testing
Android
Apple App Analytics
Functional Test Automation
(Crash) Analytics & UX Measurement
Amazon Mobile Analytics
Test Coverage Optimizer
Acceptance Testing
FitNesse
cucumber Frank
Load Testing
Profiling & Performance Testing
Performance & Load
Traceview & dmtracedump
SOASTA Mobile Performance
BlazeMeter
TestPlant eggPerformance
HPE Performance Testing & LoadRunner
Gattling
SOASTA CloudTest
Dynatrace Synthetic Monitoring
Experitest Mobile Add-On for LoadRunner
NeoLoad
Borland Silk Performer
Keynote Mobile App Monitoring
LoadUI
Progress Telerik Test Studio Mobile
Figure 1
Tool Map Structure
2.2. Functional Test Automation In the functional test automation category, we grouped all the tools that can be used to create automated tests. It includes frameworks for writing unit tests, like Espresso2 for Android or XCTest3 for iOS, but also tools for
119
120 4
Susanne Braun et al. / Procedia Computer Science 110 (2017) 117–124 Susanne Braun, Frank Elberzhager, Konstantin Holl / Procedia Computer Science 00 (2017) 000–000
the creation of automated user interface (UI) tests like Selendroid4 and Robotium5 for Android, XCTest UI6 for iOS, as well as Appium7 and Calabash8 that are cross-platform UI test automation tools. There are also tools for automated monkey tests like Monkeyrunner9 as part of this category. Next to the already mentioned Open Source and or free to use tools there are at least 16 more commercial tools, like SeeTestAutomation10 by experitest, and eggPlant Mobile11 by TestPlant. 2.3. Device Clouds Hosted device clouds free mobile app vendors from acquiring and maintaining own device farms for testing on real devices and allow developers to quickly reproduce and fix issues that only occur on specific hardware. The different hardware provided by device clouds is usually accessible via a web application that can be used to book devices, schedule automated tests or execute manual tests. The screens of the devices are captured during test execution and sometimes the platform also provides means to directly interact with the device via the browser. AWS Device Farm12 is one of the most popular hosted device clouds with currently around 190 different iOS devices and around 200 different Android and FireOS devices. Other large public device clouds are provided by Xamarin13 with currently more than 2.50014 devices for iOS and Android, the device cloud of Perfecto15 with currently more than 150 different devices and the DeviceAnywhere16 public cloud of Keynote. The cloud of Perfecto also contains a considerable amount of Windows Phone and Windows 10 mobile devices. Most of the device clouds also offer support for functional test automation either by own frameworks or by integrating with some of the popular functional test automation frameworks like Appium, Calabash, Robotium or Espresso. Xamarin for example is also the creator of Calabash. In our tool map we further distinguish whether the device farm management solution is only available in public clouds or if it can be operated on-premise, too. For example, the DeviceAnywhere cloud by Keynote and the Continuous Mobile Quality Lab by Perfecto can also be operated on-premise for managing and provisioning local device farms of the corporation. There are even device cloud solutions that can only be run on-premise like eggCloud17 of TestPlant. Some of the other device clouds are rather small, such as Experitest or SauceLabs, which offer between 30 and 60 devices. 2.4. Analytics & UX Engineering There is an increasing number of tools available for mobile app analytics, including crash analytics and tools that provide advanced UX engineering support. The most popular web analytics framework that can also be used for mobile applications is Google Analytics18. A popular Open Source web analytics platform that can also be used for mobile app analytics is Piwik19. However, with Amazon Pinpoint20 and Apple App Analytics21 there are specific solutions for mobile applications available from Amazon and Apple. Yahoo’s mobile app analytics framework Flurry is also free to use and quite widespread. Crash analytic tools like Fabric’s Crashlytics22 and Acra23 for Android provide developers with detailed crash reports and crash context information like screen shots, user ids and log messages. Appsee helps developers and UX engineers to better understand the real experience of users. By delivering video recodings of true user sessions, developers can actually see how their applications are being used and thus gain important insights on how the usability of applications can be improved. Appsee can also automatically provide video recordings of any crashed sessions. Further, it captures touch gestures and calculates touch heatmaps out of that which can be used to understand how intuitive the interaction concept of the application is.
Susanne Braun et al. / Procedia Computer Science 110 (2017) 117–124 Susanne Braun, Frank Elberzhager, Konstantin Holl / Procedia Computer Science 00 (2017) 000–000
5
Next to Appsee, Fabric is one of the top tools for mobile app analytics and mobile user experience optimization. It originally belonged to Twitter and has been acquired by Google in January 2017. Some of the most popular mobile app quality tools like Crashlytics, Answers24 and fastlane25 are part of Fabric, others like Appsee and Optimizely26 are already nicely integrated. 2.5. Performance & Load Testing Performance tools provide several statistics about, for instance, CPU workload, memory usage, network traffic, or battery status. Many tools also provide some kind of dashboard view to get the information easily. The SOASTA mobile performance tool31 provides several of such analysis during development and operations with different ways of controlling the tools, such as command line or APIs. Notification mechanisms help in finding urgent problems fast. Other tools, such as LoadUI30, only consider APIs. Many protocol technologies are supported, such as REST, SOAP, JDBC, HTTP(S) and HTML. Different kinds of performance testing can be addressed, such as increasing load up to stress testing. The tool suite by TestPlant also contains a performance tool32. One main feature is the creation of test cases in a smart way without using too technical scripts, but also more than just capture and replay. Most of the mentioned tools are commercial ones, though many of them can at least be tested for free. 2.6. Test Coverage & Acceptance Testing The final category is test coverage and acceptance testing. Here, only a few tools were found. Most of them are larger tools that provide further test functionality. The digital test optimized from Perfecto33, which also provides functional testing support, answers the question whether one is testing on the right platforms so that relevant platforms or operations systems are considered in the testing strategy. Some further tools, such as Fitnesse34 or cucumber35 focus on system and acceptance test level. Cucumber considers thereby behavioral-driven development. All tools from this category are open-source, and are further developed so that support can be expected, and often plug-ins are available that further enhance these tools. 2.7. Summary and Discussion Figure 2 presents an overview of all identified tools. The “stars” indicate the tool being open source or free of charge. One major observation we made during our tool analysis is that many tools exist that support the mobile app development and its quality assurance, which is good news. However, finding the tools that fit best into one’s own context is another question. For this, the needs have to be carefully derived (e.g., what technologies are we using, how much do we want to invest, what kind of testing tool do we need, etc.) and tools from the respective categories analyzed. Out tool map can provide a starting point for this. Of course, we do not claim that our tool map is complete, but it provides a comprehensive overview of many relevant tool vendors and their tools. Furthermore, tools can easily be added into our categorization schema. From our perspective, it is not surprising that functional test tools are most common, as usually developers and testers first want to ensure that the mobile app works in the intended way. We concentrated on iOS and Android as the most common operating systems today, and identified also several cross-platform tools. Regarding non-functional requirements, performance and load test tool is another big area. Performance is one of those qualities that the users experience directly when issues occur. Consequently, mobile app developing companies have to ensure that the apps do not have performance issues, and also have to monitor this during operation so that they can react.
121
Susanne Braun et al. / Procedia Computer Science 110 (2017) 117–124 Susanne Braun, Frank Elberzhager, Konstantin Holl / Procedia Computer Science 00 (2017) 000–000
122 6
Due to the explosion of devices, operations systems, and different versions of operation systems, it is for many companies not economical to build up those combinations in an own test environment. Several vendors therefore offer a device cloud that can be used by app developers. They are often configurable and can be adapted to the concrete needs (e.g., only a specific smart phone or OS version). The maturity and prices of those device clouds differ heavily. Another emerging field are analytic tools that provide several insights into the current quality of an app, and which provides much data so that users can be analyzed in order to learn for the quality and also new requirements of the app. Finally, some tools provide higher-level test functionality, such as acceptance test support, or give information what parts should be more tested by analyzing a certain coverage. We expect more tools to emerge in the coming years, especially more open source tools, which might be highly specialized to certain test tasks. This might also lead to an adaption of our tool categories. Furthermore, all big vendors offer support for mobile testing nowadays, and we expect that the services and tools they offer will continue to cover more test functionalities so that app developers do not have to consider different tool vendors. What we already found often is a combination of functional test automation, device cloud and analytics, for example, Firebase, Perfecto, Dynatrace, or Xamarin are such vendors. Regarding testing in general and the question how to balance between manual testing and automated testing, the clear trend is towards a stronger automation. Trends like continuous development, deployment, and delivery up to DevOps build upon a strong automation, which is also true for quality assurance, and our analysis and overview of tools show that many tools exist. However, while there might be a shift from manual testing to more automated testing, a complete substitute of manual testing by automated testing is also unrealistic nowadays. Analytics
Countly Amazon Pinpoint
Corporate / Local
Hosted
Device Clouds
Flurry
Google Cloud Test Lab
Experitest SeeTestCloud Online
Mobile Labs deviceConnect
HPE Mobile Center
testdroid
TestPlant eggCloud
Firebase Test Lab
SauceLabs Automated Testing Platform
Perfecto Continuous Quality Lab
Keynote Device Anywhere
SOASTA TouchTest
Experitest SeeTestCloud Onsite
AWS Device Farm
Xamarin Test Cloud
Borland Silk Mobile Testing
SSTS pCloudy
Apple App Analytics
Piwik Firebase Analytics Crashlytics Acra
Functional Test Automation
Fabric
iOS
Cross Platform Appium
Selendroid
Mobile Labs Trust
Jamo Automator
ios-driver & Selenium
Espresso
Calabash
Squish
Jamo M-eux test
XCTest UI
UI Automator
Experitest SeeTestAutomation
SOASTA TouchTest
Keynote Mobile Testing
XCTest
Spoon
HPE UFT (QTP) & LeanFT
SmartBear TestComplete Mobile
Ranorex
KIF
Robotium
Progress Telerik Test Studio Mobile
SauceLabs Automated Testing Platform
SSTS OpKey
Monkeyrunner
TestPlant eggPlant Mobile
IBM Rational Test Workbench
Tricentis Tosca Mobile+
Test Coverage & Acceptance Testing
Android
mixpanel
Test Coverage Test Coverage Optimizer
Acceptance Testing
FitNesse cucumber Frank
User Experience
appsee Perfecto WindTunnel Dynatrace UX Management
Performance & Load
(Crash) Analytics & UX Engineering
Google Analytics
Load Testing
Profiling & Performance Testing Traceview & dmtracedump
SOASTA Mobile Performance
Dynatrace Synthetic Monitoring
Experitest Mobile Add-On for LoadRunner
Figure 2
BlazeMeter
TestPlant eggPerformance
Keynote Mobile App Monitoring
NeoLoad
Borland Silk Performer
HPE Performance Testing & LoadRunner
Gattling LoadUI
Tools in the tool map, open source / free of use tools marked with a star
SOASTA CloudTest Progress Telerik Test Studio Mobile
Susanne Braun et al. / Procedia Computer Science 110 (2017) 117–124 Susanne Braun, Frank Elberzhager, Konstantin Holl / Procedia Computer Science 00 (2017) 000–000
7
3. Conclusion In this publication, we presented a classification of mobile app quality assurance tools, which consists of basically five categories: Functional testing, performance and load tests, test coverage and acceptance testing, device clouds, and analytics. For every category, we identified a set of representative existing tools, and showed the distribution of commercial and open source / free to use tools. Overall, we considered about 80 tools. Such a classification and list of tools can serve as a basis to evaluate the own quality assurance strategy, especially the test environment. In case no tool is used with regard to a specific tool category, one can further analyze based on our tool map whether to include a tool of the respective tool category, which can lead to an improvement of the own quality assurance environment. Acknowledgments The research described in this paper was performed in the project Opti4Apps (grant no. 02K14A182) of the German Federal Ministry of Education and Research (BMBF). References 1. M. Cohn, Succeeding with agile: software development using Scrum, Pearson Education, 2010. 2. "Espresso," [Online]. Available: https://google.github.io/android-testing-support-library/docs/espresso/index.html. [Accessed 22 March 2017]. 3. "XCTest," [Online]. Available: https://developer.apple.com/reference/xctest. [Accessed 22 March 2017]. 4. "Selendroid," [Online]. Available: http://selendroid.io/setup.html. [Accessed 22 March 2017]. 5. "Robotium," [Online]. Available: https://github.com/RobotiumTech/robotium. [Accessed 22 March 2017]. 6. "XCTest UI," [Online]. Available: https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/09-ui_testing.html. [Accessed 22 March 2017]. 7. "Appium," [Online]. Available: http://appium.io/. [Accessed 22 March 2017]. 8. "Calabash," [Online]. Available: http://calaba.sh/. [Accessed 22 March 2017]. 9. "Monkeyrunner," [Online]. Available: https://developer.android.com/studio/test/monkeyrunner/index.html. [Accessed 22 March 2017]. 10. "SeeTestAutomation," [Online]. Available: https://experitest.com/test-automation-tool-for-mobile-testing-continuous-integration. [Accessed 22 March 2017]. 11. "eggPlant Mobile," [Online]. Available: http://www.testplant.com/eggplant/testing-tools/eggplant-mobile-eggon/. [Accessed 22 March 2017]. 12. "AWS device farm," [Online]. Available: https://aws.amazon.com/de/device-farm/. [Accessed 22 March 2017]. 13. "Xamarin Test Cloud," [Online]. Available: https://developer.xamarin.com/testcloud/. 14. "Xamarin Test Cloud Device List," [Online]. Available: https://testcloud.xamarin.com/devices. [Accessed 22 March 2017]. 15. "Perfecto Public Device Cloud," [Online]. Available: https://www.perfectomobile.com/hybrid-cloud/public-shared-cloud. [Accessed 22 March 2017]. 16. "DeviceAnywhere Cloud," [Online]. Available: http://www.keynote.com/solutions/testing/mobile-testing. [Accessed 22 March 2017]. 17. "TestPlant eggCloud," [Online]. Available: http://www.testplant.com/eggplant/testing-tools/eggcloud/. [Accessed 22 March 2017]. 18. "Google Analytics," [Online]. Available: https://www.google.com/analytics/analytics/#?modal_active=none. [Accessed 22 March 2017]. 19. "Piwik," [Online]. Available: https://piwik.org/what-is-piwik/. [Accessed 22 March 2017]. 20. "Amazon Pinpoint," [Online]. Available: https://aws.amazon.com/pinpoint/. [Accessed 22 March 2017].
123
124 8
Susanne Braun et al. / Procedia Computer Science 110 (2017) 117–124 Susanne Braun, Frank Elberzhager, Konstantin Holl / Procedia Computer Science 00 (2017) 000–000
21. "Apple App Analytics," [Online]. Available: http://help.apple.com/itc/appanalytics/. [Accessed 22 March 2017]. 22. "Crashlytics," [Online]. Available: https://try.crashlytics.com/. [Accessed 22 March 2017]. 23. "Acra," [Online]. Available: https://github.com/ACRA/acra. [Accessed 22 March 2017]. 24. "Fabric Answers," [Online]. Available: https://answers.io/. [Accessed 22 March 2017]. 25. "fastlane," [Online]. Available: https://fabric.io/features/distribution. [Accessed 22 March 2017]. 26. "Optimizely," [Online]. Available: https://www.optimizely.com/de/. [Accessed 22 March 2017]. 27. Statista, Number of available apps in the Apple App Store from July 2008 to June 2015, The Statistics Portal. http://www.statista.com/statistics/263795/number-of-available-apps-in-the-apple-app-store, 2015. [accessed 01/08/2016] 28. Statista, Worldwide mobile app revenues in 2015, 2016 and 2020. The Statistics Portal. https://www.statista.com/statistics/269025/worldwide-mobile-app-revenue-forecast/, 2016. [accessed 02/01/2017] 29. Jeremiah, J., Rajani, R., Natarajan, S., Digital Transformation: Disrupting business models for a better customer experience. World Quality Report 2016-17, Eighth Edition, Capgemini, HP, Sogeti, 2016. 30. “LoadUI”, [Online]. https://www.loadui.org/ [Accessed 22 March 2017]. 31. “SOASTA Mobile Performance”, [Online]. https://www.soasta.com/performance-platform/, [Accessed 22 March 2017]. 32. “TestPlant eggPerformance”, [Online]. https://www.testplant.com/eggplant/testing-tools/, [Accessed 22 March 2017]. 33. “Test Coverage Optimizer”, [Online]. https://www.perfectomobile.com/solutions/digital-test-coverage, [Accessed 22 March 2017]. 34. “Fitnesse”, [Online]. http://www.fitnesse.org/ [Accessed 22 March 2017]. 35. “cucumber”, [Online]. https://cucumber.io/, [Accessed 22 March 2017]. 36. M. Sobejana, A. Leow, Market Guide for Mobile App Test Automation Tools, Gartner Report, 2016 37. J. M. Wargo, D. Lo Giudice, The Forrester Wave™: Mobile Front-End Test Automation Tools, 2016
