Testing context-aware software systems: Unchain the

This is the authors accepted version of the manuscript. There can be changes to the final version. Please cite: Santiago Matalonga and Guilherme H. Travassos. 2017. Testing context-aware software systems: Unchain the context, set it free!. In Proceedings of the 31st Brazilian Symposium on Software Engineering (SBES'17). ACM, New York, NY, USA, 250-254. DOI: https://doi.org/10.1145/3131151.3131190

Testing context-aware software systems: Unchain the context, set it free! Santiago Matalonga Facultad de Ingeniería Universidad ORT Uruguay Montevideo, Uruguay [email protected] Abstract— Background: In the era of digitalization, context awareness has become more important to allow software systems adaptation. Therefore, the quality assurance of such systems must consider the variation of context. However, there is a lack of software quality technologies doing so, which can increase the risk of failure of these systems. Objective/Aim: To evidence recent advances regarding the testing of context-aware software systems (CASS), focusing on the abstraction that context should freely vary during test execution as it does in production environments. Method: Based on knowledge acquired with quasi-Systematic Literature Reviews, we evaluate current testing approaches and discuss benefits and limitations of applying the observation above about CASS. Results: It was not possible to observe any software testing technology supporting the unconstrained variation of context during testing a CASS. The practitioners, meanwhile, can use three main evidence-based strategies to test CASS: to assure functional correctness before turning to testing contextaware requirements; to design test cases to target context variables and to take advantage of automatic testing tools as much as possible. Conclusions: To allow context to vary during testing freely, seems plausible, It provides a new frame of thought to enable the design of novel technologies to improve our capacity of testing CASS.

CCS Concepts Software and its engineering  Software creation and management  Software verification and validation

Keywords. Software Testing. Context-aware, Context-aware software testing.

Introduction Context-aware software systems (CASS) are a subtype of ubiquitous systems. They use context information to aid users in achieving their tasks. This capacity of sensing the application context has found useful in several application domains. Platform providers like Google are taking advantage of the myriad of sensors available in smartphones to provide contextual APIs to software developers (https://developers.google.com/awareness/). In turn, application providers are using these devices in domains such

Guilherme H. Travassos Universidade Federal do Rio de Janeiro PESC/COPPE Rio de Janeiro, Brazil [email protected] as smart traffic routing like Waze, health care, athletes training and performance enhancement, among others. The IoT and manufacturing industry are also adopting context-aware software systems to optimize production and reduce costs [1]. As context-aware applications become pervasive and more ambitious, so does the cost of their failures. Imagine the remote-monitoring health care system described in [2], dispatching an ambulance with a doctor for kilometers across India for mistakenly interpreting a push-up for fainting. Likewise, imagine a CASS designed for optimizing a horizontally-integrated supply chain as the one envisioned in [3]. In this system, the factory capacity and inventory are managed by software, signaling work orders downstream, misjudging any variable can result in costly inefficiencies either for excessive inventory or additional waiting times. These examples show the importance of assuring the quality of CASS. In particular, in this work we are interested in the current practices and technologies available for testing CASS, which is particularly challenging. Traditionally, designing a suitable set of test cases to verify a software system requires that the test designer makes tradeoff decisions between the test suite comprehensiveness and its execution effort. In CASS, context becomes a factor and the test designer capacity to plan a possible test suit worsens. The definition of context is elusive [4][5]. To the best of our knowledge, the most used context definition comes from Dey and Abow [6]: “Context is any information that can be used to characterize the situation of entities that are considered relevant to the interaction between user and application.” At design time, context can be abstracted to context variables and inputs from sensors will be regarded the representation of such variables. At run-time, context can take many forms, and even be represented by values not foreseen for the application [7], [8]. Current testing approaches for CASS primarily rely on two strategies. Either they exploit extensive computation resources to simulate possible contextual input, or they artificially constrain the context values to make testing decisions manageable. However, both strategies impose artificial limitations and threat the effectiveness of testing, since these conditions are not present in the intended production environment of context-aware software systems. A recent proposal to guide the design and execution of CASS test cases observed that context should freely vary during test execution as it does in production environments [7]. Therefore, during the design and execution of test cases, the test designer should not treat context variables as regular test input ones. First, because test input variables are deterministic, that is, the test designer has control over the test input variables and can define their values at design time. Context variables, by their nature, are not deterministic.

1

Dourish [5] makes a case for a phenomenological view of context for context-aware software. The author argues about a dual nature of context. In this view, an application can be designed to be aware to a context variable (and to the changes in its values) – deterministic. Yet, the same software system can occasionally react to a context variable – phenomenological. We argue that this phenomenological view of context represents an adequate framework when dealing with testing CASS. For instance, a self-driving car can be designed to turn left upon detection of a road cross (deterministic). However, designing a test case for this scenario, can not account for all possible hurdles and obstacles that can appear at any crossroad – phenomenological. Therefore, to set fixed testing values for context variables during testing, leads to artificial constraints, limiting testing samples and trials regarding the CASS under test. These artificial constraints are enacted by setting context variables to predefined values (for instance location [9]) or simulating those values through a software infrastructure [10]. Secondly, while test input variables impact only on the system under test, context variables have an impact both on the system under test and on the test oracle. The CASS under test can adapt its behavior after sensing a variation in the context. Therefore, for testing purposes, the test oracle must also be aware of the variations of context to provide an appropriate expected result regarding the test case. Having performed two quasi-systematic literature reviews on different aspects of testing [7] [8] , we claim that there is no available technology that can support the variation of context during the testing of CASS. Therefore, this paper discusses a new perspective for dealing with the testing of CASS and present current recommendations for those practitioners looking to test this sort of software system by comparing and contrasting current approaches against our proposed abstraction (i.e. the free variation of context during the testing). Furthermore, practical and methodological limitations of the approach are presented, aiming at to motivate researchers to work with these software technological challenges. This paper is organized as follows. First, we discuss practical and methodological motivations for our novel approach. In section –“A new abstraction for testing contextaware software systems” - we describe the elements of the new approach and how it address the methodological motivations. We then go on to discuss envisioned attributes and limitations of the approach. Finally, we close this paper with some conclusions and reflections.

Testing Context-Aware Software Systems without imposing artificial context constraints The research results discussed here comes from the CNPq CAcTUS project. The goal of this project is to understand how software engineers are currently assuring the quality of CASS. The CAcTUS project follows an evidence-based approach to software research. The results of two quasi-systematic literature reviews (qSLR) [7], [8] support our propositions. This section will first describe the implications for the industry and then discuss the methodological motivations arising from the acquired evidence.

Practical contributions Through our observations of current approaches used for testing CASS, we abstract the following practical contributions

that the industry can currently apply when testing such type of software system[12]: P1. Assure functional correctness before turning to testing context aware requirements. When the contextual input is considered for designing test cases, the test input space grows. Therefore, before tackling the problem of verifying how the software system behaves under a changing context, verify the system for its functionality in constrained contextual environments. P2. Design test cases to target context variables. Context variables are the designed parts of the software system where context is sensed. Therefore, the test designer should take care of such variables, as explicitly recommended in [13]. P3. Take advantage of automatic testing tools as much as possible. The most popular approach to deal with the unforeseen variation of context is to use platforms that can simulate pseudo-random input for context variables (for instance [10], [14], [15]). Practitioners can use these previous recommendations to test CASS because they do not require any special infrastructure (see in section “Practical limitations of the proposed approach”, PL3). The current software testing technologies can enact them. Nevertheless, we argue that they do not capture the phenomenological nature of context. Therefore, these approaches increase the risk of not achieving appropriate coverage of the system under test.

Methodological motivations This section argues for the development of a technology that enables the variation of context during testing. M1. Context influences the test cases [8]. This influence is explicit in two test case elements: the test item and the test oracle. Since the Test Item is context-aware, when executing the test case, the Test Item can sense the context and react by adapting its behavior. The Test Oracle must match the Test Item behavior change to provide a suitable expectation of results to evaluate the Test Case. M2. The application context, the test input, and the test environment are different entities and should also be distinguished during testing [7]. The ISO/IEC/IEEE 29119 Standard for Software and Systems testing [15] defines Test Input as: “the input to which the Test Item will be stimulated.” Moreover, the same standard defines the Test Environment as “the facilities, hardware, software, firmware, procedures, and documentation intended for or used to perform the software testing.” Therefore, for a Test Case, both Test Inputs and Test Environment should be completely deterministic. The test designer should be able to define the test inputs (variable types and values), and the test environment (hardware, operating systems, database, etc.). In contrast, for context, only the variable type can be determined during the test planning. Pre-setting context variable values leads to fixating context, which leads to observing the test item in an artificial environment. Therefore, limiting the possibility of identifying context-related faults (which results in M3). M3. Context must be left free to change during the test case execution [7]. In a production environment, a CASS is not constrained by the values that it can accept from its context sensors. For the execution of a Test Case, the CASS should not be imposed by artificial constraints that could limit its interaction with the context.

2

Context (free to vary) M3

Test Case Influences M1

System under test

Fuzzy Test Oracle M4 Needs

M2 Context Test Input Test Environment are different entities

Test Result Test Environment

Figure 1 Abstraction of the elements of a test case allowing the variation of context Back to the self-driving car example of the introduction, this car can be programmed to accept the full range of values included in the sensors specification. However, during testing, it is not feasible to design scenarios for all possible values. Therefore, higher abstraction oracles must be set. M4. Fuzzy Test Oracles. As a corollary of the previous two results, we argue that Fuzzy (non-deterministic) Test Oracles are needed to evaluate the test cases results. Since context has influence in the Test Case, then the Test Oracle cannot be deterministic. Therefore a behavior expectation should be sufficient to pass judgment on the suitability of the Test Item behavior modification given the current context and Test Inputs.

timely and relevant expectations to evaluate the test output (M4). Finally, the context, the test input, and the test environment are different entities that must be differentiated (M2). Test input and context are different in nature. The test input is deterministic. Moreover, though some inputs can be generated outside of the scope of the test case, these are known to the tester (hence, we represent it as an arrow not as a cloud). Also, the context and test environment since the latter one should be defined in the system under test requirements. And though some environments are not feasible to reproduce during test time, this brings on a different problem than that of the variation of context.

A new abstraction for testing context-aware software systems

Limitations of the proposed approach

Figure 1 depicts the relationships between the methodological motivations and the elements of a test case. In this figure, Context – represented by a cloud – is not completely within the scope of the Test Case. This relates how the test case cannot control all possible contextual inputs (M3). The arrows for the context towards the system under test and the test oracle, convey the influence that the context has on both of these items of a test case (M1). As per this influence, the test oracle must adapt to the contextual changes to give

This section presents some limitations that must be considered when generalizing the arguments from the previous section. This section is also divided into the discussion of the current practical applications and methodological limitations.

Practical limitations of the proposed approach PL1.The proposed abstraction is in its early stages, and it is not completely clear how it can contribute to improving the quality of CASS. The proposed abstraction was introduced in [7]. In [17], the abstraction supported the proposal of a test suit design 3

process, which has only been evaluated in vitro with students. There is no discussion about how this differentiation applies to the unit testing/code level. PL2.There are no available testing strategies to support practitioners on designing test cases for CASS, which consider the behavior changes regarding the Test Items and Test Oracles. In [17], the authors compare which test types (as defined by ISO/IEC 25010) are being used to design test cases for CASS. Their results show that the current solution is to adapt traditional test design strategies to overcome the context variation. PL3.There is currently no computerized infrastructure to support the proposed approach. To the best of our knowledge, no technical infrastructure enables the practical implementation of this abstraction yet. Even whether practitioners design test cases using the proposed abstraction, no tools nor approaches allow them to plan or execute CASS testing considering the variation of context during testing execution. • Methodological limitations of the proposed approach ML1. Reviewed literature is relatively narrow in comparison to the available technical literature. Contributions presented in this paper all come from two independently conducted secondary studies. Both qSLRs share the same Population (“Sensibility to context,” the same keywords in the search engines). Thus, we could crudely estimate the amount of missed primary sources. Table 1 presents the overlaps between both qSLRs. Using the LincolnPetersen estimator for Mark and Recapture method [18], we can estimate that either research did not consider five potential primary sources. Reference Alsos and Dahl [19] Amalfitano, D., et al. [13] Canofra, G. et al. [20] Chan W.K., Chen, T., and Lu, H. [21] Chan W.K. et al. [22] Frederiks, E.M., DeVries, B., and Cheng, B. [23] Griebe and Gruhn [24] Jang, B., Long, X. and Gao, X. [25] Merdes, M., et al. [26] Micskei, Z. et al. [27] Propp, S., Buchholz, G., and Forbrig, P.[28] Ryan and Gonsalves [29] Satoh, I [9] Tse, H. et al. [30] Wang, H. and Chan, W.K. [31] Wang, H., Chan, W.K, and Tse, H [32] Wang H., Zhai, K. and Tse, H [33] Wang, Z. Elbaum, S., and Rosenblum, D.S. [12] Ye, C. et al. [34]

In [8]

In [7]

Yes

Yes Yes Yes

Yes Yes Yes Yes Yes Yes Yes Yes

Yes Yes Yes Yes Yes

Yes Yes

Yes Yes Yes Yes

Yes

Yes

Table 1 Evidence-based technologies regarding CASS testing ML2. Industrial experts have not evaluated the proposed abstraction: The only evaluation of the test cases design process observing context variation was an in-vitro observational study. Without proper industrial validation, it is not possible to warrant that this abstraction is anything else than a

theoretical construct with a limited perspective of industrial use. However, the initial results did not suggest it.

On the issue of Quality assurance of context-aware software system, Where do we stand? There is a need for developing and deploying CASS. Four out of the 12 worst “software glitches” in 2015 were caused by CASS (see http://www.computerworlduk.com/galleries/infrastructure /top-10-software-failures-of-2014-3599618/). The industry has very few alternatives for assuring the quality of such systems. Available software testing technologies rely heavily on simulation or extensive computation. We believe that this solution is not feasible for all parties involved with the construction of CASS. New testing technologies dealing with the issues brought by the variation of context and its influence on systems’ behaviors are needed. Such technologies need to be experimentally validated to provide feasible solutions for the industry. Since the demand for CASS is continuously growing while such technologies are being developed and validated, the current and viable alternatives for the industry are P1, P2, and P3, as discussed in Section II – practical contributions.

Summarizing remarks This paper describes the implications of a novel abstraction for testing CASS, where test input and context variables are differentiated. This proposal aims at dealing with the issue that context variation worsens the test designers capacity to make trade-off decisions between test coverage and test effort. The authors argue that software testing technologies must evolve to address the context-awareness characteristic of software systems. Currently, the testing approaches deal with the variation of context by either fixating context values or simulating them based on computerized infrastructure. In either case, the test item is not receiving contextual information from the context, but through artificial methods. Thereby, limiting the test case validity. The authors advocate the idea that context should vary freely during test execution as it does in the production environments. The proposed abstraction can guide the test designer into recognizing the different nature of both types of variables into the test case: deterministic test inputs under the test designer control and context variables, which should capture the possible variations of context. So far, the proposed abstraction has been enacted in a test suit design process, which has been subject to limited in-vitro evaluations. Further research is needed to validate both the abstraction and the new software testing technologies that must be derived from it. We are currently working with industrial and academic partners to design an appropriate environment to support its experimental evaluation on testing CASS.

Acknowledgement Authors would like to thank the CAcTUS team members for the thoughtful discussion of the ideas in this paper. The “CAcTUS - Context-Awareness Testing for Ubiquitous Systems” project partially financed by CNPq – Universal 14/2013 Number 484380/2013-3.

References [1]

M. Weinberger, D. Bilgeri, and E. Fleisch, “IoT business models in an 4

industrial context,” - Autom., vol. 64, no. 9, Jan. 2016. [2]

S. Sridevi, B. Sayantani, K. P. Amutha, C. M. Mohan, and R. Pitchiah, “Context Aware Health Monitoring System,” in Medical Biometrics. Second International Conference, (ICMB), hong kong, China, 2010, pp. 249–257.

[3]

S. Wang, J. Wan, D. Li, and C. Zhang, “Implementing Smart Factory of Industrie 4.0: An Outlook,” Int. J. Distrib. Sens. Networks, vol. 2016, 2016.

[4]

C. Perera, A. Zaslavsky, P. Christen, and D. Georgakopoulos, “Context aware computing for the internet of things: A survey,” IEEE Commun. Surv. Tutorials, vol. 16, no. 1, pp. 414–454, 2014.

[5]

P. Dourish, “What we talk about when we talk about context,” Pers. Ubiquitous Comput., vol. 8, no. 1, pp. 19–30, 2004.

[6]

G. D. Abowd, A. K. Dey, P. J. Brown, N. Davies, M. Smith, and P. Steggles, “Towards a Better Understanding of Context and ContextAwareness,” in Computing Systems, vol. 40, 1999, pp. 304–307.

[7]

S. Matalonga, F. Rodrigues, and G. H. Travassos, “Test Design Techniques for Context Aware Software Testing: Results from a quasiSystematic Literature Review,” Subm Publ. J. Softw. Syst., 2016.

[8]

I. Santos, R. M. C. Andrade, L. Rocha, S. Matalonga, K. M. Olivera, and G. H. Travassos, “Test Case Design for Context-Aware Applications: Are We There Yet?,” Inf. Softw. Technol., vol. In Press, 2017.

[20]

G. Canfora, F. Mercaldo, C. A. Visaggio, M. D’Angelo, A. Furno, and C. Manganelli, “A case study of automating user experience-oriented performance testing on smartphones,” in Proceedings - IEEE 6th International Conference on Software Testing, Verification and Validation, ICST 2013, 2013, pp. 66–69.

[21]

W. K. Chan, T. Y. Chen, H. Lu, T. H. Tse, and S. S. Yau, “A metamorphic approach to integration testing of context-sensitive middleware-based applications,” in Proceedings - International Conference on Quality Software, 2005, vol. 2005, pp. 241–249.

[22]

W. K. Chan, T. Y. Chen, H. Lu, T. H. Tse, and S. S. Yau, “Integration testing of context-sensitive middelware-based applications: a metamorphic approach,” Int. J. Softw. Eng. Knowl. Eng., vol. 16, no. 5, pp. 677–703, Oct. 2006.

[23]

E. M. Fredericks, B. DeVries, and B. H. C. Cheng, “Towards run-time adaptation of test cases for self-adaptive systems in the face of uncertainty,” Proc. 9th Int. Symp. Softw. Eng. Adapt. Self-Managing Syst. - SEAMS 2014, pp. 17–26, 2014.

[24]

T. Griebe and V. Gruhn, “A model-based approach to test automation for context-aware mobile applications,” in Proceedings of the 29th Annual ACM Symposium on Applied Computing - SAC ’14, 2014, pp. 420–427.

[25]

B. Jiang, X. Long, and X. Gao, “MobileTest: A tool supporting automatic black box test for software on smart mobile devices,” in Proceedings International Conference on Software Engineering, 2007.

[26]

M. Merdes, R. Malaka, D. Suliman, B. Paech, D. Brenner, and C. Atkinson, “Ubiquitous RATs: How Resource-Aware Run-Time Tests Can Improve Ubiquitous Software System,” in 6th International Workshop on Software Engineering and Middleware, SEM 2006, 2006, pp. 55–62.

[9]

I. Satoh, “Software testing for mobile and ubiquitous computing,” in The Sixth International Symposium on Autonomous Decentralized Systems, 2003. ISADS 2003., 2003, no. Section 2.

[10]

L. Capra, W. Emmerich, and C. Mascolo, “CARISMA: Context-Aware Reflective middleware System for Mobile Applications,” IEEE Trans. Softw. Eng., vol. 29, pp. 929–945, 2003.

[27]

Z. Micskei, Z. Szatmári, J. Oláh, and I. Majzik, “A Concept for Testing Robustness and Safety of the Context-Aware Behaviour of Autonomous Systems,” in Lecture Notes in Computer Science, 2012, pp. 504–513.

[11]

S. Matalonga, F. Rodrigues, and G. H. Travassos, “Challenges in Testing Context Aware Software Systems,” in Systematic and Automated Software Testing, 2015.

[28]

S. Propp, G. Buchholz, and P. Forbrig, “Task model-based usability evaluation for smart environments,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2008, vol. 5247 LNCS, pp. 29–40.

[12]

Z. W. Z. Wang, S. Elbaum, and D. S. Rosenblum, “Automated Generation of Context-Aware Tests,” in 29th International Conference on Software Engineering (ICSE’07), 2007.

[29]

C. Ryan and A. Gonsalves, “The effect of context and application type on mobile usability: An empirical study,” in Conferences in Research and Practice in Information Technology Series, 2005, vol. 38, pp. 115– 124.

[30]

T. H. Tse, S. S. Yau, W. K. K. Chan, H. Lu, and T. Y. Y. Chen, “Testing context-sensitive middleware-based software applications,” in Proceedings of the 28th Annual International Computer Software and Applications Conference (COMPSAC )., 2004, pp. 458–466.

[31]

H. Wang and W. K. Chan, “Weaving Context Sensitivity into Test Suite Construction,” in 2009 IEEE/ACM International Conference on Automated Software Engineering, 2009, pp. 610–614.

ISO/IEC/IEEE 29119-1:2013, “Software and systems engineering Software testing Part 1:Concepts and definitions,” ISO/IEC/IEEE 291191:2013, pp. 1–64, Sep. 2013.

[32]

H. Wang, W. K. Chan, and T. H. Tse, “Improving the Effectiveness of Testing Pervasive Software via Context Diversity,” ACM Trans. Auton. Adapt. Syst., vol. 9, no. 2, p. 9:1--9:28, 2014.

F. Rodrigues, S. Matalonga, and G. H. Travassos, “CATS Design: A Context-Aware Test Suite Design Process,” in Proceedings of the 1st Brazilian Symposium on Systematic and Automated Software Testing SAST, 2016, pp. 1–10.

[33]

H. Wang, K. Zhai, and T. H. Tse, “Correlating context-awareness and mutation analysis for pervasive computing systems,” in Proceedings International Conference on Quality Software, 2010, pp. 151–160.

[34]

C. Ye, S. C. Cheung, J. Wei, H. Zhong, and T. Huang, “A study on the replaceability of context-aware middleware,” in Proceedings of the First Asia-Pacific Symposium on Internetware - Internetware ’09, 2009, pp. 1–10.

[13]

[14]

[15]

[16]

D. Amalfitano, A. R. Fasolino, P. Tramontana, and N. Amatucci, “Considering Context Events in Event-Based Testing of Mobile Applications,” in 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops, 2013, pp. 126–133. A. Bertolino, G. De Angelis, F. Lonetti, and A. Sabetta, “Let the puppets move! Automated testbed generation for service-oriented mobile applications,” in Proc. of the 34th EUROMICRO Conference on Software Engineering and Advanced Applications, 2008, pp. 321–328.

[17]

S. Matalonga, F. Rodrigues, and G. H. Travassos, “Matching Context Aware Software Testing Design Techniques to ISO/IEC/IEEE 29119,” in 2015 Spice Conference, 2015, pp. 33–44.

[18]

A. Grimm, B. Gruber, and K. Henle, “Reliability of Different MarkRecapture Methods for Population Size Estimation Tested against Reference Population Sizes Constructed from Field Data,” PLoS One, vol. 9, no. 6, p. e98840, 2014.

[19]

O. A. Alsos and Y. Dahl, “Toward a best practice for laboratory-based usability evaluations of mobile ICT for hospitals,” Proc. 5th Nord. Conf. Human-computer Interact. Build. Bridg. - Nord. ’08, p. 3, 2008.

5