Cloud Testing Framework C. R. Lima Neto
V. C. Garcia
Federal University of Bahia Federal Institute of Bahia Vitoria da Conquista, Brazil
Center for Informatics Federal University of Pernambuco Recife, Brazil
[email protected]
[email protected]
ABSTRACT Context: Cloud-based testing introduces a new set of challenges. While many companies are approaching cloud computing with cautious, testing appears to be next area of growing interest. Objective: This paper propose the development of a new framework focusing on cloud testing. We intend to define the activities, roles, techniques, and tools that will be used in an integrated framework. Method: A research methodology containing all the steps that will be executed during the proposal implementation was defined. Results: A schedule was suggested in order to allow the research development. Conclusion: It is concluded that is important to establish the standardization for the cloud environment in order to ensure the cloud applications quality.
Categories and Subject Descriptors
major impacts on software testing and maintenance. The growth of cloud-based services is evident, but clouds must still undergo development and testing before we can deploy them [15]. As the development of cloud-based services increases, so will the need for testing their applications. Therefore, this paper presents the development proposal of a Cloud Testing framework capable of testing cloud computing applications. We intend to define the activities, roles, techniques, and tools that will be used in an integrated framework. We also defined a research methodology containing all the steps that will be executed. The remainder of this paper is structured as follows: Section 2 presents the motivation, Section 3 presents the Cloud Testing Challenges, Section 4 discusses the research proposal, Section 5 presents the research methodology, Section 6 presents the development schedule, and finally, Section 7 contains the concluding remarks.
D.2.5 [Software Engineering]: Testing and Debugging
2.
General Terms Theory
Keywords Software Testing; Testing Tools; Cloud Testing; Cloud Computing; Domain-Specific Languages; DSL
1.
INTRODUCTION
The software industry has continuously been challenged to improve its engineering practice aiming at delivering products in a faster and more reliable way. Thus, competitiveness in software development has increasingly become a concern for companies of all sizes and in all markets. As a result, Cloud computing received significant attention recently as it changes the way computation and services to customers [5]. Nowadays, companies such as Amazon, Google, and Salesforce.com offer their cloud infrastructure for services. According to Gao et al. [2], Cloud computing not only brings new business opportunities, but also causes some
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. EASE ’13, April 14 - 16 2013, Porto de Galinhas, Brazil Copyright 2013 ACM 978-1-4503-1848-8/13/04 $15.00.
252
MOTIVATION
Software testing tools are available for testing in every stage of the software development life cycle [1]. Some organizations have used testing tools to manage, store and handle the tests. According to Nakagawa et al. [8], the tools availability make testing a more systematic activity and minimizes the costs, the time consumed, as well as the errors caused by human intervention. Integrating test environments and the test asset repository is a cumbersome work, that is even worse due to the lack of automated support tool, which leads to be a manually performed task. There is a need for specific testing tools that should help to manage the reusable testing assets and automate the execution of tests and the analysis of their results as well [16]. In the Software Product Lines (SPL) context, the amount of available tools decrease drastically, and the need of tools to reduce the effort during the SPL Testing Process is a gap that need to be filled. We need testing tools to manage the variability and avoid the explosion of test cases [13]. For this reason, we proposed [12] and implemented a software product lines system test case tool [11], aiming at the creation and management of test assets, which contributes for the effort reduction during the system testing level of the SPL testing process. According to Riungu-Kalliosaari et al. [15], cloud computing lets organizations carry out technical feasibility studies while trying to identify the best test tools and environments. Readily available cloud-based tools and environments enhance agility in testing and, effectively, in development over-
all. Meira et al. [6] envisaged a research framework containing a large number of research challenges. For this reason, this research proposal intends to evolve the existing tool and create a new Testing Framework by investigating Testing as a Service (TaaS) in Clouds. We will analyze the existing frameworks used in cloud testing, specially, considering effort reduction aspects since it can be faced as a bottleneck for effectively introduce testing concepts into practice. Experimentation in both academical and industrial environments will be used in order to validate and calibrate the framework, which will enhance it reliability.
3.
CLOUD TESTING CHALLENGES
According to Riungu-Kalliosaari et al. [15] cloud-based testing requires testing of additional aspects and parameters. These might not necessarily be new, but testing in the cloud might exacerbate them. There are a number of issues and challenges in testing clouds and cloud-based software. Gao et al. [2] discussed them from four areas: (1) On-demand test environment construction, (2) Scalability and performance testing, (3) Testing security and measurement in clouds, and (4) Integration testing in clouds. On the first area, there are no supporting solutions to assist engineers to set up a required test environment in a cloud using a cost-effective way. In addition, there is a lack of solutions to easily leverage engineers cloud-based applications in a cloud with the existing test tools because most of them are not cloud-enabled. On the second area, most of existing studies focus on scalability evaluation metrics and frameworks for parallel and distributed systems. Usually, the evaluation is conducted in a static and pre-fixed system environment. For this reason, the existing evaluation metrics, frameworks, and solutions did not consider the features in cloud testing. On the third area, security testing has becoming a research subject with many open questions. Since securityrelated issues are a major concern, especially in test data management. Storing and handling test data should never compromise confidentiality. One way to test cloud security is to selectively expose data to public clouds [15]. Finally, on the fourth area, not much research results have been applied in the testing integration real engineering practice. One of the major reasons is because the existing software and components are developed without enabling technology and solution to support and facilitate systematic software integration. In a cloud infrastructure, engineers must deal with integration of different Software as a Service (SaaS) and applications in/over clouds in a black-box view based on their provided APIs and connectivity protocols.
4.
RESEARCH PROPOSAL
The study we prior conducted seems to point a growing interest in a comprehensive Cloud testing, including the open issues and challenges in cloud testing and TaaS in all testing phases. In addition, test selection techniques, test automation and architecture-based cloud testing are points to be investigated. Thus, it can be considered the main goals of the Research Project. The framework we can propose shall handle quality attribute testing considering requirements in quality levels,
253
and give practical answers on the question how to maintain the traceability between development and test artifacts in the whole development life cycle as well through a comprehensive testing framework? The literature points out these as relevant set of aspects and still open challenges to be dealt with, regarding Cloud Testing. Thus, our research questions will address these aspects. This work will consider testing in the all the testing levels, as the main intention is to produce a framework aiming at cost reduction in cloud testing development activities through the whole life cycle. Thus, the emphasis shall be given to build a worthwhile framework that should be used in industrial projects. In addition, this framework will encompass a systematic method to establish a link among standard development artifacts and cloud test artifacts, which will impacts in the overall testing cost. In summary, this work strives for addressing elements that have not been adequately considered, as shown in literature, which includes reports on works developed in both academy and industry.
4.1
Using DSL to support some Challenges
According to Mernik et al. [7], a Domain-Specific Language (DSL) is a language that provides constructs and notations tailored toward a particular application domain. It trade generality for expressiveness in a limited domain, and this can bring several benefits. However, these benefits do not come for free. The cost of DSL design, development and maintenance has to be taken into account. Without appropriate methodologies and/or tools these costs can be higher than savings. We intend to extend the systematic mapping study proposed by Nascimento et al. [9] in order to verify the possibility to use DSLs in a Cloud-Based context. We believe that some Cloud Testing Challenges such as scalability, performance, and integration testing, can be solved through the use of DSLs. For this reason, we intend to create a new DSL to support the proposed Framework. But first, we need to figure out what needs to be expressed in a DSL, and then formalize the specific language for the Cloud Computing area context. Moreover, we are interested in effort reduction, using automated processes instead of repetitive manual activities. DSLs will enable automation development faster and effective.
5.
RESEARCH METHODOLOGY
This section briefly describes the strategy we shall adopt in order to develop the Research Project. A major impact is known as Testing as a Service (TaaS) in Clouds. TaaS cloud infrastructures is considered as a new business and service model, in which a provider undertakes software testing activities of a given application system in a cloud infrastructure for customers as a service based on their demands [2]. From the mapping study results [13], which gave us a general view of the testing tool area, a preliminary mapping study will be will be performed in order to evolve the results to cloud testing. If we need a more detailed investigation on a specific topic, we should perform a Systematic Review, based on Kitchenham guidelines [3]. It provides a set of well-detailed guidelines that will be useful in order to delineate our study. After gathering information necessary to work on our pro-
posal, the project will then be designed and executed. As we are to define a framework for software engineering, EPF Eclipse Process Framework - Composer1 will be used, since it provides systematic way to organize it and we can also integrate it with other framework. In order to achieve the project goal, it will be necessary to devote effort on understanding detailed aspects on the following topics: (i) cloud computing, (ii) general aspects on software testing, (iii) software testing applied in the cloud computing context (also known as Cloud Testing), (iv) testing effort reduction strategies and (v) experimental study concerns. The proposed framework key principles are systematic and flexibility. Thus, for the framework to be systematic will be defined phases, activities, tasks, tools, inputs, outputs, roles and guidelines. The framework should be flexible by enabling framework customization, thus, for achieving this goal is necessary the separation among methods, process and artifacts. Tools to support the framework shall be developed as the work is being done, so that as soon as the framework is to be delivered and evaluated, the bottlenecks will appear showing good points to be automated, in this scenario the tools should be developed. Thus, we will evaluate the need of support tools after putting the process into practice. The effectiveness of the tools will be evaluated in both academic and industrial contexts. After developing our approach, we are supposed to check its validity and suitability. Thus, case studies [4] and experiments [17] will be performed in this sense. They both can be considered fundamental for executing cost-benefit analysis of software engineering approaches. Our main intention regarding experimenting this framework is twofold: we shall apply it in the academic environment, in order to have feedback from the subjects involved, refine it and make necessary adjustments to further applying preliminary observational studies in an industrial context. Based on the empirical evidences from the experiments, we can thus construct experience bases that provide qualitative and quantitative data about the advantages and disadvantages of using the proposed approach, probably in different sets and domains. The schedule following presented give more details on the activities that will be performed towards the goals accomplishment (see Table 1).
6.
RESEARCH SCHEDULE
This research proposal will be developed during a PhD course. The main objective is to propose a new framework for Cloud Testing. We need to investigate the roles, activities, techniques, and tools required by a cloud environment. For this reason, this research will be carried out by fulltime dedication from the researcher, assisted, through periodic meetings, by his advisor and co-advisor, as well as other members of the ASSERT Labs2 research group. Furthermore, internal workshops will be presented, papers will be prepared, showing the results of the research. According to the research plan, the following activities were defined in order to achieve the project goals: 1 2
http://www.eclipse.org/epf/ http://assertlab.com/
254
1. Coursework - The preliminary activity shall be taking the set of PhD compulsory and optional courses. 2. Preliminary Mapping Study and Survey with experts in the area - We intend to realize a preliminary mapping study in order to better understand the Cloud Testing field, through synthesizing evidence to suggest important implications for practice, as well as, identifying research trends, open issues, and areas for improvement. Based on the Mapping study, we intend to evolve the research, validate, and refine the findings with some experts and researchers in the cloud area. It will provide information and can indicate aspects for improvement. 3. Performing a Systematic Review - A Systematic Literature Review on the Research Questions is a optional step, in order to refine our current data and establish future directions. Likewise the mapping study, after performing the Systematic Literature Review, papers shall be written and then submitted to a conference or journal in order to have feedback on the research directions and give to the community contributions on the work to be performed. 4. Cloud testing framework design - Based on the review results, and study on aspects such as Cloud Computing, Software Testing, Software Testing applied in the Cloud Computing context we are supposed to build a cloud testing process in order to accomplish the challenges raised in the study. 5. Tool Support development - As we identify the need of developing support tools to aid in the process application, we should elicit its requirements, design, implement and test the tools. After delivering the support tools, their use are to be evaluated. The academic context will serve as environment to analyze tools efficacy. 6. Research on Experimental Software Engineering - Figure out theoretical and practice strategies regarding experimentation, which will be helpful to evaluate the process usefulness. Kitchenham and Charters [3], and Wohlin et al. [17] strategies will be adopted in this direction. 7. Definition and execution of the experimental study - This activity will assemble aspects related to the experimentation definition and execution. 8. Evaluation of the experimental study - The proposed process will be experimented and then evaluated. 9. Cloud testing framework calibration - Data from the experimental studies shall provide substantial information to retrofit the process with refinements suggestions. 10. Reporting - Papers describing the Research results, including the process and the experimental study shall be written and published in international journals and conferences. 11. Thesis writing - Describe everything related to the research conducted along the Doctoral Project regarding the proposed work and its results. 12. PhD defense - A public oral presentation of the PhD dissertation.
7.
CONCLUSION
According to Riungu et al. [14], Cloud computing is the latest major trend in the IT industry. There is an anticipation of an increase in testing solutions in the cloud, providing flexibility and cost benefits. We agree with the authors when they believe that cloud computing promises a lot of potential for testing. This paper presented some Cloud Testing challenges and
Activity 1 2 3 4 5 6 7 8 9 10 11 12
Table 1: Schedule Deadline From March, 2013 to December, 2013 From July, 2013 to March, 2014 From February, 2014 to June 2014 From July, 2014 to June 2015 From February, 2015 to August, 2015 From September, 2015 to November, 2015 From October, 2015 to May, 2016 From May, 2016 to July, 2016 From August, 2016 to January, 2017 From August, 2016 to February, 2017 From January, 2016 to February, 2017 March 2017
in order to solve some of them we suggest a new research proposal that focus on the development of a Cloud Testing Framework. Furthermore, we presented the research methodology and research schedule that will be used to implement the proposal. As we mentioned before, this research proposal will be developed during a PhD course. The main objective is to propose a new integrated solution for Cloud Testing.
ACKNOWLEDGMENT This research was sponsored by the Program Center for the Research and Development on Digital Technologies and Communication (CTIC) of the Brazilian Minister of Science and Technology, grant 68/2012. In addition, this work was partially supported by the National Institute of Science and Technology for Software Engineering (INES [10]), funded by CNPq and FACEPE, grants 573964/2008-4, APQ-10371.03/08 and APQ-1044-1.03/10 and Brazilian Agency (CNPq processes number 475743/2007-5 and 140060/2008-1).
8.
REFERENCES
[1] M. Fewster and D. Graham. Software Test Automation: Effective Use of Test Execution Tools, volume 10. John Wiley Sons, Ltd., 1999. [2] J. Gao, X. Bai, and W. T. Tsai. Cloud-Testing Issues, Challenges, Needs and Practice. Software Engineering: An International Journal, 1, 2011. [3] B. Kitchenham and S. Charters. Guidelines for performing Systematic Literature Reviews in Software Engineering. Software Engineering Group School of, 2:1051, 2007. [4] B. Kitchenham, L. Pickard, and S. L. Pfleeger. Case studies for method and tool evaluation. IEEE Softw., 12(4):52–62, July 1995. [5] S. R. L. Meira, V. A. Bur´egio, L. M. Nascimento, E. G. M. de Figueiredo, M. Neto, B. P. Encarna¸ca ˜o, and V. C. Garcia. The Emerging Web of Social Machines. The Computing Research Repository, abs/1010.3045, 2010. [6] S. R. L. Meira, V. A. A. Buregio, L. M. Nascimento, E. Figueiredo, M. Neto, B. Encarnacao, and V. C. Garcia. The emerging web of social machines. In Proceedings of the 2011 IEEE 35th Annual Computer Software and Applications Conference, COMPSAC ’11, pages 26–27, Washington, DC, USA, 2011. IEEE Computer Society.
255
Months 10 months 9 months 5 months 12 months 7 months 3 months 8 months 3 months 6 months 7 months 14 months 1 month
[7] M. Mernik, J. Heering, and A. M. Sloane. When and how to develop domain-specific languages. ACM Comput. Surv., 37(4):316–344, 2005. [8] E. Y. Nakagawa, A. S. Sim˜ ao, F. C. Ferrari, and J. C. Maldonado. Towards a Reference Architecture for Software Testing Tools. In International Conference on Software Engineering & Knowledge Engineering, pages 157–162. Knowledge Systems Institute Graduate School, 2007. [9] L. M. Nascimento, D. L. Viana, P. A. M. S. Neto, D. A. O. Martins, V. C. Garcia, and S. R. L. Meira. A systematic mapping study on domain-specific languages. 2012. [10] National Institute of Science and Technology for Software Engineering (INES). Available in: http://www.ines.org.br/. Access date: 5 mar. 2013. [11] C. R. L. Neto, E. S. de Almeida, and S. R. de Lemos Meira. A software product lines system test case tool and its initial evaluation. In IEEE 13th International Conference on Information Reuse & Integration, IRI 2012, Las Vegas, NV, USA, August 8-10, 2012, pages 25–32. IEEE, 2012. [12] C. R. L. Neto, I. C. Machado, P. A. M. S. Neto, E. S. Almeida, and S. R. L. Meira. Software Product Lines System Test Case Tool: A Proposal. In International Conference on Software Engineering & Knowledge Engineering, pages 699–704. Knowledge Systems Institute Graduate School, 2011. [13] C. R. L. Neto, P. A. M. S. Neto, E. S. Almeida, and S. R. L. Meira. A Mapping Study on Software Product Lines Testing Tools. In International Conference on Software Engineering & Knowledge Engineering. Knowledge Systems Institute Graduate School, 2012. [14] L. M. Riungu, O. Taipale, and K. Smolander. Research issues for software testing in the cloud. 2010. [15] L. Riungu-Kalliosaari, O. Taipale, and K. Smolander. Testing in the cloud: Exploring the practice. IEEE Software, 29:46–51, 2012. [16] A. Tevanlinna, J. Taina, and R. Kauppinen. Product Family Testing: a Survey. ACM SIGSOFT Software Engineering Notes, 29:12–12, March 2004. [17] C. Wohlin, P. Runeson, M. C. O. Martin H¨ ost, B. Regnell, and A. Wessl´en. Experimentation in Software Engineering: An Introduction. The Kluwer Internation Series in Software Engineering. Kluwer Academic Publishers, Norwell, Massachusets, USA, 2000.
