A flexible public cloud based testing service for heterogeneous testing targets Sheng-Jen Hsieh Institute of Computer Science and Engineering National Chiao Tung University Hsinchu, Taiwan
Guo-Heng Luo Institute of Computer Science and Engineering National Chiao Tung University Hsinchu, Taiwan
Shyan-Ming Yuan (Corresponding author) Institute of Computer Science and Engineering National Chiao Tung University Hsinchu, Taiwan Email:
[email protected]
Hsiao-Wei Chen, Cloud Computing Laboratory, Telecommunication Laboratories Chunghwa Telecom Co., Ltd. Taoyuan, Taiwan
Abstract—To guarantee quality of the service, services need to be tested to ensure the service quality. But most of existing testing service can’t provide multi-type testing and deploy testing clients concurrently in different regions. To overcome these problems, in this paper, a cloud testing service combine cloud computing and multi-testing tools is proposed. Using the proposed service user can deploy the testing clients in different countries to access the testing targets, when the testing is finish user can easily get both of detail and summarized test results. To show the capabilities of proposed service, an experiment is performed. The results shows that the proposed service can guarantee testing quality, automatically deploy testing clients concurrently in different region, get latency from different countries, and perform press testing. Keywords—testing service; cloud; testing tool
I.
INTRODUCTION
In recent years, software service become more and more complex. To guarantee quality of the service, services need to be tested to ensure the service quality. Existing testing service offer those testing service for different type software or system, for some testing target it may request special needs. For example, when simulating the scenario that accessing web services, the clients should be distributed in different locations otherwise it may not be able to simulate actual conditions. Recently “cloud computing” has become very popular. Cloud services offers resource including computing instance, storage and network connections [1] [2]. Since cloud service providers deploy data centers in different areas, it provides the capability of geographical distribution deployment. By combining testing service with public cloud, testing services can satisfy the need of geographical distribution deployment. But most of the testing service can only setting the number of concurrent access but can’t select where the clients come from one or more different location [3]. Moreover, Different service or system need different type testing tool if user want to test different
Copyright IEICE - Asia-Pacific Network Operation and Management Symposium (APNOMS) 2014
type of testing targets they may need to choose different testing service for different testing targets, it is because most of the testing service can’t provide multi-type testing. Some testing service provide a platform for user to test their software or system on it, before testing start, user need upload their software or build the testing environment on the platform first [4], then they can start the test. Since user need upload their software or system to the platform there are some controversy, such as the copyright of the software or system and the running environment can’t be guarantee as same as the actual running environment. When the testing is start some testing service use the multi thread technique, using the multi thread technique then one instance can simulate more than one clients but the problem is how many thread can run in an instance? If the number of thread in an instance is too much then the testing quality may not be trust. To solve these problem, this paper proposed flexible public cloud based testing service for heterogeneous testing targets (F.C.H.T.S) which integrates various testing tools with cloud computing techniques. To provide the proposed service, FCHTS system is designed and implemented. Using the proposed service, the number of concurrent access client, geographical distribution and target service type can be flexibly configured. Moreover, when multi thread technique is used, the proposed service can still guarantee the test quality from client side by cloud instance capability pre-testing and when using the proposed service user just need setting some information and don’t need upload their software or system. To show the capabilities of the proposed service, two web services are selected for evaluating the quality of service in different conditions including number of concurrent clients, different client geographical locations, to guarantee the quality of testing provide by proposed service the capabilities of instance has been measured.
II.
III.
LITERATURE REVIEW
A. Testing tool There are many testing tool that can test the performance of the computer or network, and every testing tool has focus on different type of target, for example some may focus on the performance of read/ write rate, some may focus on testing the bandwidth of the network or the speed of computing…etc.
SYSTEM DESIGN
A. System overview Figure 1 illustrates the system components, there are 4 layers, the first layer is web interface, second layer is management server, third layer is Amazon Web Service (AWS) [15] and the last layer is test target.
“The Apache Jmeter is one of the powerful testing tools, it is an application designed to load test functional behavior and measure performance. It was originally designed for testing Web application but has since expanded to other test functions. The Apache Jmeter can be used to simulate a heavy load on a server, groups of servers, network or object to test its strength or to analyze overall performance under different load types” [5]. FIO is another testing tool for doing a particular type of I/O action as specified by the user, the typical use of FIO is to write a job file matching the I/O load one wants to simulate [6]. To simulate the different situation of file read/write FIO is a good choose, but if the testing scenario is to simulate the client that access the network-based file system then FIO can’t simulate the clients that distribute indifferent regions. B. Testing Service There are many testing service, each service has focus on testing different type of targets. Some service focus on test the functionality, when tester publish system or software to the testing platform the platform can input all the case of the input data to run all the path in the system or software to ensure that all the software’s function can work correctly [7] [8], “Classic Symbolic Execution” is one of the method that can execution all the path in the program [9] [10] [11] [12].
Fig. 1. System Overview
IV.
EXPERIMENT
The type of instance using in this paper is AWS m1.small.
Another type of service platform focus on stress testing. If the max number of concurrent access request that the system can afford is lower than expect, then system builder should find the possible problems then modify or expand the system. According to the test result the tester can determine to scale up or scale down the size of the system server. C. Cloud Based Testing Service Traditional testing require expensive infrastructure and resources that were only used when testing the applications or software. With the development of science and technology the complexity of the applications are more complex than before, build an in-house testing environment is more and more difficult. Compare to an in-house testing environment cloudbased testing offers several benefits. Using the commercial cloud computing infrastructure and storage can help us to create the run-time virtualization testing environment and easily to manage the testing servers in the public cloud [13] [4] [14]
Fig. 2. Figure 2 Instance capability pre-testing
In this experiment one instances is deployed. We set the number of thread of the instance from 1 to 60 and get the response latency from each case, each thread execute the http “get” method to access the College Entrance Examination Center homepage (www.ceec.edu.tw). In figure 2, the yellow line shows the latency of instance concurrent execute different number of thread and the blue line shows the standard deviation of number of thread from 1 to 60
in an instance. When the number of thread in an instance is more than 30 the standard deviation grows very fast, so when we deploy the instances we should run less than 30 threads in an instance otherwise it will affect the latency.
V.
CONCLUSION
In this paper, FCHTS is proposed. The service can offer general purpose testing service and automatically deploy the testing clients to the countries that tester selected. Currently 2 major testing type are supported, one is the network-based file system and the other one is HTTP request based web service. Usually testing service adopt multi thread technique, using multi thread technique can save costs and improve the performance utilization of the physical instance. However, when too many threads are generated, it will cause the performance of each thread drop. It shows that understanding the capability of instances is very important. In the proposed service, pre-testing is adopted for finding the maximum number of thread that each instance can execute and ensure the testing quality.
comprehensive service we will upgrade the competence of credit card. To record each test plan for the user, in the future we may add accounts system into the proposed service and every user can registered an account to record the testing result or test plan they had done. With the account system user can easily track what they had done before. ACKNOWLEDGMENT This research was supported by Chunghwa Telecom Laboratories, Taiwan under grant No. TL-102-G107. REFERENCES [1]
Yue Zhou:Wenchuang Qin, Nafei Zhu, "Challenges To Performance Testing Of The Cloud Application Developing," in Software Engineering and Computer Science.
[2]
T. M. King, A. S. Ganti and D. Froslie, "Enabling Automated Integration Testing of Cloud Application Services in Virtualized Environments," in Center for Advanced Studies on Collaborative Research, 2011.
[3]
"blazemeter," [Online]. Available: http://blazemeter.com/.
The proposed system also can achieve stress testing, form the testing result it shows the importance of load balance mechanism, a robust load balance mechanism can effectively reduce the average latency.
[4]
X. Bai, M. Li, B. Chen, W.-T. Tsai and J. Gao, "Cloud Testing Tools," in Service Oriented System Engineering, 2011.
[5]
"JMeter," [Online]. Available: https://jmeter.apache.org/.
[6]
"FIO," [Online]. Available: http://linux.die.net/man/1/fio.
Compare to other testing services, blazemeter is a famous testing service based on jmeter, it can provide powerful testing ability but it can’t deploy the testing client concurrent in different regions. Compare to the 5-layer Testing as a Service framework based on cloud infrastructure services defined by Yu, et al [4], the framework need user to upload the testing target to the platform and the proposed service don’t, according to this it can guarantee the copyright of the testing target
[7]
George Candea;Stefan Bucur;Cristian Zamfir, "Automated Software Testing as a service," 2010.
[8]
Liviu Ciortea;Cristian Zamfir;Stefan Bucur; Vitaly Chipounov;George Candea, "Cloud9: A Software Testing Service," 2010.
[9]
C. Cadar;D. Dunbar;D. R. Engler. KLEE, "Unassisted and automatic generation of high-coverage tests for complex systems programs," in Operating Systems Design, 2008.
VI.
FUTURE WORK
Since there are only 8 data centers which belong to Amazon, to support more location that the proposed service can deploy the clients, more cloud service provider should be integrated in the future. In this version the proposed system focus on test the network-based file system and HTTP request based web service test, to achieve general purpose testing we may integrate more testing tool into the proposed system to test more and more testing target that are different types. Another limitation is the max number of instance created by the system is too few it is because the competence of credit card we currently use for the AWS account is too low, to provide more
[10] P. Godefroid, N. Klarlund;K. Sen. DART:, "Directed automated random testing," in Programming Language Design and Implementation, 2005. [11] P. Godefroid, M. Y. Levin; D. Molnar, "Automated Whitebox Fuzz Testing," in Network and Distributed System Security, 2008. [12] R. Majumdar and K. Sen, "Hybrid concolic testing," in Software Engineering, 2007. [13] Z. Ganon and I. E. Zilbershtein, "Cloud-based Performance Testing of Network Management Systems," in Computer Aided Modeling and Design of Communication Links and Networks, Pisa, 2009. [14] "Cloud Testing," [Online]. Available: http://www.CloudTesting.com/. [15] "Amazon Web Services," http://aws.amazon.com/.
Amazon,
[Online].
Available:
