Using Continuous Integration and Automated Test ...

Using Continuous Integration and Automated Test Techniques for a Robust C4ISR System H.Mehmet Yüksel, Eray Tüzün, Erdoğan Gelirli, Emrah Bıyıklı,

Havelsan A.Ş., Peace Eagle Ar-Ge Team ODTU Teknokent, 06531, Ankara { myuksel, etuzun, egelirli, emrahb }@havelsan.com.tr

Abstract—We have used CI (Continuous Integration) and various software testing techniques to achieve a robust C4ISR (Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance) multiplatform system. Because of rapid changes in the C4ISR domain and in the software technology, frequent critical design adjustments and in turn vast code modifications or additions become inevitable. Defect fixes might also incur code changes. These unavoidable code modifications may put a big risk in the reliability of a mission critical system. Also, in order to stay competitive in the C4ISR market, a company must make recurring releases without sacrificing quality. We have designed and implemented an XML driven automated test framework that enabled us developing numerous high quality tests rapidly. While using CI with automated software test techniques, we have aimed at speeding up the delivery of high quality and robust software by decreasing integration procedure, which is one of the main bottleneck points in the industry. This work describes how we have used CI and software test techniques in a large-scaled, multi-platform, multi-language, distributed C4ISR project and what the benefits of such a system are. Keywords-robustnes; continuous integration; software test techniques; mission critical systems; XML driven automated test framework; C4ISR systems; multi-platform testing

I. INTRODUCTION C4ISR (Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance) applications represent an entire system including all equipment, device, methods, procedures and personnel needed for accomplishing a mission given by Command and Control units [1]. In the last decade, this term is widely used for the integration of software with hardware composed of sensors such as radar or sonar as well as weapons. In nature, a typical C4ISR project works on a large-scaled distributed environment which means there are number of processes running on different machines and communicating each other by a middleware such as CORBA, SOA or DDS (Data Distribution Service) [2]. Moreover, these processes may even run on the machines whose operating systems differ. In addition, the application might be composed of units developed under different programming languages e.g. processing units might be written in C++ while display units might be coded in Java. When all these requirements needed by a C4ISR system are considered, there are certainly a number of difficulties come up

This work was supported by Havelsan [3]

1http://www.versionone.com 2http://www.jetbrains.com/teamcity 3http://subversion.tigris.org

Prof. Dr. Buyurman Baykal Havelsan A.Ş. Peace Eagle Project Manager ODTU Teknokent, 06531, Ankara [email protected]

during the development and deployment phase of such a project. These might be listed as; • how to integrate the software components together in such a large-scaled C4ISR system, • how to guarantee such a large-scaled system working on multi-platform and multi-language, • how to guarantee such a large-scaled system is always robust and any time is deployable, • how to guarantee the whole system is free of any negative impact after refactoring the codes, • how to guarantee the whole system is free of any negative impact after changing code due to any bug fix. This works emphasize how usage of CI (Continuous Integration) and solid software automated test techniques overcome these problems for a robust C4ISR system. Brief overview of our software process is given in Section II to understand the role of CI and Test Framework in the overall project life cycle. Section III talks about how we applied CI into our project, whereas Section IV explains our Test Architecture and Procedures. In Section V and VI, we conclude our work by giving results and discussing future work. II. SOFTWARE DEVELOPMENT PROCESS Since the nature of our C4ISR project enforces us to support multiple platforms (Windows, Linux, Sun Solaris, etc.) and multi-language (C++, and Java), the complexity of our product is significantly increased. Because of these reasons, we’ve decided to put some solid process definitions in place earlier in the project cycle. The first task we have established is the adaptation of a solid project planning and management environment. The solution that we have chosen to plan and guide our development activities is a widely accepted tool, namely Version One1. This tool not only helps us to keep track of our schedule such as release and iteration planning, but also provides a medium for defect and/or issue tracking and management as an extra benefit. We strongly believed CI is the key to a successful software project. To achieve this goal, Distributed Build Management

and CI Server (Team City2) is built on top of our Version Control System (Subversion3) that also runs our automated tests. This helped us to have automated, fast, and self-testing builds in an environment where everyone is expected to commit multiple times a day. We’ve namely; • • • • •

also followed some aspects of Agile Development frequent release and iteration dates, periodic code review meetings, holding daily scrum meetings to increase communication in the group, constant refactoring and integration testing, and increased customer satisfaction with constant customer communication throughout the project.

III. CONTINUOUS INTEGRATION Integration is one of the most challenging phases in software projects. If you don’t make integration early and often, the time took to integrate will probably be greater than the time to make the real work. Since CI as a software development practice overcomes these difficulties, we have decided to apply it. Our project involves developing software across multiple platforms (Windows, Linux, Solaris, etc.) and it is developed under multiple languages namely C++ and Java. For these reasons we have decided to set up a CI mechanism on each of those platforms. For managing these projects centrally we use a tool named TeamCity. For CI to work, there should be an automatic build framework infrastructure. In our case, we preferred to use Ant [4] as build tool for the following reasons: • Based on Java so provides platform independency, • Extendible plug-in support e.g. cpptasks for C++ compilation, • Open source. To describe our CI mechanism firstly we have to define build agent and build configuration concepts. Build Agents are computers with installed software capable of running builds. Build Configuration describes a class of builds of a particular type, or a procedure used to create builds. Examples of build configurations are integration builds, release builds, nightly builds, and others [5]. For each platform, we have defined three agents running on three different machines as seen from Fig. 1. These agents also form a build chain which means they trigger each other and they use the artifact created by the previous build configurations. The build chain in Fig. 1 starts with any commit to VCS. In detail, three build configurations in the build chain are as follows;

• Unit Test Run & Code Coverage Configuration: Responsible for both building and executing unit tests. It is triggered by Build Configuration and uses the artifacts produced by Build Configuration. While executing unit tests, the coverage data is also gathered for each class. The coverage report is available for each run on the TeamCity report page. If any compilation error exists or if any unit tests fail then the build chain finishes with failure and the error log (compilation error or unit test failure) is sent to the relevant developers. If the configuration finishes without any failure then it triggers the next configuration namely Integration Scenarios Run Configuration. • Integration Scenarios Run Configuration: responsible for both building and executing integration test scenarios’ classes. It is triggered by UT Run & Code Coverage Configuration and uses the artifacts of Build Configuration. We put this configuration at the end of build chain because all the individual components must pass their unit tests before integration process starts. By running this configuration on a separate machine, we eliminated the impact of using development environment. If any compilation error exists or if any integration tests fail then the build chain finishes with failure and the error log (compilation error or integration test failure) is sent to the relevant developers. To help this build chain to work properly, we have decided to apply the following three rules: • Before committing any code, the developers need to; a. compile the whole system to detect any compilation errors, b. design and code the necessary unit tests, c. design and code the integration tests, d. run all the unit and integration tests. • In parallel to commit, the developers start to; a. design and change the smoke/deployment test procedures if needed, b. design and code the necessary benchmarking tests if needed, c. design and code the necessary distributed environment tests if needed. • After committing if any compilation errors or any test failure occurs, the relevant developer should solve the problem in less than 30 minutes or revert the check-in. As seen from Fig. 1, we use SVN(Subversion) as VCS since it supports multiple platform and open source. Specifically SVN provides code change-sets for our project management tool while it provides requirement or defect IDs during checkins. Therefore, which code parts are changed by which requirement or defect ID is easiliy traced.

• Build Configuration: Responsible for compiling both C++ and Java codes and if it’s an error free compilation then artifacts are produced for consequent uses. If an error occurs during the configuration, the build chain finishes with failure and the compilation error log is sent to the relevant developers.

2

The detailed description and benefits of the framework is provided in the next section. A.XML Driven Automated Integration/Stress Test Framework

and

Distributed

The framework permits test developers to concentrate on writing test classes. The tool accomplishes the following tasks: • Externalizing test scenarios • The automation of running the test scenarios • Launching of classes in processes with valid environment variables and class paths • Collating performance data and logs from processes to one console or log file • Distribution of the processes to different nodes for distributed testing Details for the main features of the framework are as follows. Figure 1. CI Configurations

As seen from Table I, we have collected the average build durations per configuration for each operating system. According to this, it can be said that on Linux approximately a total of 30 minutes is enough to compile, running unit tests and running integration tests while this duration is 61 minutes for Windows platform. (Difference in Build times can be mostly explained by hardware differences of different OS configurations.) TABLE I.

OS vs. Build durations per configuration

OS Build Conf. Linux 3m:10s Windows 5m:20s

Average Build Durations Integration Scenario Unit Test Conf. Run Conf. 23m:30s 8m:40s 40m:40s 15m:20s

Total 30m:20s 61m:20s

Scenario XML: Each test scenario is specified in an XML file conforming to the test framework XML grammar. The XML starts with a section for global and default variables for the test processes. For example, processTimeout variable that states a timeout for all processes is specified here. Then follows the list of process configuration sections which contains process specific configuration information. Main data in this section is main test class or executable for non-Java test classes and sequence of the process. Sequence values are used to determine the launch order of processes. A sleep value in milliseconds specifies how long to wait after launching the process before launching another. A sample test scenario xml file containing 5 processes is shown in Fig. 2.

Linux Build Server Configuration: Dual Intel ® Xeon® X5450 3.0GHz, 12 GB RAM running on Linux Ubuntu 8.04 Linux Unit Test Run/ Integration Scenario Configuration: Dual Intel ® Pentium® D 3.20GHz, 3.5 GB RAM running on Linux Ubuntu 8.04 Windows Configuration: Intel® Pentium® 4 3.20GHz, 2 GB RAM running on Microsoft Windows Server 2003

IV. TEST ARCHITECTURE AND TECHNIQUES Testing infrastructure is considered as the key area for keeping the continuous robustness of the product. Effortlessly developing automated tests makes it possible to include many different types of tests in our CI process. Because of real-time and mission critical nature of the product, it is required to test frequently to detect any degrade in performance or any bottleneck in reliability under heavy load. Multi-language (C++ and Java) and multi-platform requirements also complicates the testing process. We have developed an XML driven automated testing framework to cope with these challenges. Employing the framework along with JUnit and CppUnit testing tools enables us developing automated tests rapidly.

3

Figure 3. Testing Procedure Flow

Summary of all testing types are given in Table II. TABLE II. TESTING TYPES

Figure 2. Sample Test Scenario XML

Distributing the processes to nodes: By specifying remote node information in the process configuration section of the scenario xml, a test class can be run on a remote machine. The automation of the tests: Framework uses JUnit tool to accomplish the automation of unit integration tests independent of the programming language used in test classes. Test developer only needs to call ScenarioDriver framework class and supply a scenario xml for each test case.

Unit Tests Unit-Integration Tests Smoke/Deployment Tests Performance Tests System Integration Test

Type Automatic Automatic

Frequency Every Build Every Build

Count 454 44

Manual

Before each minor/ major release Every night Continuous

2

Scheduled Manual

20

1

The detailed description of each testing type is given in the following subsections. 1. Unit Tests Testing process starts at the beginning of the development with writing unit tests for each public method of a class. Stubs or mock objects are used. Preferably the test code is written even before the actual code. Developers required reaching coverage of at least %80 with unit testing.

B. Testing Procedure and Types Testing process starts at the beginning of the development with writing unit tests for each public method of the class. After a module is completed, Unit-Integration Tests are used to test interfaces’ of the modules. In addition, these tests are employed to test more than one integrated modules.

Running Unit tests are part of the CI process; these tests are run continuously and single unit test failure causes to halt whole build process and must be fixed immediately. Failure message is sent to all developers. Coverage metrics also calculated and displayed as part of the CI process. This procedure ensures the first level of quality of the product ant any time.

A simple application that integrates major components of the framework is developed through a scenario. This application is used in manual deployment, smoke testing and partially stress testing purposes.

When a defect is found and fixed, a unit test procedure is written to verify the fix. This is also part of regression testing which ensures that the defect will not be reintroduced in the future.

In order to accomplish performance, stress and System Integration testing, a warfare simulation game which is working on multi-platform and played with multiple participants is developed.

2. Unit-Integration Tests Unit-Integration Tests are utilized to test interfaces’ of individual modules or of combined modules. CSC interfaces and CSCI interfaces are tested as a group. These tests are executed automatically by CI Server several times e a day and result report is sent to developers.

By applying these different test techniques at different stages we aim to minimize all types of defects. The complete picture of flow of testing procedure and architecture is given in Fig. 3.

Cross-team testing technique is frequently utilized which means integration test of module A is developed by a programmer who has not worked for module A. This helps a

4

non-prejudiced and different approach to the module and thus increases the possibilty of detecting defects. 3. Smoke/Deployment Tests Refers to the first test made after a defect fix or initial assembly to provide some assurance that the system under test will not catastrophically fail. It is a preliminary to further testing that helps utilizing resources more efficiently. It is employed before each minor and major prospective software release to make an initial decision. We considered necessary to create a simple application that uses major components of the framework. By running the application, which’s screen-shot is displayed in Fig. 4 we get an initial feeling about the health of whole framework. Figure 5. Stress Test Result Report

5. System Integration Tests In order to accomplish System Integration testing, a warfare simulation game which is working on multi-platform and played with multiple participants is developed. Participants of the game can be software bots and/or real persons creating random loads and regular loads. The goal of the application is to motivate developers to find bugs while playing game, we name it bug-bashing. Also, a statistics gathering and displaying application is developed and attached to the game to evaluate performance of the system.

Figure 4. A Screen from Test Scenerio Application

4. Performance Tests Intention of the performance tests is to meausure the systems performance under specific workload. This is especially important since our C4ISR project is highly configurable and it is able to integrate with various COTS(Commerical, off-the-shelf) software products; calculating and comparing the performance metrics under various configurations is valuable. Main metric gathered is the throughput of the system since for the sytems utilizing DDS it is the most significant and meaningful parameter. In addition a Java profiling tool is used to watch and record the resources such as CPU load, memory usage and time spent in each methods. Stress tests are also used to determine the stability of the system, and utilizing them guarantees robustness, availability and no catastrophic crashing under heavy loads. Some of the tests are administrated in distributed multi-platform environment including Linux, Solaris and Windows installed nodes. The result reports of the tests, an example is shown in Fig. 5. are saved for future comparisons.

V. RESULTS AND CONCLUSION With a strong CI and Testing Framework in place, the statistics and observations listed below prove that, although recently launched, the framework succeeded with goal of achieving a robust, agile and continuously improving C4ISR infrastructure. •

•

•

• • •

We are now capable of issuing patch releases once a week and minor releases once every other month with minimal number of defects reported by our client. Project has rapidly grown to a relatively large size consisting of 1624 files and 361209 lines of code. With an average of 4.6 revisions per file, and average of 33 commits per day, it’s clear that developers are comfortable constantly refactoring code base. For such a fast-moving project it’s crucial to continuously integrate changes while avoiding the introduction of new & regression bugs. Code-base is more robust: The success rate for the builds before adapting CI and test architecture is %72 while this number increases rapidly to %93 according to TeamCity statistics, Short Build and Integration times (around 30 minutes), Increased number of automated tests (454 unit tests and 44 integration tests) When a new defect is found, it is usually related to the lack of tests (unit and unit integration tests).

5

Therefore, we know that we just have to improve existing tests or add new tests. VI.

FUTURE WORK

First of all, we plan to increase the coverage rate to %90 by adding new unit tests. More over we would like to increase the number of integration tests and decrease the duration of build chain. Secondly we are planning on extending CI paradigm by adding Static Analysis reports. By Static Analysis, it is meant that measuring the code quality and discovering error-prone codes every time any code change is committed to VCS. Furthermore, we intend to adapt Dynamic analysis tools in order to measure the performance and to solve the bottlenecks

as soon as possible by running those tools on the test scenarios prepared for performance tests. ACKNOWLEDGMENT The authors would like to thank to all the contributors to the Peace Eagle Ar-Ge Team, namely E. Şaykol, D. Saran, M. Şahin, İ. Kılınç, E. Er, G. Gezici, H. C. Gören, A. Kanber, H. N. Karaca, E.Yılmaz, N. Vuran and S. E. Şahin. REFERENCES D. S. Alberts, R. E. Hayes, “Understanding Command and Control,” CCRP Publications, 2006, pp.31-33 [2] O. M. G. (OMG), “Data distribution service (dds) specification,” v1.2, January 2007, http://www.omg.org/docs/formal/07-01-01.pdf. [3] Havelsan; http://www.havelsan.com.tr [4] Ant; http://ant.apache.org/manual/index.html [5] TeamCity Online Documentation; http://www.jetbrains.net/confluence/display/TCD4/TeamCity+Documentation [1]

6