Software Verification Considerations for the ARTIS ...

Software Verification Considerations for the ARTIS Unmanned Rotorcraft Christoph Torens ∗

Florian-M. Adolf †

= = = = = = = = = = = = = =

Autonomous Rotorcraft Testbed for Intelligent Systems Digital Signal Processing And Control Engineering, software tool, used for HITL simulation European Organisation for Civil Aviation Equipment Extended Backus-Naur Form, a notation for expressing context-free grammars Flight control computer Hardware-in-the-loop, a technique to test embedded systems Integration master, a role in the ARTIS development process Module master, a role in the ARTIS development process Operating System of a computer or embedded system Radio Technical Commission for Aeronautics Software-in-the-loop, a technique to test software Subversion, a software versioning and revision control system Unmanned aerial vehicle Unified Modeling Language, a standardized general-purpose modeling language

ev iew

ARTIS dSpace EUROCAE EBNF FCC HITL IM MM OS RTCA SITL SVN UAV UML

Re vi

Nomenclature

sio

n

This work presents the processes and tools that were installed and developed to validate the ARTIS software and achieve compliance of an unmanned rotorcraft testbed with corresponding standards. A brief introduction to the autonomous guidance and navigation capabilities of our unmanned aircraft is given in order to illustrate the software complexity and practical integration challenges introduced by such functionalities. Our software development process is presented which is aimed at a practical balance between exhaustive testing and the rapid integration of new features. It features a greedy integration procedure that is aimed at the preservation of existing features and performances. Automated tests drive the development of our mission planning, mission management and sensor fusion systems. New research code can be integrated such that side effects on existing systems are minimized.

I.

Introduction

Pr

The ARTIS project works on creating a UAV platform that aims at maximized onboard information processing and decision making in cooperation with human operators. But with the increasing degree of autonomy, software becomes more and more important. Naturally, the code base is becoming more and more complex. Therefore efforts have been made to keep the code quality high. Furthermore, aerospace is a safety-critical domain and corresponding standards as well as verification techniques must be able to cope with the growing complexity of the software and the high demands for correctness. Within the aerospace domain, the DO-178B document [25] is the standard to follow when developing software. Recently, a new DO-178C standard [26] has been issued by RTCA and accordingly European counterparts by EUROCAE, as well as corresponding supplements [27–30]. These documents address some of the challenges that emerged from the demand for growing complexity of software projects. In the UAV community there is a high interest in UAV airworthiness certification, especially the software development and verification is a problem. Complying with these standards is not easy, especially for UAVs, since the software is inheriting tasks from a pilot and thus, large parts of the software are safety-critical. This paper discusses the problems faced when developing a UAV as a research platform and when aiming to achieve DO-178C compliance. The introduction in this section is followed by an overview of the ARTIS platform in section II. The software development process is specified in section III and the software verification is described in section IV. Section V describes the ARTIS software-in-the-loop and hardware-in-the-loop test setup. The software considerations are discussed in ∗ German † German

Aerospace Center (DLR), Institute of Flight Systems, Dept. Unmanned Aircraft, Braunschweig, Germany, AIAA Member. Aerospace Center (DLR), Institute of Flight Systems, Dept. Unmanned Aircraft, Braunschweig, Germany, AIAA Senior Member.

1 of 15 American Institute of Aeronautics and Astronautics

section VI. In section VII a comparison of DO-178C to the software practices of the ARTIS development is discussed. Finally section VIII concludes this paper.

II.

Application Background

sio

n

In this section two UAVs are presented in more detail. The research rotorcraft midiARTIS [16] has been developed for automated flights. Later, the larger helicopter maxiARTIS has been developed to improve endurance and payload capacity. Both helicopters (Table 1) follow a modular avionics concept inspired by Dittrich [15]. Sensor equipment comprises DGPS, acceleration sensor and magnetometer for navigation, sonar altimeter for landing, wireless data links, manual remote control and video data links, a telemetry module and a flight control computer. Additionally, a variety of cameras or laser sensors can be installed and connected to a dedicated image processing computer. The computers and the camera are mounted on a adjustable fixture to be exchanged easily. Its role as a flying testbed enables the evaluation of a variety of approaches to adaptive flight control, vision-based navigation, environment perception and onboard decision making. At the moment, the computing power is limited by the maximum weight the rotorcraft can be equipped with. But we are in the process of enhancing our fleet with a 150 kg class helicopter superARTIS based on a Swiss UAV a to further improve endurance as well as the computing capabilities, like on-board sensor fusion. Table 1. The ARTIS flight test vehicles [10].

maxiARTIS

Re vi

midiARTIS

2003

2006

rotor diameter engine endurance

1.9 m 3.5 kW combustion engine 15–20 min

3m 5 kW turbine engine 30–45 min

empty weight avionics weight experimental payload

6 kg 4 kg 2 kg

15 kg 4 kg 6 kg

ev iew

first flight

flight control computer image processing computer

Intel P4 mobile 1.4 GHz, QNX OS Intel P4 mobile 1.4 GHz, Linux OS

Pr

A general overview about ARTIS has been given by Adolf et al. [10] where the following onboard software components and functionalities for increased autonomy are explained in detail: • Payload-directed mission elements [(vision-based gate passing, vision-based ground vehicle tracking) 9] • Real-time world modeling and path re-planning [5, 11] • Online exploration behavior in urban terrain [4] • Usage of a large urban terrain database [(city of Berlin) 7] • Composite flight behaviors that solve abstract guidance tasks [(search pattern planning and safe perimeter tracking) 3] • Trajectory following of linear segments and cubic splines [21] • Evaluation of the roadmap-based path planning [(performance, smoothness, safety) 6] a http://www.swiss-uav.com


III.

ARTIS Software Development

In this section the currently used methods for software development are presented. The software development handles the topics of general software development, coding guidelines, system and software requirements, bug tracking, trace management, software integration and software verification.

sio

n

A. System Concept and Software Requirements We established a comprehensive ARTIS software concept, i.e., goals, strategy, used operating systems and tools as well as target platforms. Also coding guidelines for the development of C, C++ and Matlab code, style guides and naming conventions are defined. For specific projects the system requirements are being captured explicitly, but in general the system requirements are subject to research and thus mature and evolve over time. This means, selected modules have been in a designated requirement analysis phase and the requirements are explicitly captured in a separated documentation, but in most cases, the requirements are captured incrementally, i.e., developed and discussed collaboratively using a ticket system. High-level as well as low-level requirements are captured using a modified Mantis bug tracking server [2]. Also bugs are tracked via this system. The bug submission and change management process is described in Fig. A.1, page 13. Several stages are defined for this process, which is executed by developers with the roles of problem reporter, developer and module manager. This way, each code change is the result of a dedicated Mantis ticket number and all source code is developed from Mantis tickets.

Re vi

B. Configuration Management Source code management is done by a central SVN [23] server, but with green, yellow and red code repositories with own branches, each. The green repository software version, which is always stable, is used in flight tests and demonstrations. Except for the dedicated integration phase, it is not allowed to commit changes to this repository. The yellow repository version is the latest working code that is planned to be integrated in the next green version release. The ARTIS project is composed of dedicated modules, e.g. for navigation, path planning and sensor fusion. Each module has a single designated module master (MM). Only the corresponding module master is allowed to check in code for his module. Other developers send code changes as patch files to the module master, who then reviews the new code, assures that the development strategies are met and performs additional tests (Fig. A.2, page 14). Finally, the red repository version of the code is viewed as the code currently under development by a developer.

ev iew

C. Software Integration Process The software integration phase usually takes place once each month. All new code to be integrated into the green version should be tested sufficiently by the respective module masters and developers. The procedure flow diagram is depicted in Fig. A.3, page 15. 1. In a meeting, the MMs present which code of the yellow version shall be integrated and the integration master (IM) is chosen. The IM is responsible for planning an efficient and smooth workflow of the integration days, and supervises its execution. 2. The IM schedules the integration of the presented code changes. Output of this is a rough timetable containing an overview of the integration steps.

Pr

3. The integration schedule is executed stepwise. Each code change is integrated on a local system. IM and MM are executing pre-tests to ensure at an early stage that the new code is correctly integrated and has no negative effects on the complete system. This release candidate is committed to a new branch of the green repository. 4. The system test is performed following the updated test protocol. The system test checks all required functionalities of the system. If the test fails, the code change responsible for the failure has to be fixed or may be excluded from integration and the test needs to be repeated completely. If the test succeeds, the local integrated version and the completed test protocol and test results are committed to the green repository.

IV.

Software Testing and Verification

This section describes the software verification process. Every developer is responsible for his own code and module masters are additionally responsible for the whole module. Therefore, modules are already thoroughly tested individually by the module masters before the integration tests (sec. III.C) are started. As examples, the testing strategies of two different modules will be presented, which show different approaches. 3 of 15 American Institute of Aeronautics and Astronautics

Re vi

sio

n

A. Sensor Fusion and State Estimation This module is responsible for integrating sensor information of magnetometer, accelerometer and GPS and to compile a position estimation of the UAV in the world. The module was originally written in C. In spite of a good result of the position estimation it was later completely rewritten in C++ for maintainability and compatibility reasons. The problem was to check, if the new module had still the same high quality position estimation in its results and was not less precise. The approach taken to verify this and to compare the results of the new module with the old one was that of comparison / regression testing. Log-data of the relevant sensors was pre-recorded in several flight tests, as well as the results of the previous implementation of the position and attitude state estimation (Longitude, Latitude, Height, Φ, Θ, Ψ) [33]. These data are stored in Matlab data files and used for regression testing: the data are fed into the new state estimation filter and the output is compared to the previous results. If the results of the new module differ from the pre-recorded results, an error is produced. To achieve this, the Matlab data are used to generate a set of unit tests for each flight. For the unit tests, the Boost Test library [1] is used. Within these tests, every logged value of the result is compared to the new position estimation. So the test can thoroughly compare the two implementations and deviations too large are logged for each value. Because the results usually are not exactly the same, but instead are very close, a version of the BOOST CHECK macro is used that evaluates true for a tolerable percentage of deviation. The following test excerpt (Fig. 1) further illustrates this. For the comparison of orientation, also the case has to be handled when the angle gets near max or min values, because the max and min angle are the same logical value. This test assures that the implementations are close. But this test shows more. Because the previous implementation was approved correct, these tests can now be used as a form of regression tests against stated requirements and the allowed deviation can be limited. The advantage of this test is that it uses real flight data instead of synthetic data and also that lots of test data are available in this form. But nonetheless, simulated flight inputs are also tested in a different test set. #define N_DIFFPERC 0.001 BOOST_CHECK_CLOSE(pSF_via_c_wrapper->pos[0], pNavfilter->pos[0], N_DIFFPERC); BOOST_CHECK_CLOSE(pSF_via_c_wrapper->pos[1], pNavfilter->pos[1], N_DIFFPERC); BOOST_CHECK_CLOSE(pSF_via_c_wrapper->pos[2], pNavfilter->pos[2], N_DIFFPERC);

ev iew

Figure 1. Excerpt of the regression test scripts comparing two implementations of state estimation [33].

B. Mission Management The mission management software is a central component coordinating all aspects of a mission as the operator commands the mission. A synchronous, event-based system model is used to coordinate incoming events and it is implemented using UML state charts. The system works with multiple threads of execution, to counter the case where a few resource-intense computing tasks take up all CPU time. The overall complexity of the sequence control system’s software implementation imposes a thorough test strategy. The testing process accounts for use cases and must address theoretical problems that can occur in finite state machines and state charts, respectively. Basically, there are four testing stages during the development of the onboard mission management system:

Pr

1. Abstract tests assure that the model is free of principle errors. 2. Unit tests assess the functionality of an individual software component or a composite component. 3. Simulation provides the infrastructure for operational tests that assess the overall system integration. 4. Flight tests finally provide a practical proof of concept for the real-world application and validate run-time and error-prone data-link behavior as well as used hardware thresholds.

In the modeling stage a set of errors can occur. Some relate to potentially isolated or unreachable states, others to missing or erroneous triggers and guards. An even more fundamental problem is the theoretically infinite set of sequences that have to be tested in an abstract manner. That means that regardless of the meaning and function behind events, states and transitions, the tests must traverse through the model even if a certain test makes no sense from a practical point of view. Abstract tests relate to model-based testing [35]. In the case of the sequence control system, the test model can be derived from the state chart model. UML state charts represent a constructive model [31], such that tests can make use 4 of 15 American Institute of Aeronautics and Astronautics

of its internal structure (known as white-box testing). These abstract tests can be divided into three groups, namely the path coverage, transition coverage and state coverage tests as highlighted in Fig. 2. For each of these groups, a set of event chains is generated and executed. Once a full coverage of all possible combinations is reached, the tests are completed. However, there is no defined final state and states can have loops, as such infinitely long test chains for path coverage tests result. Therefore, a relaxation for the path coverage criteria is implemented such that loops must be passed only once. This is a strong and feasible criterion [35].

Sequence 2

Sequences 3

Sequence 4

Sequence 1

Sequence 3

b) state coverage

Sequences 3,4

Sequences 5,6

Sequences 7,8

Re vi

Sequence 2

sio

a) transition coverage

Sequence 1

Sequence 2

n

Sequence 1

c) path coverage

Figure 2. Abstract testing strategies for state charts based on coverage of a) transitions, b) states and c) paths.

ev iew

Note that the use of Petri nets for the execution model has been investigated [8]. However, the modeled sequence control system is synchronous and the software development support for Petri nets appeared less mature compared to UML modeling tools. Thus a Petri net approach is not continued until more concurrency will be involved in the sequence controller model, which is where Petri nets are particularly strong. Missions are defined as sequences of parameterized behavior commands. Although it is not possible for the mission management to reason about if a mission plan makes “sense”, it is possible to implement a plausibility check. These checks are implemented using the EBNF grammar [19] shown in Fig. 3, which ignores malformed behavior sequences that cannot be matched against it. EBNF grammars are formal grammars of logical production rules that comprise nonterminal symbols, similar to place holders or variables, and terminal symbols, which are generated by the grammar. Grammars are widely used to implement compilers for programming languages, and, as such, a behavior sequence is a sequential program. The grammar implemented in this context defines production rules for each behavior command or task flag, such as a “hover to” or “tracker off”. Its root is defined by “” which basically enforces every mission to start with a take-off behavior and end with a land behavior. It implements some basic parameter checks, for instance for the “hover and wait” command where the waiting time must not be a negative number. Furthermore, the grammar allows a check for any necessary prerequisites for specific behaviors. For example, the trajectory planner for the forward flight needs at least three of its behavior commands.

V.

From Desktop Ideas Towards Flight Testing

Pr

New approaches to flight control and mission management for the ARTIS platform are developed and tested in a Matlab/Simulink environment (Figure 4). This can either be done using only the simulation model or it can include the whole on-board flight software. This software-in-the-loop test simulates the vehicle dynamics and emulates sensor inputs and interfaces to the ground control software via virtual wireless data links. From simulation to flight test always the same code basis is used for all tests. The integrated hardware and software is tested hardware-in-the-loop (HITL) before the flight. During these tests, the UAV itself as well as the flight control equipment are exactly the same as used later in flight tests. Sensor inputs and actuator outputs are connected to a real-time environment (Fig. 5) that simulates the aircraft flight mechanics and emulates all sensors. Thus it is possible to test the guidance, navigation and control (GNC) software and the operator’s control interface under realistic conditions. Furthermore, scenarios can be simulated that currently cannot be performed in real flight tests, e.g. due to law or resource constraints. The output of the vision algorithms and the interaction with the flight control is also tested in HITL. The image processing testing is performed in its own rig. Cameras capture images that are generated by a virtual 3D scene (Figs. 6, 7) which is projected on one screen for a


"ID" [] ; { | "0"}; "1"|"2"|"3"|"4"|"5"|"6"|"7"|"8"|"9"; [ [] "." ] ; { | "0"}; { | | }; { | | }; { | | | }; "TO" [ "-" ]; | | ; "HV" ["-"] ["-"] ["-"] ["-"]; "HT" ["-"] ["-"]; "WT" []; "WO"; { }; "FT" ["-"] ["-"] ["-"] ["-"]; "PI" ["-"] ["-"] ["-"] ["-"]; "PF" ["-"] ["-"] ["-"] ["-"]; "on" "off"; "tracker" | "avoidance"; "LD";

n

::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::=

sio

ID 20070510 TO -5 HV 0 0 -3 180 avoidance on WT 10 HV 33.3 3.51 -7 28.9 HV 37.1415 1.63484 -10 90 avoidance off WT 5 tracker on HV 37.1415 48.688 -10 90 HV 37.1415 48.688 -10 180 HV 31.81 48.688 -10 180 HV 26.4785 1.63484 -10 180 HV 26.4785 1.63484 -10 90 HV 26.4785 48.688 -10 90 HV 26.4785 48.688 -10 180 HV 21.147 48.688 -10 180 tracker off LD WO

Figure 3. The mission (left) is an example for a behavior sequence that schedules a mission plan, including a pattern tracking section and a reactive obstacle avoidance section. The EBNF grammar (right) specifies the syntax for plausible behavior sequences.

State

Vehicle Simulation Model

Sensor Raw Data

Sensor Drivers

Control

Actuator Simulation

Re vi

Sensor Emulation

Actuator Raw Data

Sensor Data

Navigation Filter Telemetry

State Estimate

Flight/Mission Control

Control

Actuator Driver

Instructions

Ground Control Station User Interface / Mission Planning

Simulation Computer

ev iew

Figure 4. The software-in-the-loop setup.

Sensor Emulation

State

Vehicle Simulation Model

Control

dSpace Actuator Raw Data

Sensor Raw Data

Sensor Drivers

Sensor Data

Navigation Filter

State Estimate

Telemetry


Control

Actuator Driver

FCC

Instructions

Ground Control Station User Interface / Mission Planning

Pr

Actuator Simulation

Ground Control Station Computer

Figure 5. The hardware-in-the-loop setup [10].

monocular camera and on two screens for a stereo camera. The screens are captured by cameras that are connected to the image processing computer. If required, the virtual camera views can be distorted to emulate more realistic image disturbances (e.g. brownout noise, lens distortions). The loop is closed by connecting the vision simulation to the flight controller.

VI.

UAV Software Verification and Compliance Challenges

Currently, regulation of UAVs in Europe with a weight exceeding 150 kg is supervised by EUROCONTROL whereas regulation of small UAVs, i.e., below 150 kg is left for national authorities [12, 34]. In Germany, this national


Navigation Filter

State Estimate


HITL or SITL +Ground Control Station

Navigation Data

Data Exchange

Additional Instructions

3D Visualization

Image Capture

Image Processing World Model

Vision Computer

Re vi

sio

n

Figure 6. The vision simulation setup [10].

Figure 7. Stereo HITL.

ev iew

regulation is based on the line of sight of a human safety pilot [13, 18], not generally trusting the software pilot or the abstraction of sensory information by the remote system. Software is not considered by these regulations. But for the long term, the concept of the safety pilot that has to have the UAV in his direct line of sight has to be discarded. To overcome this limitation, basically the autonomy of the software pilot as a whole has to be thoroughly modeled and certified for safety. The adequate verification of these autonomous functions against given requirements is therefore the big challenge. And last but not least, current regulation of UAVs is by far not finished [12]. There are still many unsolved problems concerning the integration of UAVs into segregated as well as non-segregated airspace, which is an especially complex matter. This leaves, next to technological problems, several uncertainties and even unfinished regulatory requirements. The aerospace domain is a safety-critical domain. This enforces high demands on software engineering and expensive compliance with complex standards, e.g., DO-178C. At the same time, UAVs and autonomous flight are subject to active research. This results in highly agile and complex requirements and the demand for complex modeling techniques. And with complex modeling techniques, verification is itself getting more complex. Therefore, there is a growing demand for the verification of more and more complex systems.

Pr

A. Safety-Critical Software Standards As computer science, programming languages and formal methods have had major improvements over the last decades it is difficult for standard documents to cope with new technology. For example object-oriented techniques, like C++ have been the standard programming features for many years, but the DO-178B [25] standard did not quite account for this. Although the DO-178B standard is written in an open fashion that leaves specifics open for an individual interpretation, some language features in object-oriented languages have implications that are not easily handled in the mandatory verification and tracing aspects of DO-178B [14]. Only as of late 2011, with the publication of DO-178C, there is a dedicated supplement DO-331 [28] to the standard, targeting object-oriented languages. The same statement is true for formal methods and model-based techniques, which could significantly improve the verification results. But achieving certification credit is a problem. The DO-178C standard was published nearly 20 years after its predecessor. This highlights that it is important for a standard that significant technological improvements, like object-orientation or model-based techniques, are given prompt attention [32]. Cost reduction is always a problem. DO-178C is a heavyweight standard that depends on firmly established processes and plans producing a lot of documentation. On the other hand, agile methods are getting more and more attention in industry. For example, small teams and lean development processes are usually not compatible with the DO-178C standard. Therefore, the development of cheap UAVs for a civil market segment will be extremely problematic. Even if hardware costs can be reduced, the costs for compliance to software standards and verification 7 of 15 American Institute of Aeronautics and Astronautics

are huge and are not profitable for small UAV quantities. As a result, Agile DO-178 is an interesting topic [36], trying to integrate agile methods into DO-178C and thus making development more productive and cost efficient. A question is, whether an ultra-light UAV will have to meet the same certification criteria as a large passenger machine, i.e., applying DO-178C with the same rigor. DO-178C defines five software levels (Level A - Level E), which in turn are correlated with five failure condition categories (Catastrophic - No Safety Effect). The higher the criticality level, the greater the amount of software development effort that is required. The highest criticality level copes with the loss of the aircraft, implying multiple fatal injuries of passengers and crew. For UAV systems however, there is no danger of injury for crew or even passengers. Therefore, the criteria for possible dangers and thus the failure condition categories and resulting software levels necessary need to be adapted for UAV systems. Two approaches are presented in literature [17, 22] which propose appropriate failure conditions.

VII.

Re vi

sio

n

B. Testing and Assessment of Autonomous Functions Software testing itself is not a trivial process, testing of autonomous functions is even more demanding. Especially when requirements are not fully available and regulation is not standardized. One question is whether a UAV operates safely and fulfills functional requirements. But another question is, how good a task or a requirement is fulfilled, e.g., how good was the path-planning, was it the shortest path possible, was it the quickest path, how was the quality of sensor fusion and obstacle avoidance. A challenge is therefore, to find software tests and metrics that can be used to automate the tests and also fitness functions for the assessment of such autonomous methods. For example, for path-planning there exist benchmarks that are used to measure the quality by expressing a path deviation from a standardized baseline solution. This enables developers not only to test a path-planning algorithm automatically, without a manual review from an engineer, but also it is possible to evaluate algorithms and compare them to find even better solutions. Possible criteria for quality metrics are robustness, real-time behavior and computation time of non-linear filters. For the stability criterion, it could be possible to provide proofs of allowed value margins of inputs in which the algorithm works correctly or with predetermined accuracy.

Application of DO-178C to ARTIS

Pr

ev iew

The DO-178C [26] is a standard document that describes guidelines how software should be developed to fulfill the software demands of airworthiness requirements. The major difference [20] to the predecessor document are supplements, which describe actual development and test procedures and the detailed integration of these techniques with the principles of the proposed methods of DO-178C. These new methods are model-based techniques [28], object-orientation [29] and formal methods [30]. Furthermore, there is a document for tool qualification [27]. As shown in previous sections, the ARTIS development methods emphasize quality and each software module is thoroughly tested. However, these practices follow no standard process. Until now, there was no special attention given to compatibility with aircraft software standards, mainly because of complexity concerns. Therefore, there are large gaps to a safety-critical standard as DO-178C. Nonetheless, it is the standard to follow in aircraft development and this paper is intended as a baseline study to investigate what would be necessary to achieve DO-178C certification with a research platform for UAVs as the background. In this context it is interesting to ask what is necessary for our ARTIS UAV and its development life cycle to attain standard compliance. A mapping between the institutes methodologies and DO-178C, respectively corresponding gaps and issues, is evaluated. The DO-178 standard is intended to be followed from the start of the development process. The data items are produced during several stages within the development. Application of the standard after the completion of development is not intended. Thus, achieving compliance afterwards needs intense efforts for reverse engineering of data items, as shown in the case of life cycle data [24]. A. Software Level Determination A key aspect and the basis of all processes of DO-178C is the software level determination of software components. The software is essentially replacing the pilot and therefore large parts of the software in a UAV are safety-critical. With current regulations in Germany, a dedicated safety pilot which has the UAV in sight is assumed to be present at all times. The safety pilot can switch the UAV to remote control at any incident. As a result, the software can technically be classified in the lowest category “no safety effect”. However, this fact is not desirable as stated earlier and to evaluate DO-178C compliance, at least level D has to be considered. In the following, this level will be used to compare the processes. On this level, there are 26 process objectives to achieve with 46 data items to produce. In


comparison, there are 71 process objectives on level A and the effort to comply with the standard increases heavily with each level.

Re vi

sio

n

B. Software Life Cycle The DO-178C software life cycle process encompasses the complete software development lifetime and consists of the following subprocesses. Each of these processes will be shortly addressed in the remaining section to compare the presented practices with DO-178C. The structure of the processes is depicted in Fig. 8.

Figure 8. DO-178C processes.

Software Planning Process

ev iew

1.

DO-178C requires the following five key plans: Plan for Software Aspects of Certification (PSAC), Software Development Plan (SDP), Software Verification Plan (SVP), Software Configuration Management Plan (SCMP) and Software Quality Assurance Plan (SQAP). The main aspects of the software planning processes SDP, SVP and SCMP are largely resembled in the ARTIS software concept described in section III.A and in the processes of change management (Fig. A.2, page 14) and integration management (Fig. A.3, page 15). However, there is no counterpart to the DO-178C SQAP and PSAC. 2.

Software Development Process

Pr

The ARTIS software design is an ongoing cycle. Requirements are created and managed in an agile fashion via Mantis (see sec. III.A). Also the software coding process is managed via Mantis. Whenever code is checked in to the version management system, the corresponding ticket number for that requirement is noted in the commit log in addition to a description of what is committed. In this way, a requirement can easily be traced to the code, simply by viewing the diff of the commit. The problem with this approach is that although the diff always stays the same, the code is evolving. Therefore, the latest code may not correspond to the diff, but in this case the successive commits can also be retraced step by step via Mantis. The software integration process was described in section III and maps closely the DO-178C process. 3.

Software Verification

For safety-critical software, the software verification process is often the most complex one. Also in DO-178C this process is given the main focus, as it encompasses 43 of the 71 DO-178C objectives. The main process is divided into five subprocesses, the first four are each verifying a software development process output and the fifth subprocess is verifying the verification results themselves.


4.

sio

n

Verification of the software requirements to system requirements is difficult, because these are handled agilely. However, the change management process requires a requirement (or ticket) to be confirmed by another developer. Therefore, the objectives of the requirement verification can be integrated into this step. The same applies for the software design verification, which would be covered by the requirement confirmation step in equal measure. Every code committed to the repository is reviewed by the corresponding module master. This maps to the verification of outputs of the software coding process. The ARTIS verification process emphasizes unit, integration and system tests. The software verification was described for two example modules in the previous section. This maps to verification of the software integration process. The fifth verification process, the verification of the verification process itself can only be mapped incompletely to our own processes. The coverage of the model-based test was described in section III, but coverage of other tests is planned for the future. As an easy way to trace tests to the requirements it is planned to also commit Mantis tickets for each unit test. This works as a coverage measure, because the test and code trace to the same Mantis ticket number. The new supplement DO-332 [29] allows object-oriented programing languages, but requires verification of local type consistency and dynamic memory management. These aspects are not considered yet, as it is not necessary for level D compliance. Software Configuration Management

5.

Re vi

The objectives of this process are mostly covered by the ARTIS development processes. Most notable, configuration items are identified as software modules. Baselines are managed via color-coded software levels, green version, yellow version and red version. Problem reporting, change control and change review processes have been described and depicted (see Fig. A.2, page 14). The actual configuration management is handled by SVN [23]. Software Quality Assurance and Certification Liaison

Software quality as described earlier is of utter importance, but quality assurance compliant with DO-178C is only investigated as of late. Unfortunately, within the currently used practice, there is no such dedicated process established yet. Certification liaison is a close interaction with the certification authority and since no software components have been attempted to get certified, such a process is not established yet.

ev iew

C. Review of the ARTIS Processes The presented ARTIS processes emphasize on code quality but are not sufficient for a DO-178 compliance. Although many of the process objectives are considered, the thoroughness of these processes is not good enough in many cases, even for level D compliance. Especially the overall documentation and the different traceability objectives are an important element of the standard and would have to be enhanced. We still believe the ARTIS processes provide a solid basis for the use as a DO-178 compatible standard but the processes need further improvements in order to achieve compliance.

VIII.

Conclusion and Outlook

Pr

We showed the ARTIS UAV research platform and described in detail our software development process. The illustrated methods already emphasized several aspects of software quality, although they focused on lean and manageable processes. This approach was then put in correlation with the processes of the new DO-178C standard. Additionally, we discussed problems that arise when fully integrating the DO-178 standards in UAV development and especially the lean processes of a scientific research platform. The overall goal is to achieve high quality and maintainable software and to find lightweight, automatic and deterministic approaches to model, verify and benchmark our software. The standards for aircraft certification of software are DO-178B and the new version DO-178C. But, even if we showed some parallels between our software development approach and DO-178C, it has several large gaps when it is compared to the mentioned standard, even for the lowest software criticality level. Our goal is therefore to develop a new software module for ARTIS from the beginning in complete fulfillment of the DO-178C standard to understand the full extent of the implications and obstacles when developing DO-178 compatible software with an agile mindset.


References [1] Boost Unit Test Suite. www.boost.org/doc/libs/1 50 0/libs/test/, August 2012. URL www.boost.org/doc/libs/1_ 50_0/libs/test. [2] Mantis Bugtracker. www.mantisbt.org, August 2012. URL www.mantisbt.org. [3] F.-M. Adolf. Behavior-based High Level Control of a VTOL UAV. In Proceedings of the AIAA Infotech@Aerospace Conference, Seattle, WA, USA, April 2009. [4] F.-M. Adolf. Multi-Query Path Planning for Exploration Tasks with an Unmanned Rotorcraft. In Proceedings of the AIAA Infotech@Aerospace Conference, Garden Grove, CA, June 2012.

n

[5] F.-M. Adolf and F. Andert. Rapid Multi-Query Path Planning for a Vertical Take-Off and Landing Unmanned Aerial Vehicle. AIAA Journal of Aerospace Computing, Information, and Communication, 8(11):310–327, November 2011.

sio

[6] F.-M. Adolf and J. Dittrich. Evaluation of the ARTIS Sampling-Based Path Planner Using an Obstacle Field Navigation Benchmark. In Proceedings of the 68th AHS Annual Forum, Fort Worth, TX, USA, May 2012. [7] F.-M. Adolf and H. Hirschmüller. Meshing and Simplification of High Resolution Urban Surface Data for UAV Path Planning. Journal of Intelligent and Robotic Systems, 61(1-4):169–180, 2011.

Re vi

[8] F.-M. Adolf, A. Flohr, and J. R. Müller. Formale Modellierung des Entscheidungssystems für einen autonomen Hubschrauber Beschreibungsmittel, Methoden, Werkzeuge und Anwendungen. In U. Jumar, editor, Entwurf komplexer Automatisierungssysteme (EKA 2008), number 10, pages 353–362. IFAK Magdeburg, Magdeburg, April 2008. [9] F.-M. Adolf, F. Andert, L. Goormann, and J. Dittrich. Vision-Based Target Recognition and Autonomous Flights Through Obstacle Arches with a Small UAV. In Proceedings of the 65th AHS Annual Forum, Grapevine, TX, USA, May 2009. [10] F.-M. Adolf, F. Andert, S. Lorenz, L. Goormann, and J. Dittrich. An Unmanned Helicopter for Autonomous Flights in Urban Terrain. In T. Kröger and F. Wahl, editors, Advances in Robotics Research: Theory, Implementation, Application, volume 9, pages 275–285. Springer Berlin / Heidelberg, June 2009. [11] F. Andert and F.-M. Adolf. Online World Modeling and Path Planning for an Unmanned Helicopter. Autonomous Robots, 27 (3):147–164, 2009.

ev iew

[12] K. Dalamagkidis, K. P. Valavanis, and L. A. Piegl. On Integrating Unmanned Aircraft Systems into the National Airspace System. Springer Netherlands, Dordrecht, 2012. ISBN 978-94-007-2478-5. doi: 10.1007/978-94-007-2479-2. [13] Deutsche Flugsicherung. Nachrichten für Luftfahrer Gemeinsame Grundsätze des Bundes und der Länder für die Erteilung der Erlaubnis zum Aufstieg von unbemannten Luftfahrtsystemen gemäß§16 Absatz 1 Nummer 7 Luftverkehrs-Ordnung (LuftVO). 60. Jahrgang, Langen, NfL I 161 / 12, June 2012. [14] R. Dewar and C. Coma. Certification of Object Oriented Programs. In Proceedings of the 2nd Institution of Engineering and Technology International Conference on System Safety, pages 95–99, 2007. [15] J. Dittrich and E. Johnson. Multi-sensor navigation system for an autonomous helicopter. In Proceedings of the IEEE 21st Digital Avionics Systems Conference, volume 2, pages 8C1–1 – 8C1–19 vol.2, October 2002. doi: 10.1109/DASC.2002. 1052941.

Pr

[16] J. Dittrich, A. Bernatz, and F. Thielecke. Intelligent systems research using a small autonomous rotorcraft testbed. In Proceedings of the 2nd AIAA Unmanned Unlimited Systems, Technologies, and Operations Aerospace Conference, San Diego, CA, USA, September 2003. [17] K. Hayhurst, J. Maddalon, P. Miner, G. Szatkowski, M. Ulrey, M. DeWalt, and C. Spitzer. Preliminary considerations for classifying hazards of unmanned aircraft systems. Technical Report NASA/TM-2007-214539, NASA, 2007. [18] O. Hirling and F. Holzapfel. Applicability of military UAS airworthiness regulations to civil fixed wing light UAS in germany. In 61. Deutscher Luft- und Raumfahrtkongress 2012, 2012. Berlin, 2012.

[19] International Organization for Standardization, ISO/IEC 14977. Information Technology Syntactic Metalanguage Extended BNF. 2001. URL www.iso.ch/iso/en/CatalogueDetailPage.CatalogueDetail?CSNUMBER=26153. [20] S. A. Jacklin. Certification of Safety-Critical Software Under DO-178C and DO-278A. In AIAA Infotech@Aerospace Conference, Garden Grove, CA, 2012.


[21] S. Lorenz and F. M. Adolf. A Decoupled Approach for Trajectory Generation for an Unmanned Rotorcraft. In F. Holzapfel and S. Theil, editors, Advances in Aerospace Guidance, Navigation and Control, pages 3–14. Springer Berlin Heidelberg, 2011. ISBN 978-3-642-19817-5. [22] NATO Standardisation Agency. STANAG 4671 - Unmanned Aerial Vehicle Systems Airworthiness Requirements (USAR), 2009. [23] C. M. Pilato, B. Collins-Sussman, and B. W. Fitzpatrick. Version Control with Subversion. O’Reilly Media, 2 edition, Sept. 2008. ISBN 0596510330. [24] L. Rierson and B. Lingberg. Reverse engineering of software life cycle data in certification projects. In Proceedings of the 22nd Digital Avionics Systems Conference (DASC), volume 1, Oct. 2003. doi: 10.1109/DASC.2003.1245824.

n

[25] RTCA. DO-178B/ED-12B Software Considerations in Airborne Systems and Equipment Certification, 1992. [26] RTCA. DO-178C/ED-12C Software Considerations in Airborne Systems and Equipment Certification, 2011.

sio

[27] RTCA. DO-330/ED-215 Software Tool Qualification Considerations, 2011.

[28] RTCA. DO-331/ED-218 Model-Based Development and Verification Supplement to DO-178C and DO-278A, 2011. [29] RTCA. DO-332/ED-217 Object-Oriented Technology and Related Techniques Supplement to DO-178C and DO-278A, 2011. [30] RTCA. DO-333/ED-216 Formal Methods Supplement to DO-178C and DO-278A, 2011.

Re vi

[31] B. Rumpe. Agile Modellierung mit UML: Codegenerierung, Testfälle, Refactoring (Xpert.press). Springer, November 2004. ISBN 3540209050. [32] J. Rushby. New challenges in certification for aircraft software. In Proceedings of the International Conference on Embedded Software (EMSOFT), pages 211 –218, Oct. 2011. [33] R. Spies. Objektorientierte Sensorfusion und Integration erweiterter Sensorik bei unbemannten Luftfahrzeugen. Bachelorthesis, Universität Siegen, Naturwissenschaftlich-Technische Fakultät, Department Elektrotechnik und Informatik, 2011. [34] The Joint JAA/EUROCONTROL Initiative on UAVs. UAV Task-Force Final Report: A Concept for European Regulations for Civil Unmanned Aerial Vehicles (UAVs), May 2004.

ev iew

[35] M. Utting and B. Legeard. Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2006. ISBN 0123725011.

Pr

[36] S. VanderLeest and A. Buter. Escape the waterfall: Agile for aerospace. In Proceedings of the IEEE/AIAA 28th Digital Avionics Systems Conference (DASC), Oct. 2009. doi: 10.1109/DASC.2009.5347438.


Pr

ev iew

Re vi

sio

n

Appendix

Figure 1. The bug tracking process.


n American Institute of Aeronautics and Astronautics

Figure 2. The change management workflow.

sio Re vi ev iew Pr 14 of 15

n American Institute of Aeronautics and Astronautics

Figure 3. The integration workflow.

sio Re vi ev iew Pr 15 of 15