Toward Model Based Testing: Combining AADLS with High Level Petri Nets 1
2
Hassan Reza1, Feifei Gu1, Bahram Shafai2 School of Aerospace Sciences, University of North Dakota, Grand Forks, USA
[email protected]
Electrical and Computer Engineering, Northeastern University, Boston MA, USA
Abstract—Traditional software testing, for most parts, suffers from the evolvable and/or interpretable requirements. To overcome these limitations and hence improve reliability and confidence of software products require incorporation of formal methods during the software development lifecycle. In this work, we discuss a hybrid testing method that utilizes formal specification techniques to generate test cases. To this end, the proposed method transforms architectural specification language commonly known as AADL into HPrTNs to generate test cases that can be used to validate software intensive products. Key words—Software Architecture, Model-Based testing, Architectural Analysis and Description Languages (AADLs), Hierarchical Predicate Transition Nets (HPrTNs).
1
INTRODUCTION
Software testing is final blockade to the release of software and is perceived as a dominated cost factor in the development of software intensive systems. Various research studies have suggested that more than fifty percent of the cost associated with the construction of software system is related to the software testing activity [1], [2], [11], [20], [23]. This cost can be higher in the constructions of safety critical systems because they demand higher degree of dependability. In the classical waterfall model system artifacts produced during software specification, software architecture, implementations are used for system, integration, and unit testing respectively. The undetected faults that usually resulted from mis-understanding requirements can easily move into later development stages. Studies have indicated that the costs of fixing these faults during software testing can be very expensive. To overcome the limitations of software testing, many methods and techniques have been suggested. Examples include architectural-based testing, model-based testing, and hybrid testing that combine the benefits of formal verification testing. Software architecture (SA) of a system is an initial software artifact that addresses the customer needs. In addition it documents system structural and behavioral organizations using components (computations), connectors
(communications), configurations (overall compositions), and coordination (glue and synchronizations) [10], [14]. Formal specifications of software architecture received significant attention by software engineering community in past 15 years because of the size and complexity of today’s software systems. Formal architectural specifications represent overall structure and behaviors of a system at the high level of abstraction using formal notations. They have been heavily used as a formal base to analyze and verify the behaviors of the developed system to prevent design errors early on during the software frond-end engineering. Formal architectural specifications of systems also have been used during the software back-end development (i.e., testing) [4], [5], [8], [9], [13], [21], and [29]. For instance, using software architecture and its corresponding coverage criteria, one can generate required test cases to validate: 1) the implementation of communication protocols, 2) interfaces among subsystem represented by components, 3) system configurations satisfying certain constraints, and 4) coordination among system elements [21]. On the other hand, model-based testing (MBT) is a modern testing technique for automatically generating a suite of test cases from requirements and behavioral models of systems. The main incentive for using MBT is automation and validation of test cases generated from formal models. As such it offers significant promise in reducing the cost of test cases generations and hence reducing the testing activities [3], [12], [15], [16][31]. This paper proposes: 1) the selection of software architecture that should address critical requirements of real time distributed control systems used in aerospace industries. Examples of these requirements include safety, security, availability, and performance, and 2) using SA as a formal model that can be fed into model based testing to generate test suites. The rest of this paper is organized in the following sections. Section 2 provides related work. Section 3 discusses our motivation for this work. The remaining sections provide an overview about Model-based testing, Software Architecture (SA), Architecture description language (ADL), and our current and future works.
2
RELATED WORK
Foundational works related to using architectural specifications for software testing include research studies conducted by [12], [15], and [16] in which formal architectural specifications of systems defined by architectural description languages (ADLs) [22],[29] have been used as bases for integration, code-based testing, and system testing. Closely related to our work is the research conducted in [5]; the authors discussed the architecture-based testing technique can be used to identify architectural level faults. In [16], [29], the authors investigate how the SA notions, methods, and tools can be exploited within a software development process. In particular, their approach focuses on using SA descriptions to improve software testing. Although architectural specifications have shown some promises for various levels of software testing of large and complex systems but generations of effective testing suites, for most parts, must be manually performed. Model-based testing (MBT) is perceived as a black-box testing method in which test cases and oracle are automatically generate from a formal and functional model of a system under test (SUT) [3] [31]. However, the main shortcoming of MBT is the construction and suitable selection and representation of the model using formal notations.
3
MOTIVATION
The purpose of software testing is to increase the reliability of the software under the development. Toward this goal, a tester attempts to find errors by devising successful test cases. To this end, testers design test cases that uncover different classes of errors and do so effectively and efficiently to reduce the cost [20]. The test activity consists of designing test cases, execute the software with these test cases and observe the results produced by these executions. A tester has to define the inputs data, run the system using these inputs, and finally analyze the results [20]. In traditional software development lifecycle model testing performs in the later state of the lifecycle. Some of the problems with the testing are: constraints on time and resources, criticality and complexity of software systems, and dealing with unstable requirements. Typically, testers automate the testing by writing static scripts. Such a scripts focus only on the current functionality of the software. These handcrafted test scenarios are very static, hard to change, and very time consuming. Since the requirements of the software are always unstable, the behavior of the system is constantly changes. The change in software functionality and behavior dictates writing another set of static scripts for the current behavior of the software [15]. Therefore, it is difficult to conduct software testing effectively and efficiently during traditional software development lifecycle. Architectural specifications describe the essential properties
of the system using architectural constructs (i.e., components, connectors and glue); they can also assist developers to understand the functionality and behaviors of software systems by analyzing the models of the developed systems. On the other hand, using model based testing that employs architectural specifications, engineers should be able to automatically generate the required suite of test cases based on the coverage criteria corresponding to the formal model of software architecture [21], [29] to effectively and efficiently test the SUT. Therefore, it makes sense to combine the benefits of MBT [3] and SA [10] in a very complementary manner. In this work, we propose how the hybrid testing method, known as SA-MBT, can be used to generate test suites from a SA model of a system. SA specifications in architectural description language (AADL) [32] can provide a solid foundation for developing a testing plan. We use formal representation of SA as a model to generate the test cases used at system level. To this end, we plan to use AADL to specify the system at architectural level. AADL representations then will be mapped to Petri Nets [27], and [30] modeling notation to allow model-based testing.
4 BACK GROUND: MBT, SA, AADL, AND HPRTNS Model-based design (MBD) is considered to be an effective engineering method that is used for developing highly complex and critical software systems. For example, using formal models, designers are able to analyze the behaviors of a system to detect errors and to revise the design in very cost effective manner. Examples of systems developed using MBD include avionics, control, medical, military, and manufacturing systems. The model-based testing (MBT) begins with formulating the system requirements. Using the requirements, a model for functional behaviors is constructed. The model is then used to automatically generate and execute test cases. The system is executed against these generated tests, and the actual outputs are validated against the expected outputs. The test results will indicate whether to terminate the testing or revise and debug the model. The study of software architecture has motivated from the original works by Perry and Wolf [14], and Shaw and Garlan [10]. They studied the relationship between the complexity of software intensive systems and their corresponding high level formal specifications used to mitigate against their complexity. The key promise of software architectures is that using proper software architecture and notations, the software architect should be able to manage the complexity, costs and quality of software under construction In the simplest form, software architecture of systems refers to the specification of subsystems from which more complex systems are built. More specifically, the specifications include: 1) the specification of communication protocols by which
these subsystems interact, 2) the descriptions of patterns and constraints required for their composition, 3) configuration and coordination of these subsystems [10]. As a formal model of SA, architectural specifications can provide a concrete formal foundation for developing a testing plan at system level [4]. This formal model, in turn, provides a context in which behavioral and structural errors and inconsistencies can be detected in a very cost effective manner. Architecture Description Languages (ADLs) are formal modeling notations that provide precise syntax, precise semantics together with the reasoning capability for characterizing software architecture using architectural constructs such as components (processing elements), connectors (protocol of communication and interactions), and configurations (overall organization) [10]. In additions, ADLs provide supporting tool-sets for editing/drawing, analyzing (type checking), or simulating architectural specifications [6], [7], [17],[18],[19], [24]. The research conducted by [22] provides extensive survey of existing ADLs. More specifically, their work classifies and compares properties (optional and core) in components, connections, and configurations in ADLs. Architectural Analysis and Description Languages (AADLs) are formal notations used to design and analysis the system architecture of real-time embedded systems used in avionic systems and manufacturing systems [32]. For example, using AADL, engineers are able to specify and verify the system wide critical features such as performance, security, safety, and reliability [33]. The AADL modeling language supports components, interactions and configurations. An AADL component can be used to represent both hardware and software elements of a system. Software components represented by AADLs include: system (used for modeling hierarchal structure), data, subprogram, thread, thread group, process. Hardware components include: memory, bus, and processor. Moreover, the language allows modeling of the system at different levels of abstraction using hierarchal facilities to perform related reasoning such as performance, and dependability (security, safety, availability, reliability). AADL was initially developed by Honeywell Corporation under supervision of SAE (Society of Automotive Engineers). The underlining formal theory of AADL is based on MetaH [28], which is a domain specific ADL used for: a) specification and verification Real Times systems used in avionic and aerospace industries, and b) to manage development cost and schedule over-runs. AADL main advantages include common component definitions across documents; architecture models which allow early reasoning about system related properties to avoid the possibility of rework which may result in the cost increased and delay in the product delivery. AADL has been designed to specify and analyze the nonfunctional aspects of a system such as security, reliability, performance, and safety. As such, using AADLs, it is difficult to specify and analyze the behaviors (semantics) of a system. For example, using AADL, it is difficult to reason about
properties such as deadlock, or livelock, which may exist between different elements (subsystems) of a system. On the other hand, Petri nets [25][30] are well-established modeling notations with the formal underlying theory that make them suitable to model the behaviors of complex software systems. More specifically, using Petri nets, engineers are able to specify parallelism, non-determinism, synchronicity, and/or distribute-ness relevant to complex systems. Classical Petri nets are subject to the state space explosion problem because they lack abstractions and data structure mechanisms needed to manage the complexity of today’s software systems. On the other hand, high level Petri nets are a class of Petri nets that attempt to tackle the above issues using separation of concerns and divide and conquer strategies. Examples of this class of nets include Color Petri Nets (CPN) [26], and Hierarchical Predicate Transitions Nets (HPrTNs) [27]. An HPrTN is a tuple HPrTN = (P, T, F, , SPEC, INSC) satisfying the following requirements: P is a finite set of Predicates, T is a finite set of Transitions, F is a finite set of flow relations such that F (P T) (T P), is a hierarchical function. It is defined from P T (P T). SPEC is an algebraic specification; it is a tuple (S, OP, Eq) where S and OP collectively is a called signature: S is a set of Sorts and OP = (Ops1,…,Sn, s) is a set of sorted operations for s1,…,sn, s S. The S-equations in Eq define the meanings and properties of operations in OP. Furthermore, SPEC is used to define the tokens, flow annotations, and transitions post and preconditions (constraints/contracts) of an HPrTNs. The net inscription is a tuple INSC = ( , L, R, M0). INSC relates each graphical symbol of the net structure (P, T, F, ) with an element in SPEC. INSC must hold the following requirements: 1) is a sort mapping that associates each p P with subsets of sorts in S., 2) L is a label mapping that associates each f F with a label, 3) R is a constraint mapping that associates each t T with a first order logic formula, 4) M 0 is called the initial marking; it a mapping from the set of predicates P to the set of tokens. HPrTNs are deemed as effective modeling notations for describing a system at different level of abstractions. In what follows we summarize important features of HPrTNs model: To document the semantics (behaviors) of a system via SPEC; To document conditions and constraints via INSC; To document scenarios the sequences and/or distribution of input data, and firing sequences of enabled transitions. For example using: 1) INSC ( , L, R, M0), 2)firing of a sequence of enabled transitions corresponding to various scenarios and conditions, 3) SPEC (S, OP, Eq), and 4) the structure of the net, we should be able to test the model to better understand the semantics of a system, its expectations from external environments. This understanding, in turn, allows detecting faults in the system under the test [21]. Therefore, it makes sense to use HPrTNs as a suitable formal
base augment AADL that can be used by MBT to generate a set of test cases to test the behaviors of a system.
5
METHOD
In this paper, we propose to utilize model-based testing technique to generate test cases from behavioral model of control systems in HPrTNs. Initially SA of a control system is described by AADLs. The AADLs models are transformed to HPrTNs using table 1. We then generate test cases from AADL-to-HPrTNs models. Test cases generated from HPrTNs are used as expected behaviors of a system to validate against the actual behaviors of a system at run time. More specifically, using what is known as architecturally significant paths (ASP) [29], we should be able to identify small but architecturally significant paths. These paths should correspond to smell set (roughly 1520%) of connections paths between subsystems, and are responsible for 75-80% of total interactions among subsystems. These small set of paths are important from both functionality and non-functionality point of view, because they should represent the common but important communication flows among the elements of a system. Therefore, they have to be tested for system and integration testing proposes. Once the SA model is defined in AADL and mapped to HPrTNs, the AADL-to-HPrTNs specifications are then used as an input to model-based testing to generate the test cases. The process used to achieve the above objectives includes: 1. 2. 3.
4.
(input/output data, buffer, conditions, and/or signals) Connections Processors (thread scheduling and execution mechanisms) Memory Bus (physical connections) Device (interface to external environment)
Map the SA specification of a system defined in the AADLs to HPrTNs using mapping specified in table 1. Refine abstract model of a system represented by HPrTNs into concrete HPrTNs; Identify architecturally significant paths (ASP), which are interleaved sequences of components-connectors and connectors-components connection paths; Derive a set of test cases that adequately covers ASP and coverage criteria for SA. AADLs System (hierarchical organization of components) Processes (protected address space) Thread group (collections of processes) Data (sharable data resources) subprogram Port groups (collections of input/output data, buffer, conditions and/or signals) Ports
Mapping
HPrTNs Elements HPrTNs
Super Transitions
Super transitions
Predicates Transitions Super Predicates
Predicates
Flows Super Transitions
Super Predicates Flows Predicates
Table 1: Mapping between AADLs and HPrTNs We use a set of coverage criteria defined for software architecture [29] to sufficiently test a system. The coverage criteria include: All components: This criterion requires that all components of a system be tested. All connectors: This criterion requires that all connectors and communication patterns between components are tested. All connections: This criterion requires that all compositions and interfaces between components and connectors be tested. All constraints: This criterion requires that all constraints on the connectors and components are tested. All configurations: This criterion requires that all configurations of the components are tested.
6
CONCLUSION
In this work, we discussed a testing method that combines the benefits of model based testing with that of software architecture. To this end, we propose to employ SA as an input to model-based testing to generate the test cases. The advantages of using this approach are: 1) software testing can start in parallel with the product development and discovered errors early on during the software lifecycle, 2) test cases generated using SA can be used to validate the implemented system conforms the expected behaviors. The liability of this approach is that modeling of complex systems is a challenging task because it requires understanding the functionalities of the software under test as well as the external environment where the software system is going to be deployed. Future work will show the feasibility of our work by conducting a series of case studies using control and critical systems (e.g., avionics) to generate test suites that are more cost-effective.
7
ACKNOWLEDGEMENT
This work was partially funded under Department of Defense Federal Initiative, Joint Unmanned Aircraft Systems Center of Excellence at Creech Air Force Base, Nevada, DoD Contract Number FA4861-07-R-C003.
8
REFERENCES
[1]
A. Bertolino and P. Inverardi. Architecture-based software testing. In Proc. ISAW96, October 1996. D. J. Richardson and A. L. Wolf. Software Testing at the Architectural Level. In Proceedings of the Second International Software Architecture Workshop (ISAW-2), San Francisco, California, pages 68-71. October 1996. M. Utting and B. Legeard. Practical Model-Based Testing: A Tools Approach, 2007. H. Muccini, A. Bertolino, P. Inverardi, "Using Software Architecture for Code Testing," IEEE Transactions on Software Engineering, vol. 30, no. 3, pp. 160-171, doi:10.1109/TSE.2004.1271170, 2004. Z. Jin, J. Offutt. Deriving Tests from Software Architectures. Proceedings of the 12th International Symposium on Software Reliability Engineering (ISSRE'01), pg.308, 2001. N. Medvidovic. ADLs and Dynamic Architecture Changes. Proceedings of the 2nd International Software Architecture Workshop (ISAW-2), pages 24-27, San Francisco, CA, October 1996. N. Medvidovic, D. Rosenblum. Domains of concern in software architectures and architecture description languages. Proceedings of the Conference on Domain-Specific Languages (DSL), 1997. Z. Jin. A Software Architecture-based Testing Technique. Ph.D thesis, George Mason University, Fairfax, VA, Technical report, 2000. http://www.ise.gmu.edu/techrep. H. Muccini. Software Architecture for Testing, Coordination and Views Model Checking. PhD thesis, www.henrymuccini.com/Research/PhDThesis/PhDThesis(Muccini).pdf. M. Shaw, and D. Garlan. Software Architecture: Perspectives on an Emerging Discipline. Prentice Hall, 1996. S. Loveland, G. Miller, R. Prewitt Jr., M. Shannon: Software Testing Techniques: Finding the Defects that Matter. Charles River Media. October 2004. H. Robinson. Model Based Testing. http://us.share.geocities.com/harry_robinson_testing/stareast_2005_mbt _tutorial.ppt. H. Muccini, M. Dias, D. Richardson. Towards software architecturebased regression testing. Proceedings of the 2005 workshop on Architecting dependable systems, pg: 1-7, 2005 D. E. Perry, and A. L. Wolf. “Foundations for the Study of Software Architectures.” ACM SIGSOFT Software Engineering Notes, vol. 17, no. 4, pg: 40-52, October 1992. S. Dalal, A. Jain, C. Lott, G. Patton, N. Karunanith, J. M. Leaton, and B. M. Horowitz. Model-based testing in practice. In Proceedings of the 21st International Conference on Software Engineering, pg: 285–294. ACM Press, 1999. Model Based Testing: https://www.goldpractices.com/practices/mbt/ D. Garlan, R.T. Monroe, and D. Wile. ACME: An Architecture Description Interchange Language. Proceedings of CASCON 97, pg: 169-183, 1997. The Acme Project. www.cs.cmu.edu/~acme/ H. Gall. Architectural Description. http://www.seal.ifi.uzh.ch/fileadmin/User_Filemount/Vorlesungs_Folien /AdvSE/swa-03-ACME.pdf. P. Jorgensen Software Testing: A Craftsman’s Approach. CRC Press, 2008. H. Reza, and E. Grant: A method to test concurrent systems using architectural specification, Springer 2007. N. Medvidovic, and R. Taylor: A Classification and Comparison Framework for Software Architecture Description Languages. IEEE Trans. Software Eng. 26(1): 70-93, 2000. M.J. Harrold. Testing: A Roadmap. In Future of Software Engineering, 22nd International Conference on Software Engineering (ICSE00), June 2000.
[2]
[3] [4]
[5]
[6]
[7]
[8]
[9]
[10] [11]
[12]
[13]
[14]
[15]
[16] [17]
[18] [19]
[20] [21] [22]
[23]
[24] D. Garlan, and Z. Wang. Acme-based Software Architecture Interchange. Springer Berlin / Heidelberg, Vol.1594/1999, 1999. [25] L. Peterson. Petri Net Theory and Modeling of the systems. Prentice Hall, 1981. [26] K. Jensen. Colored Petri Nets: Basic Concepts, Analysis Methods and Practical Use Vo.1, second edition, Springer, 1997. [27] X. He. A formal Definition of Hierarchical Predicate Transition Nets. LNCS. Vol.1091.1996. [28] S. Vestal. The MetaH AADL Toolset.
http://www51.honeywell.com/aero/technology/common/docume nts/metah-aadl.pdf. [29] H. Reza, S. Lande. Model Based Testing Using Software Architecture. ITNG 2010: 188-193. [30] T. Murata. Petri Nets: Properties, Analysis and Applications. Proceedings of the IEEE, vol. 77, no. 4, April 1989. [31] I. El-Far, J. Whittaker. Model-Based Software Testing. Encyclopedia on Software Engineering, Wiley, 2001. [32] P. Feiler, D. Gluch, J. Hudak. The Architecture Analysis and Design Language (AADL). An Introduction. Carnegie Mellon University, Software Engineering Institute. Technical Note: CMU/SEI-2006-TN011, February 2006. [33] P. Feiler, D., and J. Hudak.b Developing AADL Models for Control Systems: A Practitioner’s Guide. Carnegie Mellon University, Software Engineering Institute. Technical Report: CMU/SEI-2007-TR-014, July 2007.
