Implementation Support of Security Design Patterns Using Test ... - MDPI

information Article

Implementation Support of Security Design Patterns Using Test Templates † Masatoshi Yoshizawa 1, *, Hironori Washizaki 1 , Yoshiaki Fukazawa 1 , Takao Okubo 2 , Haruhiko Kaiya 3 and Nobukazu Yoshioka 4 1

2 3 4

* †

Global Software Engineering Laboratory, Department of Computer Science and Engineering, Waseda University, Shinjuku-ku, Tokyo 169-8555, Japan; [email protected] (H.W.); [email protected] (Y.F.) Information Security Division, Institute of Information Security, 2-14-1 Tsuruyacho, Kanagawa-ku, Yokohama, Kanagawa 221-0835, Japan; [email protected] Department of Information Sciences, Kanagawa University, 2946 Tsuchiya, Hiratsuka-shi, Kanagawa 259-1293, Japan; [email protected] GRACE Center, National Institute of informatics, 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430, Japan; [email protected] Correspondence: [email protected]; Tel.: +81-03-5286-3272 This paper is an extended version of Verification of Implementing Security Design Patterns Using a Test Template, Proceedings of the 9th International Conference on Availability, Reliability and Security (ARES2014), Fribourg, Switzerland, 8–12 September 2014.

Academic Editor: Eduardo B. Fernandez Received: 15 February 2016; Accepted: 2 June 2016; Published: 15 June 2016

Abstract: Security patterns are intended to support software developers as the patterns encapsulate security expert knowledge. However, these patterns may be inappropriately applied because most developers are not security experts, leading to threats and vulnerabilities. Here we propose a support method for security design patterns in the implementation phase of software development. Our method creates a test template from a security design pattern, consisting of an “aspect test template” to observe the internal processing and a “test case template”. Providing design information creates a test from the test template with a tool. Because our test template is reusable, it can easily perform a test to validate a security design pattern. In an experiment involving four students majoring in information sciences, we confirm that our method can realize an effective test, verify pattern applications, and support pattern implementation. Keywords: security patterns; model-based testing; test-driven development; aspect-oriented programming

1. Introduction Due to the increasing number of business services on open networks and distributed platforms, preventing and resolving security issues is vital [1]. Security concerns must be considered in every phase of software development from requirements engineering to design, implementation, testing, and deployment [2]. However, addressing all security concerns is difficult due to the sheer volume of them and the fact that not all software engineers are security specialists. Patterns are reusable packages that encapsulate expert knowledge. Specifically, a pattern represents a frequently recurring structure, behavior, activity, process, or “thing” during the software development process. Many security design patterns have been proposed. For example, Schumacher et al. presented 25 design-level security patterns [3]. Because security design patterns are currently abstract descriptions, they are difficult to implement. Additionally, it is hard to validate

Information 2016, 7, 34; doi:10.3390/info7020034

www.mdpi.com/journal/information

Information 2016, 7, 34

2 of 19

security design patterns in the implementation phase because an adequate test case is required. Hence, a security design pattern can be inappropriately applied, leading to serious vulnerability issues. Herein we propose a method to support the implementation of security design patterns using a test template. Our method creates a test template from a security design pattern, which consists of an “aspect test template” to observe internal processing and a “test case template”. By providing design information in the test template, a test is created to evaluate the system in the early implementation stage using a tool and fixing the code. As the test can be executed repeatedly, it can validate whether a security design pattern is appropriately applied in the implementation phase. In this paper, we address the following research questions: ‚ ‚ ‚ ‚

RQ 1: Is our method more efficient at finding defects when implementing a security design pattern than an existing method? RQ 2: Does our method create a more effective test to find implementation defects in a security design pattern than an existing method? RQ3: Compared to an existing method, does our method more effectively create a test to find implementation defects in a security design pattern and allow for a regression test? RQ 4: Is our method more effective at correcting implementation defects in a security design pattern compared to an existing method? Our contributions are as follows:

‚ ‚ ‚

We create a reusable test template derived from security design patterns. Implementation defects in a security design pattern are efficiently determined. Our method supports the appropriate implementation of security design patterns.

The remainder of this paper is organized as follows. Section 2 provides the background and problem with security design patterns. Section 3 describes related works. Section 4 details our proposed method. Section 5 shows a case study using our method. Section 6 discusses the evaluation results. Finally, Section 7 concludes the paper. 2. Background and Problem 2.1. Security Design Patterns A security design pattern is a reusable documented security problem and corresponding solution that can be used to make decisions on the conceptual architecture and detailed design of a software system. In the design phase of software development, security functions should be designed to satisfy the security properties of the assets identified in the requirement phase. Security design patterns include “Name”, “Context”, “Problem”, “Solution”, “Structure”, “Consequence”, and “See Also”. The “Structure” usually contains some design models described by the Unified Modeling Language (UML). UML is a widely accepted and standardized language to model software applications [4]. These models may contain constraints on the design in the form of OCL descriptions. OCL [5,6] stands for Object Constraint Language, which is a semiformal language that is used to express constraints and other expressions in UML and other modeling languages. For example, in [7], we successfully defined UML models with OCL descriptions for various security design patterns. These patterns can be reused in multiple software systems. Figure 1 shows two examples of security design pattern structures. The Password Design and Use pattern [3,8] describes the best security practice to design, create, manage, and use password components. In addition to configuring or managing passwords, engineers and administrators use password constraints to build or select password systems. Figure 2 shows the corresponding OCL description for Figure 1 specifying the security requirement that should be satisfied in the structure. The OCL description means that the ID and Password inputted from the login screen must agree with the ID and Password of the User Data in order for a user to be deemed a regular user and granted access to an asset. If this condition is not

Information 2016, 7, 34

3 of 19

asset. If this condition is not met, the user is considered a non-regular user and denied access to an Information 2016, 7,Figures 34 3 of 19 asset. Moreover, 3 and 4 show the expected behavior in the form of a UML communication Information 2016, 7, 34 3 of 19 diagram. Information 2016, 7, 34 3 of 19 asset. If this condition is not met, the user is considered a non-regular user and denied access to an The IfRole-based Access (RBAC) pattern a[3,8], which user is a and representative asset. this condition is not Control met, the user is considered non-regular denied accesspattern to an for asset. Moreover, Figures 3 and 4 show the expected behavior in the form of a UML communication asset. Moreover, Figures 3 and 4 show the expected behavior in the form of a UML communication access control, describes how to assign precise access rights to roles in an environment where access met, the user is considered a non-regular user and denied access to an asset. Moreover, Figures 3 and 4 diagram. to diagram. computing resources must be form controlled tocommunication preserve confidentiality and the availability showThe the Role-based expected behavior in the of a UML diagram. Access Control (RBAC) pattern [3,8], which is a representative pattern for The Role-based Access Control (RBAC) pattern [3,8], which is a representative pattern for requirements. Role-based Access (RBAC) pattern which is ain representative pattern foraccess access accessThe control, describes howControl to assign precise access[3,8], rights to roles an environment where access control, describes how to assign precise access rights to roles in an environment where access control, describes how to must assignbe precise access to rights to rolesconfidentiality in an environment where access to to computing resources controlled preserve and the availability to computing resources must be controlled to preserve confidentiality and the availability computing resources must be controlled to preserve confidentiality and the availability requirements. requirements. requirements.

(a)

(b)

(a) (b) (a) (b) Figure 1. (a) Structure of a security design pattern (Password Design and Use pattern) and (b) Figure 1. (a) (a) Structure Structureofof a security design pattern (Password Design and Use pattern) and (b) Figure1.1. design pattern (Password Design and Use and (b) and structure Figure Structure ofa asecurity security design pattern (Password Design and Use pattern) (b) structure of a(a) security design pattern (Role-based Access Control pattern ).pattern) structure of a security design pattern (Role-based Access Control pattern ). of a security design pattern (Role-based Access Control pattern). structure of a security design pattern (Role-based Access Control pattern).

Figure 2. OCL description (Password Design and Use pattern).

Figure OCL description and Use pattern). Figure 2.2.2. OCL description (Password Design and Use pattern). Figure OCL description(Password (PasswordDesign Design and Use pattern). : mis_actor : mis_actor

: mis_actor

: Login_UI : Login_UI

: password_design_and_use : password_design_and_use

: Login_UI

: password_design_and_use

1: [regular_user == false] login(incorrect_id, incorrect_password) 1: [regular_user == false] login(incorrect_id, incorrect_password) 1.1: check_identification(incorrect_id, incorrect_password) 1: [regular_user == false] login(incorrect_id, incorrect_password) 1.1: check_identification(incorrect_id, incorrect_password) login_failed 1.1: check_identification(incorrect_id, incorrect_password) login_failed

login_failed

(a) (a)

(a) Figure 3. Cont.

Information 2016, 7, 34

4 of 19

Information 2016, 7, 34

4 of 19

Information 2016, 7, 34

4 of 19 : Login_UI

: actor

: password_design_and_use

: Login_UI : actor 1: login(correct_id, correct_password)

: password_design_and_use

1: login(correct_id, correct_password)

1.1: check_identification(correct_id, correct_password) 1.1: check_identification(correct_id, correct_password)

login_successful login_successful

(b) (b) Figure 3. Login behavior (Password Design and Use pattern). (a) Failure case; (b) Success case.

Figure 3. Login behavior (PasswordDesign Design and and Use case; (b) (b) Success case.case. Figure 3. Login behavior (Password Use pattern). pattern).(a) (a)Failure Failure case; Success regular_user = false

regular_user = false

: UI

: mis_actor

: Subject_Controller

: I&A

: User_Data

: Login_UI

1: request() 1.1: request() : Subject_Controller 1.1.1: use()

: UI

: mis_actor

: I&A

1: request()

: User_Data

: Login_UI

1.1.1.1: identify()

1.1: request() 1.1.1: use()

registered_user = false 1.1.1.2:1.1.1.1: [registered_user == false] error() identify()

registered_user = false 1.1.1.2: [registered_user == false] error()

(a) regular_user = true

: UI

: actor

(a)

: Subject_Controller

: I&A

: User_Data

: Asset

1: request() 1.1: request()

regular_user = true

1.1.1: use() 1.1.1.1: identify() registered_user = true registered_user = true

: UI

: actor

: Subject_Controller

: I&A

: User_Data

: Asset

1.1.2: [registered_user == true] access_asset_data()

1: request() 1.1: request()

1.1.1: use() 1.1.1.1: identify() registered_user = true registered_user = true (b) 1.1.2:and [registered_user ==Failure true] access_asset_data() Figure 4. Behavior to access an asset (Password Design Use pattern). (a) case; (b) Success case.

Figure 4. Behavior to access an asset (Password Design and Use pattern). (b) case.Example of Implementation Problem 2.2.Success Motivating

(a) Failure case;

As an example of a pattern application, Figure 5 shows a portion (“login”) of a UML class diagram.

2.2. Motivating Example of Implementation Problem

As an example of a pattern application, Figure 5 shows a portion (“login”) of a UML class diagram.

(b) Figure 4. Behavior to access an asset (Password Design and Use pattern). (a) Failure case; (b) Success case.

2.2. Motivating Example of Implementation Problem As an example of a pattern application, Figure 5 shows a portion (“login”) of a UML class diagram.

Information 2016, 7, 34 Information 2016, 7, 34 Information 2016, 7, 34

5 of 19 5 of 19 5 of 19

Is Is pattern pattern applied applied appropriately? appropriately?

Implementation Implementation

Figure 5. Implementation Implementation of the the “login” portion portion of aa class class diagram for for payment processing. processing. Figure Figure 5. 5. Implementation of of the “login” “login” portion of of a class diagram diagram for payment payment processing.

Although Although the class class diagram in Figure applies Design Although the the class diagram diagram in in Figure 55 appropriately appropriately applies applies the the Password Password Design Design and and Use Use pattern and the Prevent SQL Injection pattern [3], it is incomprehensible in the implementation Injection pattern [3], [3], it is it incomprehensible in thein implementation phase. pattern and andthe thePrevent PreventSQL SQL Injection pattern is incomprehensible the implementation phase. to the is the between the How implement the selected patternpattern is unclear becausebecause the relation between the security design phase.toHow How to implement implement the selected selected pattern is unclear unclear because the relation relation between the security security design pattern and implementation is not defined. Figure 6 shows part of the implementation pattern and implementation is not defined. Figure 6 shows part of the implementation of the diagram design pattern and implementation is not defined. Figure 6 shows part of the implementation of of the the diagram in Figure 5 with defects, which are a miswritten SQL description and an omitted escape in Figurein5 with arewhich a miswritten SQL description and an omitted diagram Figuredefects, 5 withwhich defects, are a miswritten SQL description and an escape omittedprocess. escape process. the system may the applied pattern be Consequently, the system be vulnerable. Thus, theThus, applied be must verified via a testvia in process. Consequently, Consequently, themay system may be be vulnerable. vulnerable. Thus, the pattern appliedmust pattern must be verified verified via athe test in the implementation phase. implementation phase. phase. a test in the implementation

Figure Figure 6. 6. Part Part of of the the implementation implementation of of the the class class diagram diagram in in Figure Figure 5. 5. Figure 6. Part of the implementation of the class diagram in Figure 5.

A A conventional conventional test test only only detects detects vulnerabilities vulnerabilities due due to to known known coding coding bugs. bugs. It It cannot cannot determine determine A conventional test only detects vulnerabilities due to known coding bugs. It cannot determine if a security design pattern is appropriately applied, making validation extremely in if a security design pattern is appropriately applied, making validation extremely difficult difficult in the the if a security design pattern is appropriately applied, making validation extremely difficult in implementation phase. Similar to our work, Munetoh et al. proposed a method to verify the implementation phase. Similar to our work, Munetoh et al. proposed a method to verify the implementation phase. Similar to our work, Munetoh al. method proposed a method to verify the completeness implemented security features [9], is to control completeness of of the the implemented security features [9], but butettheir their method is limited limited to access access control completeness of the implemented security features [9], but their method is limited to access control for for Ruby-on-Rails web applications. for Ruby-on-Rails web applications. Ruby-on-Rails web applications. 3. 3. Related Related Work Work 3. Related Work 3.1. Patterns Verification 3.1. Security Security 3.1. Security Patterns Patterns Verification Verification Several articles focus pattern applications. Several articles articles focus focus on on verifying verifying security security pattern pattern applications. applications. Abramov Abramov et et al. al. have have suggested suggested Several on verifying security Abramov et al. have suggested using a stereotype for a database application to validate security patterns at the design level [10]. using a stereotype for a database application to validate security patterns at the design using a stereotype for a database application to validate security patterns at the design level level [10]. [10]. Although validate applied structurally, it Although their their method method can can validate an an applied pattern pattern it cannot cannot confirm confirm whether whether the the Although their method can validate an applied pattern structurally, structurally, it cannot confirm whether the pattern pattern behavior behavior in in the the implementation implementation code code resolves resolves vulnerabilities vulnerabilities to to threats. threats. In In contrast, contrast, our our method method tests tests the the pattern pattern behavior behavior at at the the implementation implementation level. level.

Information 2016, 7, 34

6 of 19

pattern behavior in the implementation code resolves vulnerabilities to threats. In contrast, our method tests the pattern behavior at the implementation level. Dongs et al. have proposed an approach to verify the compositions of security patterns using model checking [11]. They presented guidelines to specify the behavior of security patterns in the model specification language. They defined a synchronous message, asynchronous message, and alternative flows of a UML sequence diagram, which were transformed into CCS specifications. Although this approach formally defines the behavioral aspect of security patterns and provides transformation scripts to check the properties of the security patterns by model checking at the model level, it does not conduct an analysis at the implementation level. Therefore, even if the security patterns application can be verified at the model level, it does not guarantee that threats and vulnerabilities are resolved in the corresponding code. On the other hand, our method tests the pattern at the implementation level. Similarly, Hamid et al. proposed a method to validate the application of patterns at the model level by UML and OCL [12]. However, it is not applicable to implementation-level analysis. Modeling techniques such as UMLsec [13] and SecureUML [14] have been proposed to address security concerns. UMLsec is defined in the form of a UML profile using standard UML extension mechanisms. Stereotypes with tagged values are used to formulate the security requirements, while constraints are used to verify whether the security requirements hold during a specific type of attack. SecureUML focuses on modeling access control policies and how these policies can be integrated into a model-driven software development process. It is based on an extended model of RBAC and uses RBAC as a meta-model to specify and enforce security. RBAC lacks support to express access control conditions that refer to the state of a system, such as the state of a protected resource. In addressing this limitation, SecureUML introduces the concept of authorization constraints. Authorization constraints are preconditions for granting access to an operation. Although these techniques can support modeling security-related concerns in UML models, they cannot be directly used to check the correctness of security pattern applications at the implementation level. 3.2. Model-Based Security Testing Model-based Testing (MBT) is a technique to generate part or all of a test case from a model [15]. A model is an abstract representation expressing an operation, structure, behavior, or any other concern to be realized by the system or related to the system. Especially in the context of security testing, the process of deriving tests tends to be unstructured, irreproducible, undocumented, lacking detailed rationales for the test design, and dependent on the ingenuity of single testers or hackers, motivating the use of explicit models [16]. Tretmans and Brinksma proposed an automated MBT tool [17], while Felderer et al. proposed a method of Model-Based Security Testing (MBST) that relies on models to test whether a software system meets its security requirements [18]. Moreover, there are many other publications on MBST techniques since MBST is a relatively new research field with potential in industrial applications [19]. Felderer et al. proposed a taxonomy for MBST approaches and systematically classified 119 publications using MBST [16]. However, none of these publications focus on security design pattern applications. We have developed a formal test template based on MBT. Our test template, which is used to create a security design pattern, is abstract and reusable. Thus, it can be applied to various systems. According to the taxonomy proposed by [16], our method can be classified as an approach adopting “Functionality of Security Mechanisms” as a “Model of system security”, “Threat Models” as a “Security model of the environment”, “Explicit Test Case Specifications” as “Explicit test selection criteria”, “Prototype” as the “Maturity of evaluated system”, both “Effectiveness Measures” and “Efficiency Measures” as “Evidence measures”, and “Executable” as the “Evidence level”. 3.3. Test-driven Development (TDD) for Security Test-driven Development (TDD) is a software development technique that uses short development iterations based on prewritten test cases that define desired improvements or functions. Here our

Information 2016, 7, 34

7 of 19

testing process uses TDD, which requires development prior to writing the actual code [20]. A test case represents a requirement that the program must satisfy [21]. Although some approaches implement security requirements by adopting TDD [9,22], they do not specifically handle security design patterns and consequently require developers to manually prepare test cases when applying and testing security design patterns. This is highly burdensome. On the other hand, our method focuses on security design pattern applications. Our method uses Selenium [23,24] and JUnit [25,26], which are tools for testing applications. First a test is created where the security requirement of the corresponding pattern is satisfied. Then a test is quickly executed (first test) to detect vulnerabilities in the code. Afterwards the code is updated so that it passes the test. Finally the test is re-executed to confirm that the vulnerabilities are resolved. 3.4. Decision Table Testing A decision table is a tool to capture certain kinds of system requirements and to document internal system design [27]. The general form of a decision table consists of conditions and actions; “conditions” represent various input conditions [27] and “actions” are the actions that should be taken depending on the combination of the input conditions. Decision tables can serve as a guide to create test cases. Such a black-box test design technique based on decision tables is called “decision table testing”. Decision table testing is commonly used to research various software targets, including as web applications [28] and web services [29] at various phases such as integration testing [30] and system testing [31]. Our method adopts decision tables to describe the specifications of target security design patterns. These tables are then used to derive the test templates and corresponding test cases. 3.5. Aspect-Oriented Programming Aspect-oriented Programming aims to improve the modularity of software by providing constructs to modularize the so-called crosscutting concerns, which are concerns where the code representation cannot be modularized using traditional software development mechanisms [32]. For example, in the pointcut-advice model of aspect-oriented programming, which is embodied in AspectJ [33], a crosscutting behavior is defined by pointcuts and advice. A pointcut is a predicate that matches program execution points, which are called join points, while advice is the action to be taken at a join point. An aspect is a module that encompasses a number of pointcuts and advice [34]. Montrieux et al. presented a tool that generates Java and AspectJ code from a UML model with a verified UMLsec property [35]. Although the tool can be utilized for testing at the implementation level, it handles only RBAC as the target security design pattern, enforcing the security property to be checked. In our method, developers can register and handle various security design patterns to be tested. In our method, a test template, which is created via AspectJ, embodies the test, observes the internal processing, and supports vulnerability validation. Because the internal processing does not display the result on the screen such as escape processing in Figure 3, it is hard to validate that the processing is executed and the result is correct using browser-based testing. 4. Implementation Support of Security Design Patterns 4.1. Overview Our method verifies the applications of security design patterns and supports their implementation. Figure 7 outlines the process of our method. A test template is derived from a security design pattern by providing design information. Then the developer executes a test to validate the applied security design pattern in the implementation phase of software development. Figure 8 shows the entire picture of our method that clarifies necessary inputs and outputs by using our method’s implementation tool, called TESEM (TEst driven SEcure Modeling tool). The developer can

Information 2016, 7, 34

8 of 19

create a test case by inputting correct input data, incorrect input data, and the testing environment Selenium code into TESEM. Our tool allows a developer who is not a security specialist to create the testInformation case. 2016, 7, 34 8 of 19 Information 2016, 7, 34

8 of 19

0. Creating test template 0. Creating test template

1. Embodying test 1. Embodying test

2. Implementing design 2. Implementing design

3/5. Testing implementation 3/5. Testing implementation

4. Fixing implementation 4. Fixing implementation

[Test failed] [Test failed]

[Test passed] [Test passed]

Figure7. 7. Process Process of Figure of our our method. method. Figure 7. Process of our method.

Context Context

Problem

0. Creation 0. Creation

Problem

Solution

Researcher and Researcher and developer

Aspect test Aspect test template

Solution

Security pattern

developer

Test case

template

Security pattern

Decision

Test case template template

Decision table table

1. Generation

1. Generation 1. Generation bybyspecifying specifying parameters parameters

1. Generation by specifying by specifying parameters parameters

Developer Developer withwith support of of support TESEM TESEM

1. Creation

1. Creation

Correctinput input ••Correct Incorrectinput input ••Incorrect

Legend: Legend: Automated Automated SemiSemiautomated automated

Test Testaspect aspect

case Test case

3/5. 3/5.Instrumentation Instrumentation byAspectJ AspectJ by Implementation 2.2. Implementation

Manual Manual

Seleniumcode code Selenium 3/5. 3/5.Testing Testingby by Selenium Selenium and and JUnit JUnit

4. 4.Fixing Fixing Design with Implementation Design with Implementation security patterns under undertesting testing security patterns

Instrumented Instrumented implementation implementation

Figure 8. Entire picture of our method.

Figure8.8. Entire Entire picture of Figure of our our method. method.

Specifically, our method involves six steps.

Specifically, our method involves six steps. Specifically, our method involves six steps. Step 0: Create a test template. Step 0: Create a test template. A test template is prepared from a security design pattern. The test template consists of an Step 0:A Create a test template. test template is prepared security design pattern. The test template consists of an “aspect test template” and from a “testacase template”. A test template is prepared from a security “aspect test template” and a “test case template”.design pattern. The test template consists of Step 1: Embody tests. an “aspect test template” and design a “testinformation case template”. Step 1: Embody A test istests. embodied by the given in a test template. A test is embodied by the given design information in a test template. Step 1: Embody tests. Step 2: Implement a design. Step 2: Implement a design. Although the patternbyapplication verified, theinintended design for the security A test is embodied the given cannot designbe information a test template. Although the pattern application cannot be verified, the intended design for the security design pattern can be implemented. Step 2: Implement a design. design can bethe implemented. Step 3: Testpattern and validate applied patterns. Although the pattern application cannot be verified, the intended design for the security Based TDD, the a applied test is quickly Step 3: Test andon validate patterns.executed to validate the applied patterns in the design pattern can be implemented. implementation this step, concrete aspects aspectintest Based on TDD, aphase. test During is quickly executed to test validate thegenerated applied from patterns the templates are weaved the implementation undertest testing for thegenerated instrumentation purpose. implementation phase. into During this step, concrete aspects from aspect test templates are weaved into the implementation under testing for the instrumentation purpose.

Information 2016, 7, 34

9 of 19

Step 3: Test and validate the applied patterns. Based on TDD, a test is quickly executed to validate the applied patterns in the implementation phase. During this step, concrete test aspects generated from aspect test templates are weaved into the implementation under testing for the instrumentation purpose. Step 4: Fix. The implementation is fixed based on the defects found in Step 3. Step 5: Re-test and re-validate applied patterns. The fixed implementation is re-tested to re-validate the applied patterns. If the test is passed successfully, then the patterns are appropriately applied in the implementation phase. If the test is failed, Step 4 is repeated until the re-test passes. 4.2. Test Template and Test Case The test template consists of a test case template and an aspect test template. The former generates concrete test cases by specifying a few parameters. The latter generates concrete aspects for instrumentation to observe the internal processing of the code under testing. Testing security design pattern applications often requires such internal data access. Using our previous method [7], we prepared OCL descriptions for each existing security design pattern as test oracles. Then we prepared decision tables and the corresponding test template according to the OCL descriptions. In our method, templates are prepared and registered to our tool manually for each security design pattern. After the registration, concrete test cases are generated semi-automatically as the parameters must be inputted. The generated test cases check the implementation under testing to see whether its outputs or internal states meet the expected results stated in the test cases automatically. The flow to (1) realize a test template and (2) generate test cases is shown below. (1)

For each security design pattern to be tested, we (and the developers) prepare and register a pair of a test case template and an aspect test template using the three steps below. Once registered, the templates can be reused for further development projects. (a) (b) (c)

(2)

Developers create a decision table from the OCL description. Developers create an aspect test template for an internal processing observation from the decision table and the pattern structure. Developers create a test case template from the decision table and the pattern behavior.

By using TESEM, developers can easily generate concrete test cases as follows: (a)

(b) (c)

(d)

Developers bind the elements constituting the selected security design pattern and class/interface names in the given design model by specifying a class/interface name for each pattern element as a parameter. Developers input the Selenium code corresponding to the target security design pattern to be tested, such as a code behaving as a login function to test the login. Developers specify concrete correct values for the parameters in the test cases, such as concrete values of ID and password for testing login. Developers also specify concrete incorrect values. Finally, TESEM generates test cases together with concrete test aspects using the parameters and the information specified in the above steps.

4.3. Tool Implementation and Example In this section, we explain our method’s tool implementation as well as the test template using the Password Design and Use pattern as a concrete example. We implemented our method into

Information 2016, 7, 34

10 of 19

TESEM [7,36–39], which is a modeling and testing tool originally dedicated only to testing design models. Under the environment of JUnit together with Selenium, the generated test cases check Information 2016, 7, 34 10 of 19 whether the implementation’s outputs and internal states meet the expected results described in the Information 2016, 7, 34 10 of 19 test cases. Table 1. Decision Table (Password Design and Use pattern). Figures 1a, 2–4 depict the Password Design and Use pattern. Table 1 shows a decision table Table 1. Decision Table (Password Design and Use pattern). 1 2 3 4 following the OCL description. Inputted ID agrees with User Data Yes Yes 1 2 3No 4No Conditions Table 1. Decision Table (Password Design and Use pattern). Inputted withData User Data Yes No Yes No InputtedPassword ID agreesagrees with User Yes Yes No No Conditions Inputted Password Yes 4 No × 2 No 3 Considered a regularagrees user with User Data 1 Yes Inputted ID agrees with User Data Yes Yes No No × Considered a regular user × Can access an asset Conditions Inputted Password agrees with User Data Yes No Yes No Actions × CanConsidered accessaan asset × × × Considered non-regular user a regular user ˆ Actions × × × Considered a non-regular user Can access an asset ˆ × × × ActionsCannot access an asset Considered non-regular Cannot accessa an asset user Cannot access an asset

ˆ ˆ

× ˆ ˆ

׈ ˆ

×

Next, an aspect test template in AspectJ is created from the decision table and structure of the an aspect templateprocessing. in AspectJ is created the verify decision andstatus structure of theor patternNext, to observe thetest internal test from should thetable user (regular Next, an aspect test template in AspectJThe is created from the decision table and structure of pattern to observe the internal processing. The test should verify the user status (regular or non-regular user) whether to allowprocessing. access. In the diagram Figure(regular 1a, part of the pattern to and observe the internal Thestructure test should verify shown the userinstatus non-regular user) and whether to allow access. In the structure diagram shown in Figure 1a, part of “considered regular user non- toregular user” Inisthe the check_identification method of the or non-regular user) and or whether allow access. structure diagram shown in Figure 1a, “considered regular user or non- regular user” is the check_identification method of the part of “considered regular user non- regular user” isprocessing the check_identification of the and password_design_and_use class. Toorobserve the internal of this point,method its pointcut password_design_and_use class. To observe the internal processing of this point, its pointcut and password_design_and_use class. To observe the internal processing of this point, its pointcut and advice are defined. By observing the internal processing, it is possible to find a critical defect, which advice are defined. By observing the internal processing, it is possible to find a critical defect, which advice are defined. observing the testing. internal processing, it is possible to find a critical defect, which cannot be found using By browser-based cannot bebefound using browser-based testing. cannot found using browser-based testing. Figure 9a shows a pointcut executing the consideration of a regular regularor ornon-regular non-regularuser. user.Figure Figure 9b Figure ofaaregular Figure9a9ashows showsaapointcut pointcutexecuting executing the the consideration consideration of or non-regular user. Figure 9b 9b shows the advice to observe the pointcut result. a pointcut pointcut and andadvice adviceare aredefined definedforfor shows result. Similarly, Similarly, showsthe theadvice advicetotoobserve observe the the pointcut pointcut result. Similarly, aa pointcut and advice are defined for whether to allow orordeny InInthe structure diagram shown in in Figure1a, 1a, whethertotogrant grant whether denyaccess. access.In structure diagram shown whethertotoallow allow or deny access. thethe structure diagram shown in FigureFigure 1a, whetherwhether to grant access access is the subject_function method of the Subject_Control class. Figure 10a shows the pointcut to access the subject_function method of the Subject_Control class. Figure the is theissubject_function method of the Subject_Control class. Figure 10a shows10a theshows pointcut topointcut allow or to allow or or deny access totoanan asset. 10b shows the to determine determine the pointcutresult. result. allow deny access asset.Figure Figure 10b shows the advice to the pointcut deny access to an asset. Figure 10b shows the advice to advice determine the pointcut result.

(a) (a)

(b) (b)

Figure Pointcutconsidering considering a regular regular or non-regular non-regular user Figure 9. 9. (a) Pointcut or user and (b) (b) advicetotodifferentiate differentiate Figure 9.(a) (a) Pointcut considering a aregular or non-regular user and (b) and advice to advice differentiate between between the two types of users. between the two types of users. the two types of users.

(a)(a)

(b) (b)

Figure 10. (a) Pointcut judging access to an asset and (b) advice to allow access to an asset. Figure 10. (a) judging and(b) (b)advice advicetoto allow access toasset. an asset. Figure 10. Pointcut (a) Pointcut judgingaccess accessto toan an asset asset and allow access to an

Finally, a test case template is created from the decision table and pattern behavior. Figure 4 Finally, a test case template isiscreated from the decision decisiontable tableand and pattern behavior. Figure 4 a test case template created from the pattern behavior. Figure 4 shows Finally, the behavior of a login to which the Boolean value of a regular_user is returned when an actor shows the behavior of aoflogin to which the value of ofaaregular_user regular_user is returned whenactor an actor shows a login theBoolean Boolean is returned inputs anthe IDbehavior and password intotoawhich Login_UI as well value as the behavior by which an actorwhen sendsan a request inputs an ID and password into a Login_UI as well as the behavior by which an actor sends a request inputs an ID and password into a Login_UI as well the behavior by which an actor sends a request to the subject_controller. A test case template performs these behaviors and confirms whether the to subject_controller. the subject_controller. test casetemplate template performs performs these and confirms whether the the to the AA test case thesebehaviors behaviors and confirms whether internal process is correctly performed as actions of the decision table. internal process is correctly performed as actions of the decision table. internalFigure process correctly actions of the used decision table. a test and define “true_id”, 11isshows partperformed of the testascase template to create Figure 11 shows part ofthe thetest testcase case template template used to create a atest and define “true_id”, Figure 11 shows part of used to create test and “true_id”, “true_pass”, “false_id”, “false_pass”, and “request”. Because these templates dodefine not contain a “true_pass”, “false_id”, “false_pass”, and “request”. Because these templates do not contain a specific “true_pass”, “false_id”, “false_pass”, and “request”. Because these templates do not contain a specific value, the defined terms must be embodied in order to use them as a test. value, the defined terms must be embodied in order to use them as a test. specific value, the defined terms must be embodied in order to use them as a test.

Information 2016, 7, 34 Information 2016, 7, 34

11 of 19 11 of 19

Figure Figure11. 11.Part Partofofthe thetest testcase casetemplate. template.

4.4. 4.4.Limitations Limitations Some Someexpertise expertiseabout aboutsecurity securityand andOCL OCLmay maybe berequired requiredtotoprepare preparethe thedecision decisiontables tablesand andtest test templates. relation totothat, that, there a possibility that developers can prepare test templates. In In relation there is aispossibility that developers can prepare incorrectincorrect test templates templates original expectations of the corresponding design pattern by against theagainst originalthe expectations of the corresponding security design security pattern by misunderstanding misunderstanding the pattern. In the future, we plan to develop a technique and its implementing the pattern. In the future, we plan to develop a technique and its implementing tool to support tool to support from transformations from security designdecision patternstables to formal decision tables and transformations security design patterns to formal and templates. templates. There are two possible situations that would produce false positives: (1) developers apply situations that in would produce false positives: (1) developers apply thein the There patternare totwo the possible target implementation a slightly different manner from the case assumed pattern the target and implementation in a (2) slightly different manner from the assumed in theintest the testtotemplates test cases, and developers misunderstand thecase patterns applied the templates and test cases, and (2) developers misunderstand the patterns applied in theof corresponding design. Regarding (1), developers should prepare the corresponding test templates corresponding Regarding (1), developers should prepare the corresponding test templates such a variantdesign. of the pattern application. Regarding (2), since TESEM covers both the design phase ofand such a variant of the pattern application. Regarding (2), since TESEM covers both the design the implementation, it is easy to avoid such misunderstandings by using our tool beginning in the phase the implementation, it is easy to avoid such misunderstandings by using our tool designand phase. beginning inmethod, the design phase. In our only security design patterns that are described in general pattern templates with In our method, only security designAs patterns that earlier, are described in general pattern templates added OCL descriptions are test targets. mentioned we successfully defined UML models with added OCL descriptions are test targets. As mentioned earlier, we successfully defined UML with OCL descriptions for various security design patterns [7]. In the future, we have a plan to prepare models with of OCL descriptions various security design patterns [7]. In we have a plan a guideline describing OCL for descriptions for security patterns based onthe ourfuture, experience. to prepare a guideline of describing OCL descriptions for security patterns based on our experience. 5. Case Study 5. Case Study As a validation, we applied our method to an e-commerce system. Figure 12a,b shows the As a validation, weofapplied our method to an e-commerce system. Figure shows the structure and behavior the e-commerce system, respectively. “Password Design 12a,b and Use pattern”, structure the e-commerce system, respectively. Design Use pattern”, “Preventand SQLbehavior Injectionofpattern”, and “Role-based access control“Password pattern” are used and as the premises. “Prevent SQL Injection pattern”, and “Role-based access control pattern” are used as the premises. We embody a test case using previously defined design information (e.g., the make_a_payment We embody a test case using defined design information the make_a_payment method of Payment_Control classpreviously due to apply “Password Design and (e.g., Use pattern” in Figure 12a) method Payment_Control classitem dueand to apply Design Use pattern” (Step 1).of Additionally, “select push “Password a button” is used and to access an assetininFigure Figure12a) 12b. (Step 1). Additionally, “select item and push a button” is used to access an asset in Figure 12b. Figure Figure 13 shows part of the created test case. In Figure 13, the “make_a_payment_test” method13is shows part in of the the “request” created test case. In the “make_a_payment_test” is embodied in embodied method inFigure Figure 13, 11 because the “request” methodmethod in the test case template the “request”tomethod in Figure 11 because the “request” method in the test template corresponds the “make_a_payment_test” method in the system. Stereotypes, such case as corresponds to the “make_a_payment_test” method in the system. Stereotypes, such as and , attached to the class diagram show these corresponding relations. and , attached to the class diagram show these corresponding relations.

Information Information2016, 2016,7,7,34 34

12 of12 19of 19

Information 2016, 7, 34

12 of 19

(a)(a) : actor : actor : Login_UI : prevent_SQL_Injection : password_design_and_use : User_Data : Login_UI : prevent_SQL_Injection : password_design_and_use : User_Data

: RBAC Cart : RBAC: :payment_UI payment_UI:: Patment_Control Patment_Control : :Cart

1: login() 1: login() 1.1: sanitize_input_data() 1.1: sanitize_input_data() check_identification() 1.1.1:1.1.1: check_identification() 1.1.1.1: identify() 1.1.1.1: identify() 1.1.1.1.1: access_control() 1.1.1.1.1: access_control() 1.1.1.2: transition() 1.1.1.2: transition() 2: select item()

2: select item()

2.1: make_a_payment()

2.1: make_a_payment() 2.1.1: payment() 2.1.1: payment()

(b)

(b)

Figure 12. Structure and behavior of a purchasing system. (a) Structure; (b) Behavior.

Figure Structureand andbehavior behaviorofofa apurchasing purchasing system. Structure; Behavior. Figure 12. 12. Structure system. (a)(a) Structure; (b) (b) Behavior.

Figure 13. Part of the created test case.

Figure test case. Figure13. 13.Part Partofofthe thecreated created test case.

Information 2016, 7, 34

13 of 19

Information 2016, 7, 34

13 of 19

After implementing the design (Step 2), a test is executed to validate the application of the patterns After2016, implementing design (Step 2),14a shows test is executed validate the application the in the implementation (Step 3). Figure the test to result for the “Passwordof and Information 7, 34 phase the 13Design of 19 patterns in the implementation phase (Step 3). Figure 14 shows the test result for the “Password Use pattern”. Design andimplementing Use pattern”. the design (Step 2), a test is executed to validate the application of the After patterns in the implementation phase (Step 3). Figure 14 shows the test result for the “Password Design and Use pattern”.

Figure 14. Test results (Password Design and Use pattern).

Figure 14. Test results (Password Design and Use pattern). Next, the implementation fixed (Step 4). The errorand message in Figure 14 indicates that Figure 14.is Test results (Password Design Use pattern). “whether to allow or deny access” thein implementation is retested and Next, the implementation is fixedcannot (Stepbe 4).determined. The error Finally, message Figure 14 indicates that “whether patterns are re-validated (Step 5). Figure which shows in theFigure re-test 14 results, confirms Next, access” the implementation is fixed (Step 4). The15, error message indicates that applied to allowthe or applied deny cannot be determined. Finally, the implementation is retested and the that “Password Design andaccess” Use pattern”, “Prevent SQL Injection pattern” and “Role-based Access “whether to allow or deny cannot be determined. Finally, the implementation is retested and patterns are re-validated (Step 5). Figure 15, which shows the re-test results, confirms that “Password Control pattern” are applied appropriately. the applied patterns are re-validated (Step 5). Figure 15, which shows the re-test results, confirms Design that and“Password Use pattern”, “Prevent SQL Injection pattern” and “Role-based Access Control pattern” Design and Use pattern”, “Prevent SQL Injection pattern” and “Role-based Access are applied appropriately. Control pattern” are applied appropriately.

Figure 15. Re-test results.

6. Evaluation

Figure15. 15. Re-test Re-test results. Figure results.

6.1. Experimental Overview 6. Evaluation

6. Evaluation

To evaluate the Research Questions, we conducted experiments involving four students in 6.1. Experimental Overview information sciences. We estimated the defect detection and correct implementation of security 6.1. Experimental Overview design patterns by our technique and an we existing method for a login involving system offour an e-commerce To evaluate the Research Questions, conducted experiments students in system and EMSsec [40]. EMSsec is a student information management system, was information sciences. We estimated the defect detection and correct implementation of security To evaluate the Research Questions, we conducted experiments involving which four students in developed as a measure to address a common problem regarding security and privacy. We injected design patterns by our technique and an existing method for a login system of an e-commerce information sciences. We estimated the defect detection and correct implementation of security three defects the ones shown ina Figure 3 into these systems. system and similar EMSsecto[40]. student information management system, which was design patterns by our techniqueEMSsec and anisexisting method for a login system of an e-commerce system As theasbaseline for tothe comparison, the problem existing regarding method issecurity one where general testing developed a measure address a common and only privacy. We injected and EMSsec [40]. EMSsec is aones student management system, which developed as environments, including and information Selenium, Additionally, there iswas no further three defects similar to the JUnit shown in Figure 3are intoprovided. these systems. a measure to a common problem regarding security and privacy. We injected three defects support such as test case Thus, the existing method Asaddress the baseline for generation. the comparison, existing methodis iscompletely one wheremanual. only general testing Students were asked to find and correct the three defects. Our experiment involved the similar environments, to the ones shown in Figure 3 into these systems. including JUnit and Selenium, are provided. Additionally, there is no further following steps: such as test Thus,the the existing is completely manual.only general testing Assupport the baseline forcase thegeneration. comparison, existingmethod method is one where Students were JUnit asked to find and correct the three defects. Our experiment thesupport environments, including and Selenium, areexperiment. provided. Additionally, there is involved no further Step 1: Complete a questionnaire before the following steps: Step 2: Listen to an explanation of our method, JUnit, and Selenium.

such as test case generation. Thus, the existing method is completely manual. Stepwere 3: andacorrect theand defects by the existing (our) method. Our experiment involved the 1: Detect Complete questionnaire before experiment. Students asked to find correct the three defects. Step 4: and the defects our (existing) method. 2: Detect Listen to ancorrect explanation of ourby method, JUnit, and Selenium. following steps: Step 5: questionnaire afterby the experiment. 3: Complete Detect andacorrect the defects the existing (our) method. Step 4: Detect and correct the defects by our (existing) method. Step Questionnaire before the experiment Step 1: 1:Complete a questionnaire before the experiment. Step 5: Complete a questionnaire after the experiment. Step 2:To Listen to anstudents’ explanation our completed method, JUnit, and Selenium. measure ability,ofthey a questionnaire prior to participating in the Step 1: Questionnaire before the experiment experiment. questionnaire of the Step 3: DetectThe and correct theconsisted defects by thefollowing: existing (our) method.  What is your students’ skill level regarding developing a program? measure ability, they completed a questionnaire Step 4:To Detect and correct the defects by our (existing) method.prior to participating in the  What is The yourquestionnaire knowledge level regarding experiment. consisted of thesecurity? following: Step Complete questionnaire after thesoftware experiment.  5:What is youraknowledge level regarding testing? skill level regarding developing a program?  What is your knowledge level regarding web systems development? security? Step Questionnaire before the experiment selected from the following answers: “very high”, “high”, “not high”, and “low.”  1:Students What is your knowledge level regarding software testing?  What is your knowledge level regarding web systems development? prior to participating in the To measure students’ ability, they completed a questionnaire Students selected from the following answers: “very high”, “high”, “not high”, and “low.”

experiment. The questionnaire consisted of the following: ‚ ‚

What is your skill level regarding developing a program? What is your knowledge level regarding security?

Information 2016, 7, 34

14 of 19

Information 2016, 7, 34

14 of 19

‚ What is your knowledge level regarding software testing? Step 2: Listen to an explanation of our method, JUnit, and Selenium. ‚ What is your knowledge level regarding web systems development? We explained how to use our method, JUnit, and Selenium. Students selected from the following answers: “very high”, “high”, “not high”, and “low.” Steps 3 and 4: Detect and correct the defects by the existing method and our method. Step 2: Listen to an explanation of our method, JUnit, and Selenium. Students detected and corrected defects using our method and the existing method. The We explained how to divided use our method, andThe Selenium. students were randomly into twoJUnit, groups. first group of students 1 and 2 used the existing method followed by our method, while the second group of students and 4 used our Steps 3 and 4: Detect and correct the defects by the existing method and our 3method. method first. Students detected and corrected defects using our method and the existing method. The students Steprandomly 5: Complete the after-experiment questionnaire. were divided into two groups. The first group of students 1 and 2 used the existing method followed by our method, while the second groupcompleted of studentsa 3questionnaire and 4 used our method Upon completing the experiment, students about theirfirst. experience. The Step questionnaire contained the following three items: 5: Complete the after-experiment questionnaire.  Which method do you prefer to find defects in the software? completing experiment, questionnaire about their experience. The  Upon Which method dothe you feel is morestudents efficientcompleted at finding adefects? questionnaire contained the following three items:  Which method do you prefer to modify defects in the software?

‚ Which method do you prefer to find defects in the software? 6.2. Experimental Results ‚ Which method do you feel is more efficient at finding defects? ‚ Table Which method do you to the modify defects in the software? (Step 1, 5). 2 below shows the prefer result of before and after questionnaire 6.2. Experimental Results Table 2. Results of the Before and After Questionnaire (Step 1, 5) (VH: very high, H: high, NH: not high, 2L:below low, O:shows our method, RO: of relatively our method, RE:questionnaire relatively existing method, Table the result the before and after (Step 1, 5). E: existing method).

Table 2. Results of the Before and After Questionnaire (Step 1, 5) (VH: very high,Which H: high, NH: not high, Which Which L: low, O:What our method, RO: relatively ourismethod, RE: relatively existing method, E: existing method). is What is What your method do your skill level Students What is your regarding skill level developing Students regarding a program? developing a 1 2 3 4

1 2 3 4

your knowledge What is your level knowledge regarding level security? regarding

knowledge levelis your What knowledge regarding level software regarding testing?

What is your method do skill level you prefer Which regarding when What is yourweb method do skill level systems youfinding prefer regarding development? when defects in finding web systems defects in software? development? NH NH NH NH LL NH NH

program?

security?

software testing?

H H NHNH NHNH NHNH

NH NH L L NH NH HH

H H NH NH NH NH NH NH

software? O O O O

O O O O

you feel more Which method do efficiently you feel more finds efficiently defects in finds defects insoftware? software? O O OO OO OO

method do you prefer Which method when do you prefer modifying when defects in modifying defects in software? software?

O O O O RO RO RO

Figures 16 16 and and 17 17 show show scatter scatter plots plots of of the the results results for Figures for each each student. student. Using Usingour ourmethod, method,the the detection time decreased for all students. In addition, Student 2 detected more defects, whereas detection time decreased for all students. In addition, Student 2 detected more defects, whereas Students1,1,3,3,and and44detected detected the the same same number number of Students of defects defectsusing usingboth bothmethods. methods.As Asfor forthe thecorrection correction time, the second method (i.e., our method for students 1 and 2, and the existing method time, the second method (i.e., our method for students 1 and 2, and the existing method for for students students 3 3 and 4) required less time regardless of which method was used. and 4) required less time regardless of which method was used.

Number of Detected Defects

Number of Detection and Detection Time (Existing Method) 4 3 2 1

0 0

20

40

60

80

Detection Time (min) Student 1

Student 2

Student 3

(a) Figure 16. Cont.

Stundet 4

100

Information 2016, 7, 34 Information 2016, 7, 34

15 of 19 15 of 19

Defects Detected Defects of Detected Number of Number

Number of Detection and Detection Time (Our Method) 4 3 2 1

0 0

20

40

60

80

100

Detection Time (min) Student 1

Student 2

Student 3

Student 4

(b) Figure 16. Scatter plots of (a) the existing method and (b) our method for the number detected Figure 16. Scatter plots of (a) the existing method and (b) our method for the number detected defects defects and detection time. and detection time.

Correction of Correction Number of Number

Number of Correction and Correction Time (Existing Method) 4 3 2 1

0 0

5

10

15

20

25

30

35

40

Correction Time (min) Student 1

Srudent 2

Student 3

Student 4

(a)

Correction of Correction Number of Number

Number of Correction and Correction Time (Our Method) 4 3 2 1

0 0

5

10

15

20

25

30

35

40

Correction Time (min) Student 1

Srudent 2

Student 3

Student 4

(b) Figure 17. Scatter plots of (a) the existing method and (b) our method for the number of corrected Figure 17. Scatter plots of (a) the existing method and (b) our method for the number of corrected defects and correction time. defects and correction time.

6.3. Discussion 6.3. Discussion Based on the results, we answer our RQs. Based on the results, we answer our RQs. RQ 1:RQ Is our method more efficient at finding defects when when implementing a security designdesign 1: Is our method more efficient at finding defects implementing a security pattern pattern than than an an existing existing method? method? Regardless Regardless of of whether whether our our method method was was implemented implemented before before or or after after the the existing existing method, method, our our method required less time as shown in Figure 16. Thus, we conclude that our method can find method required less time as shown in Figure 16. Thus, we conclude that our method can find implementation defects more moreefficiently efficiently than existing method. By focusing on a particular implementation defects than thethe existing method. By focusing on a particular security security design implementation, pattern implementation, our allows methodtest allows test templates to be andtotest design pattern our method templates to be reused andreused test cases be cases to be semi-automatically generated; it contributes to such time reduction. semi-automatically generated; it contributes to such time reduction. 2: Does our method create a more effective testfind to find implementation defects a security RQ 2:RQ Does our method create a more effective test to implementation defects in ainsecurity design pattern than an existing method? design pattern than an existing method?

Information 2016, 7, 34 Information 2016, 7, 34 Information 2016, 7, 34

16 of 19 16 of 19 16 of 19

All students successfullydetected detected more defects defects using our method than the existing method. AllAll students usingour ourmethod methodthan thanthe the existing method. studentssuccessfully successfully detected more more defects using existing method. Moreover, students 1, 2, and 3 found all three defects with our method, confirming that our method Moreover, students 1, 2, and 3 found all three defects with our method, confirming that our method Moreover, students 1, 2, and 3 found all three defects with our method, confirming that our method can effectively test implementation defects. cancan effectively test effectively testimplementation implementationdefects. defects. In our method, test cases are semi-automatically generated from test templates by only In In ourour method, test cases are semi-automatically generated from test by onlyby inputting method, test cases are semi-automatically generated fromtemplates test templates only inputting the necessary information for binding class/implementation names in the given design theinputting necessarythe information for binding class/implementation names in the given design and elements necessary information for binding class/implementation names in the given design and elements in the patterns together with correct/incorrect input data and Selenium code. We in and the patterns with correct/incorrect input data andinput Selenium We believe that this elementstogether in the patterns together with correct/incorrect data code. and Selenium code. We believe that this feature somewhat mitigates the likelihood of incorrect test cases. Nevertheless, as believe that this mitigates feature somewhat mitigates the likelihood of incorrect test cases. feature somewhat the likelihood of incorrect test cases. Nevertheless, as weNevertheless, mentioned inasthe we mentioned in the limitations section, there is a possibility that developers can prepare incorrect we mentioned in there the limitations section, is a possibility that developers cantemplates prepare incorrect limitations section, a possibility thatthere developers cancorresponding prepare incorrect test against test templates against is the original expectations of the security design pattern bythe test templates against thecorresponding original expectations ofdesign the corresponding security design pattern by original expectations of the security pattern by misunderstanding the pattern. misunderstanding the pattern. misunderstanding the pattern. RQ3: Compared to an existing method, does our method more effectively create a test to find RQ3: Compared to an existing method, does our method more effectively create a test to find implementation defects in a security design and allow for a regression RQ3: Compared to an existing method, doespattern our method more effectively create test? a test to find implementation defects in a security design pattern and allow for a regression test? implementation defects in a security design pattern and allow for a regression test? Figures 18 and 19 show the test case developed by Student 1. Figure 18 is the test case, which was Figures 18 and 19 show the test case developed by Student 1. Figure 18 is the test case, which rewritten for every test. test case could not reused,1.whether the results arewhich correct Figures 18 and 19 Because show thethe test case developed bybeStudent Figure 18 is test the test case, was rewritten for every test. Because the test case could not be reused, whether the test results are cannot be confirmed. In contrast, the test case in Figure 19reused, can be reused, the accuracy was rewritten for every test. Because the testshown case could not be whetherallowing the test results are correct cannot be confirmed. In contrast, the test case shown in Figure 19 can be reused, allowing the cannot beconfirmed. confirmed.Students In contrast, the 4test case shown in Figure 19to can be reused, allowing the of correct the result to be 2 and created test cases similar Figure 18 (Figure 19) when accuracy of the result to be confirmed. Students 2 and 4 created test cases similar to Figure 18 (Figure 19) accuracy of the result to method. be confirmed. Students 2 and 4 created test cases similar to (Figure 19) using theusing existing (our) Using our method, all students were ableable toFigure run a 18 regression test. when the existing (our) method. Using our method, all students were to run a regression when using the existing (our) method. Using our can method, all students weretest. able to run a regression From these results, we conclude that our method develop a regression test. From these results, we conclude that our method can develop a regression test. test. From these results, we conclude that our method can develop a regression test.

Figure 18. Part of the test case for Student 1 using the existing method. Figure Student 11using usingthe theexisting existingmethod. method. Figure18. 18.Part Partof ofthe thetest test case case for Student

Figure 19. Part of the test case for Student 1 using our method. Figure19. 19.Part Partof ofthe the test test case case for Figure for Student Student11using usingour ourmethod. method.

Information 2016, 7, 34

17 of 19

RQ 4: Is our method more effective at correcting implementation defects in a security design pattern compared to an existing method? Although students 1, 3, and 4 found the same number of corrections using both methods, student 2 was only able to modify one defect using the existing method, but modified all three defects using our method. These findings demonstrate that our method can help correct implementation defects of security design patterns for users with low skills about web systems development compared to the existing method. However, our method did not shorten the correction time because it is assumed that more time is necessary to infer the correction contents from the test case generated by a tool automatically. We believe that the semi-automated detection of implementation defects supports further corrections since testers or developers can easily confirm whether or not their implementation under testing is truly correct against security design patterns. However, as mentioned previously, our method does not directly support defect correction. 6.4. Threats to Validity In the experiment, we did not verify whether our method is applicable to any type of system. Additionally, the numbers of security design patterns and developers were insufficient. Hence, it is possible that our method is not applicable to all security patterns. Although we used representative patterns and a typical model for software development to demonstrate the usefulness of our method, we need to examine more general patterns and employ large-scale examples. 7. Conclusions and Future Work Because software developers are not necessarily security experts, security design patterns may be inappropriately applied. Additionally, even if security design patterns are properly applied in the design phase of software development, threats and vulnerabilities may not be mitigated in the implementation phase. Hence, we propose a support method for security design patterns in the implementation phase of software development. Our method offers two significant contributions. First, it can create an effective test to find the implementation defects of a security design pattern irrespective of the user’s background. Second, compared to the existing method, our method can help correct defects for users with a low web development ability. There are several limitations to our method. Some expertise about security and OCL may be required to prepare the decision tables and test templates. Moreover, there is a possibility that developers can prepare incorrect test templates against the original expectations of the corresponding security design pattern by misunderstanding the pattern. To mitigate the limitation, we plan to develop a technique and its implementing tool to support transformations from security design patterns to formal decision tables and templates. Moreover, in our future work to evaluate the scope of our method, we intend to perform an experiment using a system with more complex defects. Acknowledgments: This work was supported by JSPS KAKENHI Grant Numbers 25330091, 15H02686, 16H02804, and IISF SSR Forum 2015 and 2016. Author Contributions: Masatoshi Yoshizawa and Hironori Washizaki conceived and designed the method, performed the experiments, and wrote the paper. Yoshiaki Fukazawa, Takao Okubo, Haruhiko Kaiya and Nobukazu Yoshioka designed the method and the experiments. Conflicts of Interest: The authors declare no conflict of interest.

Information 2016, 7, 34

18 of 19

References 1. 2. 3. 4. 5. 6. 7.

8. 9.

10.

11. 12.

13. 14.

15.

16. 17. 18. 19. 20.

21.

Yoshioka, N.; Washizaki, H.; Maruyama, K. A Survey on Security Patterns. Prog. Inform. 2008, 5, 35–47. [CrossRef] Devanbu, P.T.; Stubblebine, S. Software Engineering for Security: A Roadmap. In Proceedings of the Conference on the Future of Software Engineering, New York, NY, USA, 4–10 June 2000; pp. 227–239. Schumacher, M.; Fernandez-Buglioni, E.; Hybertson, D.; Buschmann, F.; Sommerlad, P. Security Patterns: Integrating Security and Systems Engineering; John Wiley & Sons: Chichester, UK, 2006. Object Management Group. Unified Modeling Language (UML) Version 2.5, 2015. Available online: http://www.omg.org/spec/UML/ (accessed on 3 June 2016). Object Management Group. OCL 2.4, 2014. Available online: http://www.omg.org/spec/OCL/2.4/ (accessed on 9 June 2016). Warmer, J.; Kleppe, A. The Object Constraint Language: Precise Modeling with UML; Addison-Wesley: Reading, MA, USA, 1998. Kobashi, T.; Yoshizawa, M.; Washizaki, H.; Fukazawa, Y.; Yoshioka, N.; Kaiya, H.; Okubo, T. TESEM: A Tool for Verifying Security Design Pattern Applications by Model Testing. In Proceedings of the 8th IEEE International Conference on Software Testing, Verification, and Validation (ICST 2015), Graz, Austria, 13–17 April 2015; pp. 13–17. Fernandez-Buglioni, E. Security Patterns in Practice: Designing Secure Architectures Using Software Patterns; John Wiley & Sons: Chichester, UK, 2013. Munetoh, S.; Yoshioka, N. Model-Assisted Access Control Implementation for Code-centric Ruby-on-Rails Web Application Development. In Proceedings of the International Conference on Availability, Reliability and Security, Regensburg, Germany, 2–6 September 2013; pp. 350–359. Abramov, J.; Shoval, P.; Sturm, A. Validating and Implementing Security Patterns for Database Applications. In Proceedings of the 3rd International Workshop on Software Patterns and Quality (SPAQu), Orlando, FL, USA, 25 October 2009. Dong, J.; Peng, T.; Zhao, Y. Automated verification of security pattern compositions. J. Inf. Softw. Technol. 2010, 25, 274–295. [CrossRef] Hamid, B.; Percebois, C.; Gouteux, D. Methodology for Integration of Patterns with Validation Purpose. In Proceedings of the European Conference on Pattern Language of Programs (EuroPLoP), Irsee, Germany, 11–15 July 2012; pp. 1–14. Jürjens, J. Secure Systems Development with UML; Springer: Berlin, Germany, 2005. Lodderstedt, T.; Basin, D.A.; Doser, J. SecureUML: A UML-Based Modeling Language for Model-Driven Security. In Proceedings of the 5th International Conference on the Unified Modeling Language (UML), Dresden, Germany, 30 September–4 October 2002; pp. 426–441. Dalal, S.R.; Jain, A.; Karunanithi, N.; Leaton, J.M.; Lott, C.M.; Patton, G.C.; Horowitz, B.M. Model-based Testing in practice. In Proceedings of the International Conference on Software Engineering, Los Angeles, CA, USA, 16–22 May 1999; pp. 285–294. Felderer, M.; Zech, P.; Breu, R.; Büchler, M.; Pretschner, A. Model-based security testing: A taxonomy and systematic classification. Softw. Test. Verif. Reliab. 2016, 26, 119–148. [CrossRef] Tretmans, J.; Brinksma, E. TorX: Automated Model-Based Testing. In Proceedings of the Conference on Model-Driven Software Engineering, Nuremberg, Germany, 11–12 December 2003; pp. 11–12. Felderer, M.; Agreiter, B.; Breu, R.; Armenteros, A. Security Testing by Telling TestStories. In Proceedings of the Conference on Modellierung, Klagenfurt, Austria, 24–26 March 2010; pp. 24–26. Schieferdecker, I.; Grossmann, J.; Schneider, M. Model-based security testing. In Proceedings of the 7th Workshop on Model Based Testing, Tallinn, Estonia, 25 March 2012; pp. 1–12. Kim, H.; Choi, B.; Yoon, S. Performance testing based on test-driven development for mobile applications. In Proceedings of the International Conference on Ubiquitous Information Management and Communication, Suwon, Korea, 15–16 January 2009; pp. 612–617. Fraser, S.; Astels, D.; Beck, K.; Boehm, B.; McGregor, J.; Newkirk, J.; Poole, C. Discipline and practices of TDD: (test driven development). In Proceedings of the Conference on Object-oriented Programming, Systems, Languages, and Applications, Anaheim, CA, USA, 26–30 October 2003; pp. 268–270.

Information 2016, 7, 34

22.

23. 24. 25. 26. 27. 28.

29. 30.

31.

32.

33.

34.

35.

36. 37.

38.

39. 40.

19 of 19

Sonia; Singhal, A. Development of Agile Security Framework Using a Hybrid Technique for Requirements Elicitation. In Proceedings of the International Conference on Advances in Computing, Communication and Control (ICAC), Mumbai, India, 28–29 January 2011; pp. 178–188. Selenium. Available online: http://docs.Seleniumhq.org/ (accessed on 9 June 2016). Holmes, A.; Kellogg, M. Automating functional tests using Selenium. In Proceedings of the Agile Conference, Minneapolis, MN, USA, 23–28 July 2006; p. 6. JUnit. Available online: http://junit.org (accessed on 3 June 2016). Tahchiev, P.; Leme, F.; Massol, V.; Gregory, G. JUnit in Action; Manning Publications Co.: Shelter Island, NY, USA, 2010. Copeland, L. A Practitioner’s Guide to Software Test Design; Artech House: London, UK, 2004. Di Lucca, G.A.; Fasolino, A.R.; Faralli, F.; de Carlini, U. Testing Web Applications. In Proceedings of the 18th International Conference on Software Maintenance (ICSM), Montreal, QC, Canada, 3–6 October 2002; pp. 310–319. Noikajana, S.; Suwannasart, T. Web Service Test Case Generation Based on Decision Table. In Proceedings of the 8th International Conference on Quality Software (QSIC), Oxford, UK, 12–13 August 2008; pp. 321–326. Hartmann, J.; Imoberdorf, C.; Meisinger, M. UML-Based Integration Testing. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), Portland, OR, USA, 21–24 August 2000; pp. 60–70. Briand, L.; Labiche, Y. A UML-Based Approach to System Testing. In Proceedings of the 4th International Conference on the Unified Modeling Language: Modeling Languages, Concepts, and Tools (UML), Toronto, ON, Canada, 1–5 October 2001; pp. 194–208. Endrikat, S.; Hanenberg, S. Is Aspect-Oriented Programming a Rewarding Investment into Future Code Changes? A Socio-technical Study on Development and Maintenance Time. In Proceedings of the International Conference on Program Comprehension, Kingston, ON, Canada, 22–24 June 2011; pp. 51–60. Kiczales, G.; Hilsdale, E.; Hugunin, J.; Kersten, M.; Palm, J.; Griswold, W. An overview of AspectJ. In Proceedings of the Conference on Object-Oriented Programming, Budapest, Hungary, 18–22 June 2001; pp. 327–354. Tanter, É. Execution levels for aspect-oriented programming. In Proceedings of the 9th International Conference on Aspect-Oriented Software Development, Rennes and Saint Malo, France, 15–19 March 2010; pp. 37–48. Montrieux, L.; Jürjens, J.; Haley, C.B.; Yu, Y.; Schobbens, P.Y.; Toussaint, H. Tool support for code generation from a UMLsec property. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering, Antwerp, Belgium, 20–24 September 2010; pp. 357–358. TESEM. Available online: http://patterns.fuka.info.waseda.ac.jp (accessed on 3 June 2016). Kobashi, T.; Yoshioka, N.; Okubo, T.; Kaiya, H.; Washizaki, H.; Fukazawa, Y. Validating Security Design Pattern Applications Using Model Testing. In Proceedings of the 8th International Conference on Availability, Reliability and Security (ARES2013), Regensburg, Germany, 2–6 September 2013; pp. 62–71. Yoshizawa, M.; Kobashi, T.; Washizaki, H.; Fukazawa, Y.; Okubo, T.; Kaiya, H.; Yoshioka, N. Verification of Implementing Security Design Patterns Using a Test Template. In Proceedings of the 9th International Conference on Availability, Reliability and Security (ARES2014), Fribourg, Switzerland, 8–12 September 2014; pp. 178–183. Kobashi, T.; Yoshioka, N.; Kaiya, H.; Washizaki, H.; Okubo, T.; Fukazawa, Y. Validating Security Design Pattern Applications by Testing Design Models. Int. J. Secure Softw. Eng. 2014, 5, 1–30. [CrossRef] EMSsec. Available online: http://lab.iisec.ac.jp/~{}okubo_lab/Members/okubo/wiki/index.php?EMSSec (accessed on 3 June 2016). © 2016 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).