Session S1G INTEGRATING TESTING AND DESIGN METHODS FOR UNDERGRADUATES: TEACHING SOFTWARE TESTING IN THE CONTEXT OF SOFTWARE DESIGN Stephen Frezza1 Abstract Software testing is a challenging topic for undergraduates, but when integrated into a software design course provides many opportunities to enhance student learning in both testing and design. This paper describes an innovative approach to integrating testing into an introductory software design course for Computer Science and Engineering Undergraduates. The paper outlines the rational and benefits of the approach. Index Terms Teaching Software Design, Teaching Software Quality, Teaching Software Testing
INTRODUCTION The rational for integrating software testing topics concurrently with software design topics stems from several sources, both pedagogical and theoretical: 1. 2.
3.
Pedagogical: need to provide adequate motivation for software testing topics Pedagogical: need for students to have skills in software and test design prior to their application in project courses (capstone and other) Theoretical: desire to demonstrate the inherent relationship between design methods and test design methods
The teaching approach presented here is to intertwine introductory design concepts/techniques with associated testing methods. The integration of testing and design topics allows students to experience different approaches to the design concepts, and reinforces the life-cycle value of testing (e.g., early test planning). This results in students having a stronger mix of design skills to apply to project course work and internship work. The motivation for this idea stems from common industry complaint that students do not understand (or fit into) robust software processes [1]. The complaint is leveled directly at the education common to most entry-level software engineers. The unstructured, individual-oriented programming techniques common to entry-level engineers are no longer sufficient, and can hamper companies’ efforts to improve their software processes [2,3]. These insufficiencies constitute a gap between undergraduate software engineering education and software engineering in practice. A recent Software Engineering Institute (SEI) report succinctly states the problem thus: “Students are not prepared to make the jump 1
between computer science or software engineering in school to software engineering in the professional world”[3]. By integrating software design and software testing techniques, students are armed to step beyond unstructured, individualoriented programming and apply these skills in their project courses and in their professional careers.
BACKGROUND The Software Engineering project courses are common to many Computer Science (CS), Software Engineering (SE) and many Computer Engineering (CE) undergraduate programs. Many of these courses include solving openended problems in teams [4]. These projects often require the students to apply their skills and knowledge in requirements analysis, software design, implementation, testing, as well as project management and team skills [4]. For most such course instances, there is not enough time in the course for students to both learn and apply all of these knowledge and skill areas in their project. One approach to this limitation is to integrate testing concepts into their programming courses. The difficulty here is in demonstrating the value and application of test design to earlier life-cycle products, such as requirements and design. This approach centers on integrating the presentation of OO testing and design topics, so that students enter their project courses armed with 1) the application of design/analysis techniques to problems, and 2) the ability to formulate and execute test plans from the design models they develop. This motivating this integration to students works well because many OOT patterns rely on OOA/D models as a prerequisite [5]. Integration also brings with it learning advantages in both design and testing. In particular, students can experience first-hand the motivation for developing test plans early in the life-cycle, and the ability to use their test-case designs to help analyze the quality of their design artifacts.
COURSE IMPLEMENTATION Entitled “Software Design & Test,” the course has as its prerequisite an OO form of CS2 (Data Structures in C++). The course is required of all Computer Science (BSCS)and Electrical Engineering Computer and Software Option (BSEE) majors. It is typically taken in the students’ fifth semester, and most students have completed an introductory course in Java as well. Thus all students have completed two
Stephen Frezza, Gannon University, Electrical & Computer Engineering, 109 University Square MB3181, Erie, PA 16541
[email protected]
0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference S1G-1
Session S1G semesters of C++ programming, and most have three or even four semesters of programming behind them. The course serves as the prerequisite for a software engineering project course.
TABLE II SOFTWARE DESIGN & T EST COURSE OUTLINE Review of Object Orientation Introduction to Testing Combinational Test Models Structural Modeling Dynamic Modeling State Machine Design State Machine Testing Inheritance & Polymorphism Class Testing Patterns Architectural Modeling Integration Testing and Test Patterns
Our Software Design & Test course uses the following course description: An advanced treatment of methods for producing a software design, the testing of that design and ensuing code. Focus is on Object-Oriented analysis and design methods, black -box (functional) testing techniques. Includes treatment Unified Modeling Language (UML) techniques and their application to software development. Following this title and description, the focus is on achieving the three learning objectives listed in Table I. Central to these learning objectives is the connection: Design and testing go together: Good design includes test design prior to implementation. The course begins with a brief review of object-orientation and follows with introductory testing topics. This permits students to start into a programming assignment and review fundamentals from other courses. Table II below briefly outlines the course organization. TABLE I ♦
SOFTWARE DESIGN & T EST COURSE OBJECTIVES Develop a suitable object-oriented design based on requirements Identify Strengths and weaknesses of different software designs
♦
Develop and apply appropriate tests for object -oriented designs
♦
Finding a suitable starting point has turned out to be critical to successful assimilation of the course material, because much of the knowledge and skills applied in each unit is dependent on the knowledge and skills in previous units and prerequisite OOP courses. Thus should be ready to develop working code for small problems. On repeated occasions, many students have entered the course who are competent in developing code, but few have a solid grasp of OO fundamentals and even fewer have ever attempted more than a cursory approach to testing. This experience led to the decision to start the course with a review of the OO paradigm and an OO programming assignment to use for design analysis later in the course. As Table II indicates, we follow up immediately with introductory testing material. The goal of the course is to teach the students fundamental design techniques, and to help them master the skills and techniques for testing: that testing can become a mental discipline that can help them at all phases of the software lifecycle. Table III summarizes the key motivational points for the testing material: moving up the stages of ‘Mental Discipline’[7]. The introductory testing material focuses on combinatorial test techniques, and uses a web form (Java Applets) of the testing classic “Triangle Problem” [5,8].Over
the next weeks, students plan, design and implement ad-hoc, control-flow, Binary Decision Diagram (BDD), and Domain tests for the problem [5]. With test plans from the four techniques in hand, they then re-run all four on a new implementation and compare the cost & effectiveness of the results. We deliberately chose to NOT have students write and test their own code, as the immediate focus is on testing, and their difficulties in implementation can distract them from the testing concepts. (Experience has shown that not all students are successful at getting their first OO problem working!) Once students have the testing fundamentals, the course shifts back to design topics and UML notation [9]. These topics include structural (class) modeling, dynamic (interaction) modeling and state machine (State Chart) modeling. From this point forward in the course, the design and testing topics are integrated in their presentation: State machine design preceeds FSM testing, inheritance/ polymorphism preceeds class testing, architectural modeling preceeds integration testing. TABLE III 0 1
P HASES IN A T ESTER’ S MENTAL LIFE Testing = Debugging: testing exists to support debugging Show that software works
2 3 4
Show that software doesn’t work Reduce perceived risk of software failure Mental discipline: Plan testing before, during design
INTEGRATING DESIGN & TESTING The issue of dependency and sequencing among course topics is critical to the success of the integration of the design and testing topics. However, proper synchronization of topics and assignments is also one of the strengths of the approach, particularly in helping students synthesize their knowledge and experience to grow in their ‘mental discipline’ as testers. For example, a topic like State Machine Testing has the prerequisite knowledge of State Machine Design, Structural (Class) modeling, and Combinatorial Test Models. The advantage is that most of the dependency among the more complex topics is on lower cognitive levels of understanding, such as terms and definitions. Thus a student
0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference S1G-2
Session S1G may not understand well a design topic prerequisite (like State Machine Design), but will have another opportunity to learn the topic while assimilating the equivalent testing pattern (e.g., State Machine Testing). The cost of including the testing pattern is less class time to go into greater depth in design topics or a detailed approach to the UML and associated tools (in this case, Rational Rose). However, one of the big advantages of integrating the teaching of testing models and techniques alongside of their counterpart design concepts is that students inherently get two different but mutually-reinforcing approaches to design. The concepts presented in each test model map well to the design techniques, and include very practical model checklists [5,6], which strongly reinforce concepts related to what makes a good design. This is the first of the pedagogical reasons for integrating testing into design topics: students get two approaches to the design concepts: Fundamentals and introductory patterns from their design text/notes, and an analysis/quality perspective from their testing text/notes. The counterpart to this pedagogical rational for integrating testing and design concepts is theoretical: demonstrating the early life-cycle value of testing. The pithy phrase we use to summarize this is “the best test is the one you never have to run” [7]. Here we mean that test design is not just an activity that occurs at the end of the lifecycle, but rather its real value is during development, while the design is being formulated (e.g., moving up the Mental Stages list). Test points/failure conditions are discovered by systematically designing tests prior to implementation, however knowledge of the failure condition leads to redesign: A test-point/failure condition that cannot occur because of the way the design is done is a test that never needs to be executed. The presuppositions built into this approach, which sometimes take students some time to catch onto, is twofold. To achieve the ‘best test’ criteria: • •
The test is designed prior to completing the design The design is modified to make the test condition impossible (highly improbable) This integrated approach provides students with ample opportunities to learn and experience these benefits first hand. The testing techniques force a detailed analysis of each of the designs under test - thus a student may not grasp the design concept well at the first presentation/assignment, but the test model follows this up, requiring the student to decompose the problem while providing a structured means of assessing the quality (testability) of their design artifacts. Similarly, in analyzing their design to develop a test plan, students identify conditions (including those only implied in the problem statement) that their original design did not account for, which in turn leads to appropriate design modifications. Once the test cases are designed (part of test planning), they have the opportunity to modify the design to make the design easier to test. In this way students begin to
experience the efficiencies of good design – implementations that are easier to test, fewer test cases to run, and more ability to develop value-added test cases: ones that have a high probability to expose bugs, not just demonstrate that the code works.
TESTING & DESIGN INTEGRATION ADVANTAGES The tight integration of design and testing topics enhances several subtopics that are more difficult to learn independent of this integration. From a design perspective, system approaches to design are enhanced by the fact that students learn to plan for testing during (or even prior to) design. In this manner, they are forced to consider system views: Inputs & outputs, system & subsystem responsibilities and to review these issues. Learning of design quality is enhanced in that the analysis involved in developing and using test models provides immediate feedback to the quality of the design prior to its completion. From a testing perspective, integrating the material has the advantages of clearly demonstrating the value of the effort of early test planning and design. Thus, students can experience the value of an advanced testing ‘Mental Discipline’ (although this does not guarantee converts!) Another advantage of topic integration is in the related area of testing and quality assurance. Since many OOT patterns rely on OOA/D models [5,6,7] as a prerequisite, a more ‘traditional’ software testing course must either require design as a background (inherently instilling the ‘bad habit’ of divorcing design and test planning), or can only teach general techniques. From an overall learning perspective, the integration of the design and testing material helps prepare the students for project work, most particularly for their software engineering project course. In the follow-on project course, students are already prepared to create analysis models, designs and test plans from either model. Thus, the project course can focus more on process-oriented topics, project management, and software quality issues.
LESSONS LEARNED In its present form, this course has been run three years in succession. Since it is a tight mix of design, testing and includes the use of development tools, it is highly relavent to their educational and professional growth. In the same measure, it is also very challenging course, and failure in any of the students’ educational supports can severely restrict their ability to achieve the course objectives. These key supports include prerequisites, texts, labs/homeworks, and of course, faculty. The most prevalent difficulty in teaching the course has been the selection of suitable textbooks. In this and prior instantiations of this course we have tried several design texts, and have settled on [9] with some confidence. However, a suitable testing text has not been located. The
0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference S1G-3
Session S1G most successful (from a student perspective) has been using a highly recommended reference text [5], even though the students complain that it is difficult to read and only several chapters are used. In the third (2001) instantiation of the course, we attempted to go without a testing text, allowing the students to rely on handouts and notes. This proved to be a bad decision, and is reflected in the students’ reviews (see Figure 1). The second most prevalent difficulty has been with course prerequisites (CS2). The issue has been with students who have completed the prerequisites but are not confident in their ability to develop small to medium-sized programs independently. In all of the course offerings, this was a significant hurdle for some students in each class offering, but was a major factor in the third (2001) offering. This inability significantly slowed down the course, and led directly to an increased difficulty for the students to achieve the course objectives. The quality of their designs coming out of the prerequisite course is not at issue – rather the ability to create a working implementation. Part of the introductory design assignment involves students creating an object-oriented program from a simple specification given in [9] (which includes a partial solution). Later, the students are asked to reverse-engineer, analyze, re-design, and reimplement the same problem, applying what they have learned about design principles.
which he gave to his peers. Ultimately, this turned into permanent employment.
CONCLUSIONS Integrating software testing and design topics for undergraduates is not only possible, but provides many cognitive and theoretical benefits for students. It helps prepare them better for project course experiences, and for successful starts in careers as software development professionals. In particular, integrating testing and design leads to at least five significant learning enhancements in topics that are more difficult to achieve in separate courses: Design Topics Enhanced Systems Approaches Design Quality Testing Topics Enhanced Testing as Mental Discipline Test Planning Testing & Quality Assurance The approach has been tried in several instances, with good results in 2 of 3 instantiations (Student reviews of 4.4/5.0 in effectiveness) The approach suffers from a weaker ability to make up for deficiencies in student programming ability (prerequisites), and is in lacking in support of a suitable undergraduate testing text.
ACKNOWLEDGMENT 5
The author wishes to acknowledge the Rational SEED program, which has generously supported this course since its inception, and continues to support this course with access to professional design tools.
4.5 4
REFERENCES 3.5 3 2.5 1999
[1]
Wasserman, A., “Towards a Discipline of Software Engineering,” IEEE Software, (13:6), November 1996.
[2]
Jones, C., “Gaps in Programming Education,” IEEE Computer, (28:4), April 1995
[3]
Martin, Richards, et al., “Proc. of the Workshop on Executive Software Issues,” (CMU/SEI-89-T R-6), Carnegie Mellon University, Pittsburgh, PA.
[4]
Voshall, R., & Frezza, S., “Capstone for Cross-Disciplinary Ethics & Design,” Proceedings of the ASEE1999 Spring Conference, NorthCentral Section. April 1999.
[5]
Binder, Robert V., “Testing Object-Oriented Systems – Models, Patterns & Tools,” Addison Wesley, 1999.
[6]
Beizer, Boris, “Software Testing Techniques, 2nd Ed.,” Int. Thompson Computer Press, 1990.
[7]
Beizer, Boris, “Software Testing and Quality Assurance,” Int. Thompson Computer Press, 1996.
[8]
Meyers, Glenford J., “The Art of Software Testing,” John Wiley& Sons, 1979.
[9]
Page-Jones, Meilir Fundamentals of Object-Oriented Design in UML, Addison Wesley, 2000.
2000 2001
FIGURE 1 STUDENT EVALUATIONS OF “OVERALL EFFECTIVENESS OF COURSE ”
One of the more interesting lessons learned has been in the professional intrest in the students; on several occasions, students who completed the course reported offers for permanent positions based on what they were able to explain of their background in the design, testing and UML. One of the more rewarding anecdotes involved a student in the second instantiation of the course. While in the first weeks of his summer internship, one student found a major (prerelease) bug! This student was subsequently assigned by his management to develop a 1-hour seminar on software testing
0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference S1G-4
