SQA Models: A comparative Study

SOFTWARE QUALITY ASSURANCE MODELS: A COMPARATIVE STUDY A DISSERTATION

By

Alshaimaa Adel Tantawy Abdou Teaching Assistant - Information Systems Department College of Computers and Informatics - Zagazig University.

A Thesis Submitted to College of Computers and Informatics Zagazig University In Partial Fulfillment of the Requirements for the Degree of Master of Science In Information Systems and Technology. Supervision Committee

Prof. Mohammed Abbas Shouman Decision Support Department Faculty of Computers and Informatics ‫ ــ‬Zagazig University.

Assoc. Prof. Khalid Ali Eldrandaly Information Systems Department Faculty of Computers and Informatics ‫ ــ‬Zagazig University.

INFORMATION SYSTEMS DEPARTMENT FACULTY OF COMPUTERS AND INFORMATICS ZAGAZIG UNIVERSITY – EGYPT.

2009

Certification

CERTIFICATION This dissertation is submitted to Zagazig University for the degree of Master of Science in Information Systems. I certify that this work has not been accepted in substance for any academic degree or qualification in any other university or institution and is not currently being submitted in candidature for any other degree. Any portions of this thesis for which I am indebted to other sources are mentioned and explicit references are given.

Alshaimaa Adel Tantawy Abdou

Researcher Name:

Teaching Assistant in Information Systems Department College of Computers and Informatics Zagazig University.

Date: 17 / 09 / 2009

Signature:

i

Abstract

ABSTRACT Software Quality Assurance (SQA) is a planned and systematic approach to ensure that software processes and products confirms to the established standards, processes, and procedures. The goals of SQA are to improve software quality by appropriately monitoring both software and the development process to ensure full compliance with the established standards and procedures. Quality standards are documented agreements, protocols, or rules that outline the technical specifications or criteria to be used to ensure that products, services, processes, and materials meet their intended purpose and requirements. There are several models and approaches for SQA, such as Total Quality Management Approach (TQM), Capability Maturity Model Integration (CMMI), ISO/IEC 9000-3, and the Software Process Improvement and Capability dEtermination (SPICE). However, the proper selection and implementation of these models or approaches is often a difficult, a time consuming, and a costly task for software organizations especially small and medium ones. This research provides a comparative study between eight quality models and approaches (TQM Approach, CMMI Model, ISO 9000-3, SIX-SIGMA Approach, BOOTSTRAP Methodology, TRILLIUM Model, TICKIT Scheme, and SPICE Standard) in a textual description illustrating the history, origins, principles, characteristics, representation, advantages, and disadvantages …etc. The study also, provides a proposed framework of thirty characteristics in five categories in order to present the final comparative study in a general and a simplified overview for these quality models and approaches. The resulting comparative study provides practical insight and guidance when selecting which SQA model or approach to adopt in a software-producing organization. Such guidance is needed because of the multitude, diversity and comprehensiveness of existing models and approaches.

ii

Abstract Using the expert system technology and the resulting SQA models comparative study, the dissertation provides an intelligent system for the software quality assurance models selection and implementation [SQAMSI]. The Component Object Model (COM) technology is successfully used in designing the proposed SQAMSI system to strengthen the software interoperability. The proposed system has two phases. These phases are selection phase and implementation phase. In the Selection phase, an expert system is used to identify the more suitable SQA model or approach that meets the desired individuals’ / organizations’ objectives, and requirements. In the implementation Phase, an expert system is used to play the role of a “Virtual Software Quality Editor”. The proposed system supports individuals and software companies in implementing three different software quality models (CMMI, ISO, and SPICE). By identifying the Gauging Absence of Prerequisites (GAP) between the prerequisites and what actually exists in the present environment, the SQAMSI advisory system provides assessment results and suggestions to the companies so that successful implementation of these quality models can be achieved in a fastly and cheaply manner. For the validation and verification process, the proposed system is tested and evaluated by the quality experts in the Software Engineering Competence Center (SECC), also the proposed system SQAMSI is implemented on a real case study in the E-SmartSoft Information Systems Company. KEYWORDS COM Technology, Comparison Techniques, Expert Systems, Quality Management, Quality standards, and Software Quality Assurance.

iii

Published Papers

Published Papers

[1] Shouman M., Eldrandaly K. A., and Tantawy A. A.,

"Software

Quality Assurance Models : A Comparative Study", in the proceeding of the 43rd Annual Conference On Statistics, Computer Science, and Operations Research, Cairo University, Cairo, Egypt, 22 -25 December, 2008, pp. 16-35.

[2] Shouman M., Eldrandaly K. A., and Tantawy A. A., "Software Quality Assurance Models and Expert Systems", in the proceeding of the 9th international conference on production engineering, design, and control, Alexandria, Egypt, 10 – 12 February, 2009.

iv

Acknowledgements

ACKNOWLEDGEMENTS

First and for most, I thank Allah who has helped and guided me throughout my entire life. I wish to express my sincere appreciation and thanks to my supervisors: Prof. Shouman and Assoc. Prof. Eldrandaly, for their suitable assistance, encouragement, supervision, constructive feedback, revision of the manuscript, and for sharing with me their ideas and providing me with valuable papers from which I learned great deal. I would like to express my gratitude to Eng. Manal Helmy (Quality Expert in SECC), and Eng. Mohamed Fathy (Quality Manager in ESmartSoft Company). They graciously provided relevant data and information which proved to be extremely important to the completion of this thesis. Special thanks for Prof. Sameh and Assoc. Prof. Monier for their participation in the examination committee. Last, but not Least, I could not forget the immense love and continuous support of my family, and my college friends.

v

Table of Contents

TABLES OF CONTENTS Page

Content

ABSTRACT ………………………………………………………………………………..ii PUBLISHED PAPERS …………...……………………………………………………....iv ACKNOWLEDGEMENTS ……………………………………………………………....v TABLES OF CONTENTS …………………………………………………………….....vi LIST OF ABBREVIATIONS……………………………………………………………ix LIST OF FIGURES...…..………………………………………………………………….xi LIST OF TABLES ……………………………………………………………………….xiii

CHAPTER ONE: INTRODUCTION. 1.1 Background and Motivation ……………………………………..…………….. 1 1.2 Research Objectives and Scope …………………………………….………...... 2 1.3 Organization of the Dissertation ……………………………………………….. 2

CHAPTER

TWO:

LITERATURE REVIEW BACKGROUND.

AND

TECHNICAL

2.1 Quality Concept …………………………………………………………………4 2.2 Software Quality Assurance …………………………………………………….4 2.2.1 Software quality assurance concept ………………….……………………..5 2.2.2 Goals of software quality assurance…………………………………………6 2.2.3 Components of software quality assurance …………………………………6 2.2.4 Software quality assurance plan…………. …………………………………7 2.3 Quality Systems (Standards) ……………………………………………………9 2.3.1 Importance of quality standards……………………………………............10 2.3.2 Software process assessment………………………………………............10 2.3.3 Software process improvement ………………..…………………………..11 2.4 SQA Models and Approaches………………………………………………….11 2.4.1 Developers of SQA standards………………...……………………………11 2.4.2 Benefits of using SQA standards…………………………………..............12 2.4.3 Factors of selection of SQA models ………………………………………12 2.5 Available SQA models and approaches ………………..……………………..14 2.5.1 TQM approach …………………………………………..………………...14 2.5.2 CMMI model……………………………………………………………….14 2.5.3 ISO/IEC 9000-3…………………………………………………………….14 2.5.4 Six-Sigma approach………………………………………………………..15 2.5.5 Bootstrap methodology…………………………………………………….15 2.5.6 AMI approach ……………………………………………………………..16 vi

Table of Contents 2.5.7 SynQuest methodology…………….………………………………………16 2.5.8 Trillium model……………………………………………………………...16 2.5.9 PSP framework……………………………………………………………..17 2.5.10 TickIT scheme ..…………………………………………………………..17 2.5.11 SPICE standard………….………………………………………………..17 2.6. Comparison Techniques and SQA Models: Literature Review ………………18 2.6.1 Characteristics comparison technique ………………………………..........18 2.6.2 Model mapping comparison technique…………………………….............19 2.6.3 Needs mapping comparison technique ……..……………………..............21 2.6.4 Bilateral comparison technique ………………………………………........22 2.6.5 The integration ability of the SQA models ………………………………..24 2.7. Expert Systems and Quality Management: Literature Review ……….……....26 CHAPTER THREE: SOFTWARE QUALITY MODELS AND APPROACHES: A COMPARATIVE STUDY. 3.1 Total Quality Management (TQM) Approach ….……………………………..31 3.1.1 TQM origins and history……………………………………………….…..30 3.1.2 TQM principles and characteristics………………………………………..31 3.1.3 TQM representation………………………………………………….…….32 3.1.4 TQM criticism (advantages and disadvantages)………………………..… 33 3.1.5 TQM national awards………………………………………….………….. 33 3.2. Capability Maturity Model Integration (CMMI) Model…….………………….36 3.2.1 CMMI origins and history……………………………………………….....36 3.2.2 CMMI principles and characteristics……………………………………....37 3.2.3 CMMI representation……………………………………………………....37 3.2.4 CMMI criticism (advantages and disadvantages)……………………….…38 3.3. ISO/IEC 9000-3 (ISO 9001 Standard) …………………...…………………….39 3.3.1 ISO origins and history.................................................................................39 3.3.2 ISO principles and characteristics……………………………………….....40 3.3.3 ISO representation………………………………………………………….41 3.3.4 ISO criticism (advantages and disadvantages)…………………………..…42 3.4. Six-Sigma (6σ) Approach…..………………………………………………….43 3.4.1 Six-Sigma origins and history……………………………………………...43 3.4.2 Six-Sigma principles and characteristics…………………………………..44 3.4.3 Six-Sigma representation…………………………………………………..45 3.4.4 Six-Sigma criticism (advantages and disadvantages)……………………...48 3.5. Bootstrap Methodology ..………………………………………………………49 3.5.1 Bootstrap origins and history……………………………………………....49 3.5.2 Bootstrap principles and characteristics………………………………..…..49 3.5.3 Bootstrap representation………………………………………………...….51 3.5.4 Bootstrap criticism (advantages and disadvantages)…………………….…53 3.6. Trillium Model..……………………………………………………………….54 3.6.1 Trillium origins and history………………………………………………...54 3.6.2 Trillium principles and characteristics……………………………………..55 3.6.3 Trillium representation……………………………………………………..55 3.6.4 Trillium criticism (advantages and disadvantages)……………………...…57 3.7. TickIT Scheme………………………………………………………..………..58 3.7.1 TickIT origins and history.......................................................................…..58 3.7.2 TickIT principles and characteristics…………………………………..…..58

vii

Table of Contents 3.7.3 TickIT representation …………………………………………………..….60 3.7.4 TickIT criticism (advantages and disadvantages)……………………….…60 3.8. SPICE (or ISO/IEC 15504) Standard………………………………………….61 3.8.1 SPICE origins and history ……………………………………………..…..61 3.8.2 SPICE principles and characteristics…………………………………..…..63 3.8.3 SPICE representation …………………………………………………..….64 3.8.4 SPICE criticism (advantages and disadvantages)……………………….…65 3.9. The Comparative Study of SQA Models..……………………………………..66 3.9.1. The objectives of the proposed comparative study ...….………………….66 3.9.2. The proposed comparison framework ……………….……………………67 CHAPTER FOUR: A KNOWLEDGE-BASED SYSTEM FOR SQA MODELS SELECTION AND IMPLEMENTATION. 4.1 The Proposed SQAMSI Advisory System …………………………………….78 4.2 Development of the Proposed SQAMSI Advisory System ……...…………...79 4.3 Expert System Module ………………………………………………………...80 4.3.1 Knowledge acquisition .…………………………………………………....80 4.3.2 Expert system building tool ………………………………………………81 4.3.3 Architecture of the proposed expert system ……………………………...83 4.3.3.1 Production system data……………………………………………….84 4.3.3.2 Production system rules…………………………………………..….84 4.3.3.3 Inference engine……………………………………………..……….85 4.3.3.4 Knowledge representation language…………………………………87 4.4 Database Module ……………………………………………………………....90 4.5 SQAMSI User Interface ……………………………………………………….94 CHAPTER FIVE: SYSTEM VALIDATION AND IMPLEMENTATION. 5.1 System Verification and Validation ……………….…………………………..95 5.2 Case Study ………………………………………….……………………….....96 5.3 Test Data for the Case Study …………………….………………………….....97 5.4 System Implementation ………………………….………………………….....97 5.4.1 Selection phase …………………………..………………………………...99 5.4.2 Implementation phase...………………………..…………………………105 5.5 Results and Discussion …………………………….…………………………113 CHAPTER SIX: CONCLUSIONS AND RECOMMENDATIONS. 6.1 Summary and Conclusions …………………………………………………...114 6.2 Recommendations ……………………………………………………………115 REFERENCES ...……………………………………………………….………..116 ARABIC SUMMARY ……………………………………………………………123

viii

List of Abbreviations

LIST OF ABBREVIATIONS Abbreviation

Definition

BCS

British Computer Society.

BOOTSTRAP

European software process assessment and improvement methodology.

BSI

The British Standards Institute.

CAE

Canadian Award for Excellence.

CMM

Capability Maturity Model.

CMM-SW

Capability Maturity Model for software.

CMMI

Capability Maturity Model Integration.

COM

Component Object Model Technology.

DFSS

Design for Six-Sigma.

DMAIC

Define-Measure-Analysis-Improve-Control.

DTI

The British Department of Trade and Industry.

EF/QIP/GQM

Experience Factor/ Quality Improvement Paradigm/ GoalQuestion-Metric

EQA

European Quality Award.

ES

Expert System.

ESA

European Space Agency.

ESPRIT

European Strategic Programme for Research and development in Information Technology

GAP

Gauging Absence of Prerequisites.

GQM

Goal-Question-Metric method.

IDEAL

Initiation-Diagnosis- Establishment- Action-Learning.

IEC

International Electrotechnical Commission.

IEEE

Institute of Electrical and Electronic Engineering.

ix

List of Abbreviations IS

Information System.

ISO

International Standardization Organization.

IT

Information Technology.

KPA

Key Process Area.

MBNQA

Malcolm Baldrige National Quality Award.

PDCA

Plan-Do-Check-Act cycle.

QC

Quality Control.

QFD

Quality Function Deployment.

QIP

Quality Improvement Paradigm.

QMS

Quality Management System.

SECC

Software Engineering Competence Center.

SEI

Software Engineering Institute.

SPA

Software Process Assessment.

SPC

Statistical Process Control.

SPI

Software Process Improvement.

SPICE

Software Process Improvement and Capability dEtermination.

SPIQ

Software Process Improvement for better Quality

SPQA

The United State Senate Productivity and Quality Award.

SQA

Software Quality Assurance.

SQAMSI

Software Quality Assurance Implementation Program.

TQM

Total Quality Management.

TRILLIUM

Telecommunication capability.

product

x

Models

development

Selection

and

and

support

List of Figures

LIST OF FIGURES Figure

Page

Figure 2.1: SQA components …………………………………..……………………..7 Figure 3.1: The history of CMMs ………….………………………………………..36 Figure 3.2: ISO 9001 certification process …………………………………………..41 Figure 3.3: High-level DMAIC improvement methodology ………………………...46 Figure 3.4: The combinatory metrics in six-sigma……….…………………………..48 Figure 3.5: BOOTSTRAP and the methodologies it is based on…………………….51 Figure 3.6: BOOTSTRAP attribute tree……………………………………………...52 Figure 3.7: Architecture of the Trillium model ……………………………………...56 Figure 3.8: SPICE relationships with other standards and models………………….62 Figure 3.9: SPICE framework structure……………………………………………...62 Figure 3.10: Software process improvement framework..…………………………...64 Figure 3.11: The SPICE capability levels...………………………………………….65 Figure 3.12: The progression characteristic…………………………………………73 Figure 4.1: The three-tier architecture of the proposed system ……………………...80 Figure 4.2: Rules as part of the component development process ………………….82 Figure 4.3: Visual rule studio ruleset designer ……………………………………...83 Figure 4.4: Forward chaining inference processing ………………………………....86 Figure 4.5: The structure of a Visual rule studio object …………………………......88 Figure 4.6: A fragment of the ruleset of the proposed system ……………………...88 Figure 4.7: Blackboard architecture of the SQAMSI system …………………….....89 Figure 4.8: The ActiveX DLL VB project of the proposed ES component ………....91 Figure 4.9: A fragment of the VBA code written to implement ADO ………….......93 Figure 4.10: The proposed system’s user interface ……………………………….....94 Figure 5.1: The SQAMSI welcome screen ……………………………………….......98 xi

List of Figures Figure 5.2: The SQAMSI phase selection …………………………………………….98 Figure 5.3: The SQAMSI selection phase …………………………………………….99 Figure 5.4: The customer location question...………………………………………..100 Figure 5.5: The company scope question..…………………………………………..100 Figure 5.6: The company aim question………………………………………………101 Figure 5.7: The company size question……...……………………………………….101 Figure 5.8: The assessment type question……………………………………………102 Figure 5.9: The progression type question...…………………………………………102 Figure 5.10: The improvement focus question…………………………………….....103 Figure 5.11: The improvement people question...……………………………………103 Figure 5.12: The final result of the SQAMSI selection phase...……………………...104 Figure 5.13: The SQAMSI help screen………………………………………………105 Figure 5.14: The SQAMSI implementation phase …………………………………..106 Figure 5.15: The quality model selection ……………………………………………106 Figures 5.16: Loading or creating a new account …………………………………..107 Figures 5.17: Creating the username and password ……………………………….. 107 Figure 5.18: Loading or creating an assessment project …………………………...108 Figure 5.19: Choosing the CMMI evaluation style …………………………….…..108 Figure 5.20: Choosing the CMMI maturity level …………………………………..109 Figure 5.21: A sample of CMMI assessment questions ……………………………109 Figure 5.22: The end of the assessment questions in CMMI level 2 …………………110 Figure 5.23: The assessment and evaluation report ……………………………….. 111 Figure 5.24: The SQAMSI text assessment and evaluation report …………….......112 Figure 5.25: The SQAMSI bar graph assessment report …………………….…….112 Figure 5.26: The SQAMSI pie graph assessment report …………………………..113

xii

List of Tables

LIST OF TABLES

Table

Page

Table 3.1: Sigma degree and defects per million …………………………………....45 Table 3.2: The span of each capability area in Trillium model………………………56 Table 3.3: Categorization of comparison characteristics ……………………………68 Table 3.4 (a): The comparison of SQA models and approaches [TQM, CMMI, ISO 9000-3, and Six Sigma] ……………….………………………..........76 Table 3.4 (b): The comparison of SQA models and approaches [Bootstrap, Trillium, TickIT, and SPICE] ………………………………………………….77

xiii

CHAPTER ONE INTRODUCTION

Chapter One

Introduction

Chapter One

INTRODUCTION 1.1

Background and Motivation Software Quality Assurance (SQA) is a planned and systematic approach to

ensure that software processes and products confirms to the established standards, processes, and procedures. The goals of SQA are to improve software quality by appropriately monitoring both software and the development process to ensure full compliance with the established standards and procedures. Quality standards are documented agreements, protocols, or rules that outline the technical specifications or criteria to be used to ensure that products, services, processes, and materials meet their intended purpose and requirements. They also provide a basis for measurement because they provide criterion, or basis, for comparison. Standards play a vital role in raising the level of quality, safety, reliability, maintainability, efficiency and interchangeability while also providing such benefits in an economical manner. There are several models and approaches for software quality assurance, such as capability maturity model integration (CMMI), ISO/IEC 9001, and the software process improvement and capability determination (SPICE). However, the proper selection and implementation of these models is often a difficult, a time consuming and a costly task for software organizations especially small and medium ones. For any software-producing organization in order to select the more appropriate SQA model or approach to be adopted, it is very complicated process, such that each quality model or approach has its own properties, strengths, weaknesses, and so on. Also, the organization has its own customers, resources, management; objectives … etc. all of these factors affect the selection process. This dissertation studies the available SQA models and approaches in details, providing the origins, principles, characteristics, representation, advantages and

1

Chapter One

Introduction

disadvantages for the most popular eight models (TQM, CMMI, ISO, SIX-SIGMA, BOOTSTRAP, TRILLIUM, TICKIT, and SPICE) via a comparative study using a proposed comparison framework of thirty characteristics in five categories. The goal of the comparative study is to provide practical insight and guidance when selecting which SQA model or approach to adopt in a software-producing organization. Such guidance is needed because of the multitude, diversity and comprehensiveness of existing models and approaches. Expert system technology is beginning to play an important role and will become more common in the future in quality management. Using the expert system technology and the resulting comparative study, the dissertation provides an intelligent system for software quality assurance models selection and implementation [SQAMSI]. 1.2 Research Objectives and Scope Achieving a high level of product or service quality is now the objective of most software organizations. There are several models for software quality assurance, however, the proper selection and implementation of these models is often a difficult and a costly task for software organizations especially small and medium ones. This dissertation will conduct a comparative study between the most popular eight quality models and also will develop a knowledge-based system for SQA models selection and implementation. The following objectives constitute the scope of the study: 1. Conducting a comparative study between the eight selected SQA models and approaches. 2. Developing a knowledge-based system for SQA models selection and implementation. 3. Applying the proposed system to a real case study in order to examine its validity. 1.3

Organization of the Dissertation This dissertation consists of six chapters as follows:

2

Chapter One

Introduction

Chapter One: Introduction This chapter describes the importance of the research topic, highlights the motivitations and objectives of this study and summarizes the contents and organization of the dissertation.

Chapter Two: Literature Review and Technical Background. This chapter discusses current state-of-the-art research on software quality assurance, quality standards and approaches, quality management, the different comparison techniques, expert systems, systems integration techniques and the problem of the software quality assurance models selection and implementation.

Chapter Three: Software Quality Models and Approaches: A Comparative Study. This chapter discusses in details eight SQA models and approaches, illustrating their characteristics, benefits, and weaknesses, in order to provide a complete comparative study between them according to a proposed comparison framework contains 30 characteristics categorized in five categories.

Chapter Four: A Knowledge-Based System for SQA Models Selection and Implementation. This chapter describes the development of a proposed knowledge-based system for SQA models selection and implementation.

Chapter Five: System Validation and Implementation. This chapter presents the verification and validation process of the proposed system on a real case study. It discuses the system implementation steps and focuses on the evaluation of the different features of the proposed system.

Chapter Six: Conclusions and Recommendations. This chapter presents a summary of the research findings and its significance. Suggestions for improvement and future work are also discussed.

3

CHAPTER TWO LITERATURE REVIEW AND TECHNICAL BACKGROUND

Chapter Two

Literature Review and Technical Background

Chapter Two Literature Review and Technical Background This chapter discusses the current state-of-the-art of software quality, software quality assurance, comparison techniques, and expert systems applications in quality management.

2.1 Quality Concept Software quality is becoming vital issue in the filed of information technology for developing and maintaining software applications. On the other hand, since quality in many respects a subjective issue, its meaning and achievements remains unclear. Quality is something, which is complex and varies form one industry to another. Quality needs to be model on the basis of customer need, context of the market, industrial goals, system requirements etc. Quality has many layers, and no universal definition will apply in every case. Even the experts could not agree on single definition for the quality [1]. An organization's focus on the strategic importance of software quality depends on whether they are producers or users of software. Software developers see themselves as either the producers of a product that will be sold in the marketplace or contractors developing a product for a specific client. Software users see software as a tool to be used to support them in the way they do business in their specific sector. Both business views are very different but in both cases quality is an underlying consideration to helping them achieve their business objectives [2]. Quality must be defined and measured if improvement is to be achieved. Yet, a major problem in quality engineering and management is that the term Quality is ambiguous, such that it is commonly misunderstood [3]. There are many popular definitions for quality concept. Manufacturing quality concepts initiated by Deming and continued by Juran and Gryna such as conformance for requirements and fitness for customer use, have been applied to software products [4].

4

Chapter Two


Pressman and Humphrey [4] showed quality as a multi-dimensional concept perceived differently in different domains. Fournier has indicated that, ‘‘Quality very often signifies different things to different people in the context of a software system’’. Software Quality is the extent to which an industry-defined set of desirable feature is incorporated into a product so as to enhance its lifetime performance. This definition focuses on the existence of a product in the first place and that its quality has a time dimension. It also focuses on features which will enhance the product. Finally it requires the features to be incorporated from the beginning-by way of user requirements or similar specification- and not bolted on as an afterthought [2].

2.2 Software Quality Assurance 2.2.1 Software quality assurance concept Quality Assurance is defined as: "a planned and systematic pattern of all actions necessary to provide adequate confidence that an item or product conforms to established technical requirements" [5]. Also, it is the function responsible for managing quality. The word “Assurance” means that if the processes are followed, management can be assured of product quality. Quality assurance is a catalytic function that should encourage quality attitudes and discipline on the part of management and workers. Successful quality assurance managers know how to make people quality conscious and to make them recognize the benefits of quality to themselves and to the organization [6]. Software Quality Assurance (SQA) is the set of systematic activities providing evidence of the ability of the software process to produce a software product that is fit to use [5]. The SQA group embraces a framework as a guideline to judge the quality of a software system. This model identifies 14 quality factors in three stages of the development life cycle; design, performance, and adaptation. Although it is difficult, if not impossible, to develop a piece of software that adheres to all of these 14 factors;

correctness, maintainability, verifiability, efficiency,

integrity, reliability, usability, testability, expandability, flexibility, portability, reusability, interoperability, and intra-operability, the model provides a good frame of reference to understand software quality [4].

5

Chapter Two


Finally, SQA can be defined as the planned and systematic set of activities that ensure that software process and products conform to requirements, standards, and procedures [7]:

 "Processes" include all activities involved in designing, coding, testing and maintaining;

 "Products" include software, associated data, documentation, and all supporting and reporting paperwork. 2.2.2 Goals of software quality assurance The software development is a complex process full risk. There are Technical Risks such as software will not perform as intended or be too hard to operate, modify, and/or maintain; there are Programmatic Risks such as the project will overrun cost or schedule. The goals of SQA are to reduce these risks by [7]: 

Appropriately monitoring the software and the development process.



Ensuring full compliance with standards and procedures for software and process.



Ensuring that inadequacies in product, process, or standards are brought to management's attention so that they can be fixed. SQA is not responsible for producing quality products or for making quality

plans. They are responsible for auditing the quality actions and for alerting management to any deviations [7]. 2.2.3 Components of software quality assurance Most SQA activities can be categorized into software testing, that is, verification and validation, software configuration management, and quality control. But the success of a SQA program also depends on a coherent collection of standards, practices, conventions, and specifications, as shown in Figure 2.1 [6]. SQA activities include product evaluation and process monitoring, which ensure the product and the process used in development are correctly carried out and standards are followed. SQA audit, another SQA activity, is a key technique used to perform product evaluation and process monitoring [7].

6

Chapter Two


Software Quality Assurance Software Testing

Quality Control

Standards

Procedures Software Configuration Management

Conventions

Specifications

Figure 2.1: SQA components [Adapted from 6]. 2.2.4 Software quality assurance plan (SQAP) The SQAP is an outline of quality measures to ensure quality levels within a software development effort. The plan is used as a baseline to compare the actual levels of quality during development with the planned levels of quality. If the levels of quality are not within the planned quality levels, management will respond appropriately as documented within the plan [6]. The SQAP provides the framework and guidelines for development of understandable and maintainable code. These ingredients help ensure the quality sought in a software project. An SQAP also provides the procedures for ensuring that quality software will be produced or maintained in-house or under contract. These procedures affect planning, designing, writing, testing, documenting, storing, and maintaining computer software. It should be organized in this way because the plan ensures the quality of the software rather than describing specific procedures for developing and maintaining the software [6]. The SQAP may be a separate document or may be included in another planning document, such as the software development plan. An SQAP shall be prepared at the beginning of the development effort. This plan shall describe [14]: 

SQA activities that will be performed,



Individuals or organizations responsible for performing these activities and their reporting responsibilities,

7

Chapter Two




Techniques, tools. and facilities used in SQA activities, and



Relationship among SQA and development and activities, schedules, and milestones.

The objectives of software quality are typically achieved by following a SQAP that states the methods the project will employ to ensure the documents or products produced and reviewed at each milestone are of high quality. Such an explicit approach ensures that all steps have been taken to achieve software quality and provides management with documentation of those actions. The plan states the criteria by which quality activities can be monitored rather than setting impossible goals, such as no software defects or 100 percent reliable software. Steps to develop and implement a SQAP are; document the plan, obtain management acceptance, obtain development acceptance, plan for implementation of the SQAP, and execute the SQAP [6]. The IEEE Standard for SQAPs applies to the development of a SQAP. The existence of this standard should not be construed to prohibit additional content in a SQAP. An assessment should be made for the specific software item to assure adequacy of coverage. Where this standard is invoked for an organization or project engaged in producing several software items, the applicability of the standard should be specified for each of the software items. The standard purpose is to provide uniform, minimum acceptable requirements for preparation and content of SQAPs. The SQAP shall include the sections listed below to be in format conformance with this standard. The sections shall be ordered in the described sequence. The first page of the SQAP shall include the date of issue, the status of the document, and the identification of the issuing organization. The SQAP sections shall include the following [104]: 1. Purpose. 2. Reference Documents. 3. Management. 4. Documentation. 5. Standards, Practices, Conventions, and Metrics.

8

Chapter Two


6. Software Reviews. 7. Test. 8. Problem Reporting and Corrective Action. 9. Tools, Techniques, and Methodologies. 10. Code Control 11. Media Control. 12. Supplier Control. 13. Records Collection, Maintenance, and Retention 14. Training. 15. Risk Management. 16. Glossary. 17. SQAP change procedure and history. Additional sections may be added as required. Some of the material may appear in other documents. If so, then reference to these documents shall be made in the body of the SQAP. For those projects in which all required SQAP material is included in the project plan, or other governing document, the governing document shall map the material to the list of SQAP sections. The SQAP shall be approved by the manager of each unit of the organization having responsibilities defined within this SQAP or their designated representatives [104].

2.3 Quality Systems (Standards) The concepts and philosophies of quality management have received a great deal of attention over the years. Although popularized by Japanese, many organizations in different countries have initiated quality improvement programs. Based on writings and teachings of such quality gurus as Shewhart, Deming, Juran, Ishikawa, and Crosby, the core values of these quality models have a central theme that includes a focus on the customer, incremental or continuous improvement, problem detection and correction, measurement, and the notion that prevention is less expensive than inspection. Although guilds are the first organizations to ensure quality standards, there are a number of different organizations and approaches for defining and implementing

9

Chapter Two


quality standard in organizations. Standards are documented agreements, protocols, or rules that outline the technical specifications or criteria to be used to ensure that products, services, processes, and materials meet their intended purpose. Standards also provide a basis for measurement because they provide criterion, or basis, for comparison [8].

2.3.1 Importance of quality standards Standards play a vital role in raising the level of quality, safety, reliability, maintainability, efficiency and interchangeability while also providing such benefits in an economical manner. Standards, as developed by various consortiums of experts from the industry, government and academia, provide a good source of reference for process improvements. They specifically address the issues of various vertical industries and hence are very unique in nature. Organizations adopt the best-suited option among the whole gamut of standards available [9]. International standards are established for many technologies and industries. Countries that do business with each other need to have an agreed upon set of standards to make the process of trade more logical and because a lake of standardization can create trade barriers. For example, credit cards adhere to a standard size and thickness so that they can be used worldwide [8]. Quality standards are necessary tools for any company to build up its quality management system. They are documents that define the contractual, functional, and technical requirements for all quality activities that will ensure that a product, process, service, and system are fit for its intended purpose. Undoubtedly, they have given the industrial world many opportunities for technology transfer, intercommunication, product interchangeability, and customer satisfaction [10].

2.3.2 Software process assessment Software Process Assessment (SPA) is a systematic procedure to investigate the existence, adequacy, and performance of an implemented process system against a model, standard, or benchmark [11]. Often, an assessment method accompanies the process improvement framework to facilitate the comparison of organizational

10

Chapter Two


practices with the practices as prescribed by the framework. Generally, three types of assessment are distinguished; self-assessment, second-party assessment, and thirdparty assessment [12]. From the viewpoint of reference systems there are four types of assessment methods; model-based, standard-based, benchmark-based, and integrated assessment which is a kind of composite model-based and benchmark-based methods [13].

2.3.3 Software process improvement Software Process Improvement (SPI) is a systematic procedure to improve the performance of an existing process system by changing the current processes or updating new processes in order to correct or avoid problems identified in the old process system by means of a process assessment [11]. SPI actions are taken to change an organization’s software process so that they meet the organization’s business needs and help it to achieve its business goals more effectively. SPI is the goal of SPA, acting on issues found in an assessment and enhancing the effectiveness of processes in the process system. There are various philosophies underpinning SPI. Key categories of SPI philosophy are; goal-oriented, benchmark-based, and continuous process improvement [13]. There are two basic SPI methods – assessment-based and benchmark-based process improvement. The former improves a process system from a given level in a defined scale to a next higher level; the latter provides improvement strategies by identifying gaps between an organization’s process system and a set of established benchmarks. In addition, a combined approach may be adopted [13].

2.4 SQA Models and Approaches 2.4.1 Developers of SQA standards Many SQA standards have been written by both defense and non-defense organizations in the last fifteen years. The defense-oriented standards are written to provide direction for Department of Defense (DOD) contractors regarding SQA and to provide DOD personnel a way of evaluating contractor compliance with DOD

11

Chapter Two


requirements. The non-defense standards are written by professional society and industry committees. These standards are aimed to establishing an SQA level of practice that developers would either comply with on a voluntary basis or would be compelled to follow because of customer demands [14]. The most prominent developers of SQA standards are [15]:  Military and Government Agencies: US Department of Defense (DoD), National Aeronautics and Space Administration (NASA), and Federal Aviation Authority (FAA)  National Standards Bodies: Canadian Standards Association (CSA), and American National Standards Institute (ANSI).  International Standards Bodies: International Standards Organization (ISO), and International Electro-technical Commission (IEC).  Professional and Technical Associations: Institute of Electrical and Electronic Engineers (IEEE), Software Engineering Standards Committee (SESC), and Electronics Industries Associations (EIA).  Un-chartered by widely recognized bodies: Internet Engineering Task Force (IETF), WWW Consortium (W3C), and International Council on Systems Engineering (INCOSE).

2.4.2 Benefits of using SQA standards [16] 

The ability to apply methodologies and procedures of the highest professional level.



Better mutual understanding and coordination among development teams but especially between development and maintenance teams.



Greater cooperation between the software developer and external participants in the project.



Better understanding and cooperation between suppliers and customers, based on the adoption of standards as part of the contract.

2.4.3 Factors of SQA models selection One problem faced by a software development organization is the selection of the particular standard to use from among the many SQA standards that have been

12

Chapter Two


written and approved. This selection may be dictated by the customer, but then the customer faces the same problem. If an organization has several types of customers (some Defense-oriented and other commercial), that organization may become concerned that its SQA activities, and even to some extent its software development process, must be different for each customer's imposed quality standard [14]. According to Herndon and Salars [17] the selection of any formal SQA model should consider these factors:  All of the process improvement models, tools and techniques require an up-front investment of both time and money.  Take time to research the best process improvement approach to achieve the optimum investment in support of the business goals.  All up-front investments in any of these models, tools and techniques should be planned with achievable and realistic performance goals. As with any investment, cash flow will be impacted for 4-8 months, depending upon the size of investment.  Selection of a specific SQA model, tool or technique does not exclude other options. For example, adopting both Six-Sigma and CMMI models is typical in larger organizations. However, small settings may not have the staff and cash resources to consider multiple options [18].  Implementing process improvements required to achieving the performance goals is via a cultural change. Also, according to Hari and Sanne [9] some of the driving factors for adopting a SQA standard could be; internal need for establishing and continually improving the processes, contractual obligations, and pressure from competitors. The Preparation of any standard is a long, iterative process involving compromise and condensation of details. In many cases, standards describe the lowest common denominator techniques and methods for basic quality assurance activities rather than reflecting the current technology. Standards give little or no guidance on how the SQA activities should be performed; rather, they state what should be done [14].

13

Chapter Two


2.5 Available SQA Models and Approaches There are several models and approaches for SQA, the following paragraphs describe briefly the available SQA models and approaches: 2.5.1 Total quality management (TQM) approach TQM is an approach of management that is originated in Japan in the 1950's. TQM is originally coined in 1985 by the Naval air systems. Various specific methods for implementing the TQM philosophy are found in the works of Crosby, Deming, Feigenbaum, Ishikawa, Juran, and Gryna. TQM emphasizes using multi-functional teams to solve problems [1]. The teams are trained to use basic statistical tools to collect and analyze data. Worldwide, there are several quality awards, such as; the Malcolm Baldrige national quality award (MBNQA) in U.S.A, the European quality award (EQA) in Europe, the Deming prize in Japan, and the Canadian award for excellence (CAE) in Canada [19]. 2.5.2 Capability maturity model® integration (CMMI) model CMMI is developed by the Software Engineering Institute (SEI) of CarnegieMellon University under the sponsorship of the US DoD. CMMI is a process improvement approach that provides organizations with the essential elements of effective processes. It can be used to guide process improvement across a project, a division, or an entire organization. There are multiple CMMI models available, as generated from the CMMI Framework. Also, there are two types of representations, staged and continuous. The five maturity levels are Initial, Managed, Defined, Quantitatively Managed, and Optimizing [20]. 2.5.3 ISO/IEC 9000-3 (ISO 9001 standard) ISO 9001 is a generic international standard, adopted on a country-by-country basis, and written for use by the widest possible audience. As a result, the standard provides requirements (What needs to be done) and does not issue specific prescriptive solutions (How to do it). Being so broadly focused, the ISO 9001 standard does not offer details about its application to specific domains of expertise. To assist in the application of the standard for specific domains, a series of guidelines are available; e.g., ISO 9000-3 is a guideline for the software development industry [21]. ISO 9000-3 is a guidebook on applying the requirements of ISO 9001

14

Chapter Two


to software development and maintenance organizations, and not a software quality system standard. It describes a number of elements that should be included in an organization’s quality system for software [22]. 2.5.4 Six-Sigma approach Six-Sigma approach is originated by Motorola in the mid-1980s. The Greek symbol σ represents the standard deviation to measure variability from the mean or the average. In organizations, variation is often the cause of defects or out-of-control processes and translates into products or services that do not meet customer needs or expectations [8]. Six-Sigma model is a fact-based, data-driven philosophy of quality improvement that values defect prevention over defect detection. It applies anywhere variation and waste exist, and every employee should be involved. In simple terms, six sigma quality performance means no more than 3.4 defects per million opportunities (DPMO) [23]. Six-Sigma process improvement methodology included five logically linked phases (DMAIC): Define → Measure → Analyze → Improve → Control. Six-Sigma methodology can also be used to create a brand new business process from ground up using DFSS, One example sequence of DFSS is Define → Measure → Analyze → Design →Verify [24]. 2.5.5 Bootstrap methodology Bootstrap is designed to help European commercial applications, and it concentrated on small and medium-size software systems and put more emphasis on rating individual software practices (especially Technical practices) rather than global technical and management processes. Bootstrap measures the process capability according to six capability levels; incomplete, performed, managed, established, predictable, and optimizing process. Bootstrap method widens the assessment scope to the organizational level. Furthermore, assessment is more finegrained than in CMM the capability levels are refined into quarters. Bootstrap model is SPICE compliant, and it is more technology oriented than other models, process for deriving an action plan from the results is also provided, and very efficient and effective means not only to assess a current status of software process quality and to initiate appropriate improvement actions [25].

15

Chapter Two


2.5.6 Application of metrics in industry (AMI) approach It has been developed in Europe as a co-operative project involving nine European centers of excellence and is sponsored by the European commission through the ESPRIT programme. It is an instance of applying two processes techniques; the Goal-Question-Metric (GQM) paradigm and process assessment. AMI is a quantitative approach to SPI, its aim is to provide a practical and validated approach to installing and using quantitative approaches to control and improve the software process. The AMI method implements four distinct activities: Assess, Analyze, Metricate, and Improve [26]. 2.5.7 SynQuest model SynQuest is a family of self assessment methods for software processes. It regards approximately 40 features of the software process and asks for fulfillment of the most important IS0 9000 criteria in each of those features. It is a PC-based software-tool has been sold approximately 100 times since 1995 especially in the German speaking area of Europe. It has been used successfully as a starting point for process improvement experiments in the IT-programme of the EU. Larger institutions have used SynQuest for guided assessments performed parallel in different groups of the company – an efficient way to compare work processes within an organization. SynQuest can be used by any size of organization, can run assisted or unassisted, and requires only a few hours to understand and use it. Results are as reliable as possible in any self assessment environment. Its advantage is that self assessments can be repeated easily - without extra costs [27]. 2.5.8 Trillium model Since April 1991, a new assessment model, subsequently named "Trillium", has been developed in partnership with Bell Canada, Northern Telecom and BellNorthern Research. Trillium aims to benchmark an organization's product development and support capability in a commercial context. It is based on SEI-CMM V1.1, also covers ISO, Bellcore, MBNQA, IEEE and IEC standards. The Trillium scale spans levels 1 through 5. The levels can be characterized as: unstructured, repeatable and project oriented, defined and process oriented, managed and integrated, fully integrated [28]. The goal of the Trillium model is to provide a means

16

Chapter Two


to initiate and guide a continuous improvement program [29]. It is used in a variety of ways; to benchmark an organization’s product development and support process capability against best practices in the industry, in self-assessment mode, to help identify opportunities for improvement within a product development organization, and in pre-contractual negotiations; to assist in selecting a supplier [30]. 2.5.9 Personal software process (PSP) framework It has been developed by SEI to address the need for process improvement in small organizations and small project teams. The PSP is a self-improvement process designed to help individual engineers to control, manage, and improve the way they work. It is a structured framework of forms, guidelines, and procedures for developing software. Properly used the PSP provides the individual engineer with historical data that is needed to better make and meet commitments making the routine elements of software engineering more predictable and more efficient. The PSP has a maturity framework much like that of the CMM model [26]. 2.5.10 TickIT scheme TickIT initiative in 1991 in London is a certification scheme developed to apply ISO 9001, but with the advantage of having been "tuned" to deal with the special requirements of software development. The main principles and objectives of TickIT are, the interpretation of ISO 9000 for the sector, the need to ensure continuing conformity for certified suppliers, the necessity to perform assessments with experienced and skilled assessors, as witnessed by their ability to satisfy examiners, and the benefit of accredited training and examination for entrants on the assessor register. However, since the scope of TickIT is identical to ISO 9001, there is virtually no content relating to process improvement, except in the context of changes resulting from corrective and preventive actions [21]. 2.5.11

Software process improvement and capability determination (SPICE or ISO/IEC 15504) standard.

It is an international standard for SPA. It is mainly used in Europe and Australia by the automotive industry. Its goal is to provide assessment results that are repeatable, objective, and comparable. SPICE framework would be suitable for use in

17

Chapter Two


the primary contexts of process assessment, process improvement and capability determination. The assessment framework depends upon an assessment architecture that defines the practices and processes that should be implemented. The six capability levels defined within SPICE are; not-performed, performed-informally, planned and tracked, well-defined, quantitatively-controlled, and continuouslyimproving. The assessment output is thus seen as a profile, where the assessment of capability for each level is plotted for each of the process areas and processes. The assessment scope is defined at the process level, rather than at the organizational unit level. SPICE suite seems to have combined the best features of CMM and ISO 9000 standards in addition to the notion of assessment [31].

2.6

Comparison Techniques Literature Review

and

SQA

Models:

The goal of comparing SQA models and approaches is to provide insight into their similarities and differences. According to [32] there are four classes of comparison techniques that can be used on SQA models; characteristics, model mapping, needs mapping, and bilateral comparison. Each of these will be discussed. It should be clear that it would be neither beneficial nor possible to define disjoint comparison methods. Thus, some overlap is inevitable. Also, differences in level of detail and point of view should be expected as they satisfy different requirements. 2.6.1 Characteristics comparison technique This technique compares the various SQA models in a table by defining an extensive list of relevant characteristics. Each model is then described in terms of these characteristics and the results are presented in a tabular format. This type of comparison is well suited for a general overview of the models and it can be used as a basis for other comparison methods. A lot of information can be inferred from a table listing the characteristics of several models. However, the comparison is on a high level and details must be collected elsewhere. For this purpose one of the other comparison methods can be used [32]. Here are the previous related works of this comparison technique,

18

Chapter Two


Haase [27] compared some of the assessment concepts in six SQA models: ISO 9000, SPICE, CMM, Bootstrap, SynQuest and EQA. The comparison stressed practical experience and the cost-benefit ratio. The comparison characteristics were size, method, aim, assessor (who), documents, and costs. Sørumgård [33] compared CMM, ISO 9001 and the EF/QIP/GQM approach in his Ph. D. thesis. He compared the models in pairs using both a textual description and a tabular list of characteristics. Sheard [34] compared between CMM, SE-CMM, IPD-CMM, SDCE, ISO 9000, software life cycle standards and Trillium according to their characteristics; scope, purpose, length (pages), major focus, and notes. Cattaneo et al. [35] discussed software process assessment and software process improvement and findings from evaluating the CMM assessments of three different companies and their problems in deriving a systematic improvement approach from the assessment. Moreover, they gave some guidelines how improvement steps can be derived from the results. In their research, they compared between CMM, ISO 9000, and MBNQA according to applicability, standard, structure, and coherence. Halvorsen and Conradi [32] proposed a software process improvement framework taxonomy. Their taxonomy consists of 25 relevant characteristics, which can be used to point out framework similarities and differences on a high level. An example of how the taxonomy can be applied to six common SPI frameworks TQM, CMM, ISO 9000, SPICE, EF/QIP/GQM, SPIQ is provided. 2.6.2. Model mapping comparison technique Much of the recent comparison work has dealt with model mapping. Each model, at least the more structured ones, consists of a more or less defined set of statements or requirements dealing with the content and focus of the model. Model mapping is the process of creating a map from statements or concepts of one model to those of another. There are two distinct ways to do this [32]:

19

Chapter Two




Mapping between existing models.



Mapping existing models into basis model established specially for the purpose of comparing. This provides a common point of reference from which the models can be evaluated, compared and contrasted. In the characteristics approach the goal is to describe key attributes of each

SQA model. However, the purpose of mapping is to identify overlaps and correlations between models and create a map of these statements or correlations. There can exist strong, weak or no correlation as suggested by Tingey [36]. Model mapping is especially useful when an organization employs two or more different SQA models, as corresponding statements can be identified and redundancy reduced. The extra effort needed to employ more than one model can therefore be minimized. Model mapping is definitely a more low-level and detailed comparison method than characteristics. Because mapping goes into the specifics of each model, it is not very useful for a general overview. However, mapping into a basis model and supplementing with a quantitative analysis can indicate overall focus and content. Model mapping requires that some simplifying assumptions are made [32]. The results are necessarily biased by these assumptions as suggested by Tingey [36]. Here are the previous related works of this comparison technique, Tingey [36] had dedicated a whole book to the comparison of CMM, ISO 9000 and MBNQA. His book presented an in-depth comparison of the three approaches, including a complete mapping between statements and a quantitative analysis of framework focus. His extensive analysis uncovered the relative strengths of these models. Wang et al. [37] provided a quantitative evaluation of SPICE, CMM, ISO 9000 and Bootstrap. The configuration of the sample spaces and the partially overlapped fundamental process activities defined in the four models were quantitatively analyzed and contrasted. The analysis of features, orientation, profile

20

Chapter Two


and correlation provided showed objectively the general strengths and weaknesses of these models in 3 subsystems, 12 categories and 55 processes. It was interesting to note that, SPICE was the most organization oriented model; CMM was the most management oriented model; Bootstrap was the most technical oriented model.

2.6.3 Needs mapping comparison technique Needs mapping is not a direct comparison between models. Instead, it considers organizational and environmental needs that must be taken into account when selecting which SQA model to adopt. The requirements imposed by such needs are often highly demanding and can limit the choice of model severely. Nonetheless, they are of utmost importance and must be considered carefully. Here are some examples [32]:  Certification requirements, for example to ISO 9001, often imposed on a subcontractor.  Top-level management requires that the chosen SQA approach should be incorporated in a TQM strategy.  Financial limitations. There certainly exist other examples as well, and they can vary substantially from organization to organization, or depend on the business environment. Furthermore, the needs may vary over time as the organization or environment evolves [32]. Here are the previous related works of this comparison technique, Van der Pijl et al. [38] compared ISO 9000 and the CMM. In a project assessing the applicability of ISO 9000 and CMM in IS development environments in a large bank, a set of criteria for assessing the usefulness of guidelines for quality systems are formulated. It was concluded that both standards are useful but there is a growing need for more specific standards. Kuilboer and Ashrafi [39] attempted to obtain a broader insight into the SPI practices (ISO, CMM, and MBNQA), they conducted a survey targeted at software developers in New England. They collected 67 responses and used descriptive

21

Chapter Two


statistics to analyze the survey results. In addition, they examined the impact of SPI methodologies on quality factors and compare the impact to the importance of quality factors for software developers. The Spearman correlation coefficient was used to determine the degree of correlation between the two. Williams and Kennedy [40] discussed the requirement engineering process. They compared between SPICE, CMM, ISO 9000 and Bootstrap according to in which degree they can support requirements management and document management. Ashrafi [4] investigated the impact of ISO 9000 standards and CMM key process areas on software quality, first by theoretical comparison and then with empirical data. Her findings revealed that each methodology has had a different level of impact on software quality factors. These findings could help softwaredevelopment organizations select the methodology or combination that best meets their quality requirement.

2.6.4 Bilateral comparison technique In a bilateral comparison two models are compared textually, for example the CMM and ISO 9001 [41]. The difference between this comparison method and the two previously described is its textual nature. A bilateral comparison is often a summary or explanation of findings from other the comparison methods. The bilateral comparison can take on the point of view of one model and describe the other in terms of that. This is convenient for people with detailed knowledge of one model, as they can easily get insight into other using familiar terms. The amount of detail included in a bilateral comparison can vary widely, depending on the purpose for which it is written. Frequently the level of detail is somewhere in between that of characteristics and the mapping approach [32]. Here are the previous related works of this comparison technique, Bamford and Deibler [42] compared ISO 9001 with the SEI-CMM. From their comparison, probably, while an advanced maturity level indicates that certain

22

Chapter Two


key engineering practices were implemented, IS0 9001 requires business practices that are not addressed in the SEI questionnaire. Because of its focus on software engineering, the CMM model offers significant additional guidance, particularly in the areas of technology, process definition, and metrics. Paulk [43] compared the CMM and ISO 9001, showing their differences and similarities. The results of the analysis indicated that, although an ISO 9001compliant organization would not necessarily satisfy all of level 2 key process areas; it would satisfy most of level 2 goals and many of level 3 goals. Because there are practices in CMM that are not addressed in ISO 9000, it is possible for level 1 organization to receive ISO 9001 registration; similarly, there are areas addressed by ISO 9001 that are not addressed in CMM. level 3 organization would have little difficulty in obtaining ISO 9001 certification and level 2 organization would have significant advantages in obtaining certification. Paulk [41] examined how the CMM and ISO 9001 relate. He had essentially mapped clauses of ISO 9001 to CMM key practices. The mapping was based on an analysis of ISO 9001, ISO 9000-3, TickIT, and the TickIT training materials. The biggest similarity was their bottom line; “Say what you do; do what you say.” The differences between the two documents was; The explicit emphasis of CMM on continuous process improvement, ISO 9001 addresses only the minimum criteria for an acceptable quality system, and the CMM focuses strictly on software, while ISO 9001 has a much broader scope that encompasses hardware, software, processed materials, and services. Oskarsson and Glass [44] compared between ISO 9001 and CMM. They concluded that the first criterion for choosing between ISO 9001 and CMM is of course the need. If you need CMM, for example to be able to do business with US DoD, go for CMM; if, on the other hand, you need an ISO 9001 certificate to do international business, go for that. However, if both are needed, or if there is not a specific need for a documented CMM level, they recommended that a software organization first goes for certification to ISO 9001 according to TickIT, and thereafter uses CMM to improve further.

23

Chapter Two


Saiedian and Mcclanahan [45] compared between CMM and ISO 9000. They found that both of the two frameworks share a common concern regarding software quality and process management. Since it was not clear which of these two frameworks is most effective in achieving their shared objectives, it is valuable and timely to provide an objective overview of both models and to compare and contrast their features for quality software development. Messnarz [46] compared the architecture of Bootstrap V2.3 with SPICE V1, and discussed the new SPICE compliant Bootstrap V3 process model. He also discussed an approach which applies the steps discussed in the SPICE Process Improvement Guide (PIG) and how this relates to the standard Bootstrap action planning guidelines. CEO Issue Sheet [47] concluded that although all three are quality measurement systems, MBNQA for performance excellence, ISO 9001:2000 registration, and six sigma, each offer a different emphasis in helping organizations improve performance and increase customer satisfaction. Bishop [48] compared between the ISO 9000 quality standard and TQM. He said that, “both are necessary for any organization to become world class. But ISO 9000 is far superior because it offers a set of guidelines for quality management and can stand alone, while TQM can not”.

2.6.5 The integration ability of the SQA models There are also some other previous studies which discuss the integration ability of the different SQA models as illustrated in the following literature, Vasiljevic and Skoog [49] produced an SPI model for small organizations (SPISO) that can be used by small software organizations to evaluate their own capacity of delivering high quality or maturity. Purchaser can also use the model to evaluate small software supplying organizations. The model was based on parts from established SQA models, such as; SW-CMM, CMMI and ISO 9001:2000. Siviy et al. [18] discussed the concept of integrating SQA models with each others and focused on the joint use of two models CMMI and six-sigma. While some

24

Chapter Two


models can be mapped where one model subsumes the other, CMMI and six-sigma cannot because they are different types of models. Their joint deployment is synergistic. The potential value that was added is the accelerated achievement of performance goals, accelerated achievement of CMMI adoption (as a “meta goal” toward performance), stronger foundational measurement and analysis skills to enable better quantification of results, and all of the corresponding culture change that goes along with these improvements. Yoo et al. [50] provided a unified model for the implementation of both ISO 9001:2000 and CMMI by ISO-certified organizations. ISO 9001 is a standard for QMSs and CMMI is a model for process improvement. If an ISO-certified organization wishes to improve its processes continuously, implementing CMMI would be a good choice, as it provides more detailed practices for process improvement than the ISO standards. Albeanu et al. [24] described the usage of the six-sigma methodology for SQA and how a mixed six-sigma and CMMI could be applied to increase the capability and maturity level of the software department. The experience was reported for some small and medium-sized software projects. Dey [51] discussed how to complement ISO 9001:2000 with six-sigma. ISO 9001:2000 introduces a strong focus on measurement, analysis and improvement. He discussed how to meet the new requirements and drive significant business performance improvement with a complementary six-sigma program. From studying the previous literature review, it is shown that the previous related works discuss only limited number of SQA models or approaches and also compare between them according to limited issues in special cases or situations. There is no general study with public aims in this filed. So, this research aims to produce a general comparative study between the available SQA models and approaches in order to help the software organizations select the more suitable model or approach to adopt.

25

Chapter Two 2.7


Expert Systems and Literature Review

Quality

Management:

Expert systems are fast becoming the leading edge of artificial intelligence technology because of the need for such systems in commercial and scientific enterprises and also because artificial intelligence technology has evolved to the point where expert systems development has become well understood and feasible in many domains. An expert system is a computer program that embodies the expertise of one or more experts in some domain and applies this knowledge to make useful inferences for the user of the system [52]. Expert system technology is beginning to play an important role and becomes more common in the field of quality management. Reviewing the literature shows that several authors from 1991 until 2009 have provided interesting studies in this domain. The following paragraphs summarize some of these studies. Crossfield and Dale [53] discussed the potential for the use of expert systems in TQM. According to their study it is feasible to run expert systems on personal (and other) computers in order to assist with aspects of TQM such as statistical process control, quality costing, goods receiving, corrective action procedures, supplier development, quality function deployment and field failure analysis, and that the potential application of expert systems to TQM is received with some degree of enthusiasm by the personnel with whom the concept is discussed. It is also pointed out that the most difficult task in producing an expert system is acquisition of knowledge, and that knowledge elicitation can be facilitated by use of the quality assurance mapping technique (Q-MAP). Cheng et al. [54] presented an ISO 9000 expert system which is written in Prolog. It is rule-based and backward chained. The knowledge of this expert system came from a consultant in this domain. The system gave the user the percentage of conformance to a particular statement in the ISO 9001 standard and delivers a quality manual which the user could use to implement the ISO quality system. Chan et al. [55] provided reports on the development of an expert system designed for auditors of quality management, discussing its development process

26

Chapter Two


(QMA-1), composition and structure. The system was tested in 20 Chinese organizations and user feedback reported. Outlines plans for future development of the system. Oppermann and Reiterer [56] proposed the ISO 9241 evaluator as a practical expert-based evaluation method for conformance testing with the ISO 9241 standard, which could be integrated in a comprehensive evaluation approach. Bayraktar [57] proposesd a knowledge-based expert system approach which has the guidance attributes for the auditing process of “contract review”, “ purchasing”, and “handling, storage, packaging, and delivery” quality system elements in the IS0 9000. In regard to the proposed system called expert system approach in the quality assurance system (ESAQAS). He presented the development and the application process of ESAQAS. Khan and Hafiz [58] described a Gauging Absence of Prerequisites (GAP) analysis expert system for ISO 9000 implementation by using an expert system shell called CRYSTAL 4. This expert system consisted of four sections; the introduction, the recommendation, the GAP analysis, and the trouble-shooting section. The ES had been validated using two different scenarios. Paladini [59] described the basic actions proposed for quality inspection. These actions involved structuring a decision supporting expert system (DSES) to help with those decisions related to the preliminary activities for inspection development, most of them relating to determining of the need or convenience of carrying out the inspection itself. Once the opportunity to carry it out was defined, the expert system helps the user to select the type of inspection to adopt from amongst: (1) automatic or sensorial inspection; (2) inspection by samples or complete; (3) acceptance or rectifying and, in the most relevant module, (4) inspection by attributes or by variables. Maroukhine and Berestneva [60] aimed at elaborating new models and algorithms for measuring the quality of different components of education process. One of the typical problems in the field of education was improving its quality. Nowadays there have been significant changes in the processes of international

27

Chapter Two


standardization for quality management such as ISO 9000:2000, which regulates the process of development, certification and maintenance of QMS at an enterprise. Herrera et al. [61] proposed a new methodology for self-diagnosis, based on concepts, goals and activities defined by CMM. They also define rules that allow the automation of the methodology through the use of an ES, which would play the role of a "virtual quality auditor", supporting the definition of an action plan and thus reducing the investment required for processes improvement. Reznik and Dabke [62] considered possible improvements in the methods of processing measurement results and uncertainty evaluation. They reviewed the procedures applied for the measurement uncertainty calculation according to ISO Guide and reported on an ES developed to teach and advise how to implement these procedures. Then they investigated the broader problem of the feasibility of applying new emerging intelligent methods such as ESs, fuzzy logic, neural networks and interval analysis for expression and evaluation of the measurement uncertainty. Fuzzy and interval theory models were proposed for describing some uncertainty components and two examples were presented in which a fuzzy specification of uncertainty was considered appropriate, one being the engineering measurement of sphericity and another one being measurement of computer security characteristics. Liao et al. [10] proposed an expert advisory system for ISO 9001 implementation by using an expert system shell called Visual Rules Studio®. This expert advisory system integrated the ISO 9001 quality system guidelines and an evaluation approach based on the MBNQA criteria into a knowledge-based expert system. By identifying the critical ISO elements and comparing the company’s current quality performance with ISO standards, this advisory system provided assessment results and implementation suggestions to the organization. The advisory system had been validated by a group of quality professionals. Goldberg and Shmilovici [63] proposed an ES approach that can classify the results of the quality assurance auditing (QAA). It could be used for training and monitoring of novice auditing personnel, reduction of the variation between different auditing experts, and facilitate decision making during a quality audit.

28

Chapter Two


Keller and Grif [64] offered to organize the knowledgebase of the expert system for QMS from several levels differing by a degree of detailed elaboration of the information. Introduction of QMS in higher educational establishments presented serious difficulties. The QMS in higher educational establishments requires vast information and data size. There are no dedicated specific programs for QMS. Eldrandaly [52] presented a prototype knowledge-based advisory system to play the role of "a virtual quality editor" to help individuals, organizations, and software companies who desired to implement the CMMI and ISO/IEC 9000-3 quality models. By identifying the gauging absence of prerequisites (GAP) between the prerequisites and what actually existed in the environment, this advisory system provided assessment results and suggestions to the companies so that successful implementation of these quality models can be achieved. Also, the system could be used for training novice auditing personnel. Lau et al. [65] proposed an intelligent system with a newly designed algorithm and the universal process data exchange standard to overcome the challenges of demanding customers who seek high-quality and low-cost products. The intelligent quality management system was equipped with the “distributed process mining” feature to provide all levels of employees with the ability to understand the relationships between processes, especially when any aspect of the process is going to degrade or fail. An example of generalized fuzzy association rules was applied in manufacturing sector to demonstrate how the proposed iterative process mining algorithm found the relationships between distributed process parameters and the presence of quality problems. From the previous literature review, it is shown that most of the proposed systems concentrate only on the implementation process of the models. So, this research aims to develop a knowledge based system for SQA models selection and implementation based on the resulting comparative study in order to help softwareproducing organizations to select and implement the more appropriate SQA model according to their quality requirements.

29

CHAPTER THREE SOFTWARE QUALITY MODELS AND APPROACHES: A COMPARATIVE STUDY

Chapter Three

SQA Models and Approaches: A Comparative Study.

Chapter Three Software Quality Models and Approaches: A Comparative Study.

This chapter discusses in details eight SQA models and approaches, illustrating their characteristics, benefits, and weaknesses, in order to provide a comparative study between them according to a textual description and a proposed comparison framework contains 30 characteristics categorized in five categories. The eight selected models and approaches are: 

TQM approach, with four national awards: 1. MBNQA 2. EQA 3. The Deming Prize 4. CAE



Capability maturity model integration (CMMI),



ISO/IEC 9000-3 (ISO 9001 Standard),



Six sigma approach,



Bootstrap methodology,



Trillium model,



TickIT scheme, and

 SPICE (or ISO/IEC 15504) standard.

30

Chapter Three


3.1 Total Quality Management (TQM) Approach 3.1.1 TQM origins and history In 1931, the American physicist, Walter Shewhart draws on the discipline of engineering, statistics and economics to develop a concept of statistical process control (SPC). Shewhart believes that total quality improvement could be achieved by organizations if they followed the Shewhart cycle [66]. Shewhart is credited as being the inventor of the control chart and the Plan-Do-Check-Act (PDCA) cycle. Deming and Juran teach the Japanese, in the 1950's, how to implement these ideas of process control and process improvements. Deming’s experience as a consultant in statistical methods is summarized in 14 management principles which are the basis of what is now called TQM. Juran develops many of the management tools and concepts of quality management and improvement. TQM become popular in the West since the early 1980's [67]. TQM is the art of managing the whole to achieve excellence. The TQM aim is straightforward; to improve the competitiveness, effectiveness, and responsiveness of the entire business. It is a way of ensuring that business processes are managed so that customers, both internal and external, are completely satisfied [1]. To create quality a product, a quality culture needs to be instituted from management on down so that everyone shares the responsibility for quality [68]. TQM Philosophy revolves around customer driven management. Its major emphasis is on determining customer need or expectation from the product. Total quality is the culture of the organization; it is attitude of people how they perform their assigned work with aims to provide, its customers with products and services that satisfy their needs [1].

3.1.2 TQM Principles and Characteristics Despite variations in its implementation, the key elements of a TQM system can be summarized as; customer focus, process, human side of quality, and measurement and analysis. Furthermore, an organization that practices TQM must have executive leadership, must focus on infrastructure, training, and education, and

31

Chapter Three


must do strategic quality planning. Clearly, measurement and analysis are the fundamental elements for gauging continuous improvement [3]. There are a number of different approaches to TQM but they have several characteristics in common. They all require organizations to; involve everyone in efforts to find better ways, develop and tap the full potential of all employees, manage by fact, using reliable data and information, manage business processes, not just functions or departments, and focus on satisfying customers needs and achieve financial goals [1]. The simple objective of TQM is "Do the right things, right the first time, every time". Arya [69] suggested that the TQM principles are; quality can and must be managed, life cycle costs, not front end costs, processes, not people are the problem, management involved and lead, everyone has a customer and is a supplier, every employee is responsible for quality, problems must be prevented, not just fixed, and quality improvements must be continuous, planned, and organized.

3.1.3 The TQM Representation Various organizational frameworks have been proposed to improve quality that can be used to substantiate the TQM philosophy. Specific examples include; PDCA cycle, Quality Improvement Paradigm/ Experience Factory (QIP/EF) Organization, SEI-CMM, and Lean Enterprise Management [3]. TQM has been widely implemented throughout the world. Many firms have arrived at the conclusion that effective TQM implementation can improve their competitive abilities and provide strategic advantages in the marketplace. Several studies have shown that the adoption of TQM practices can allow firms to compete globally. Implementing TQM is a never-ending process [19]. TQM approach emphasized using multi-functional teams (professional staff and workers from all departments involved) to solve problems. The teams are trained to use basic statistical tools to collect and analyze data. The TQM approach can be implemented using different statistical tools such as, check sheets, pareto diagrams, histograms, run charts, flow charts, cause and effect diagrams, scatter diagrams, and force field analysis [70]. 32

Chapter Three


3.1.4 TQM Criticism (advantages and disadvantages) According to Cater-Steel [66] in the US, from a survey of 20 companies, firms which adopt TQM experienced an overall improvement in corporate performance, greater customer satisfaction, increased market share, and improved profitability. Also, in a comparative analysis of TQM and ISO 9000, Zhang [19] interviewed managers of ten manufacturing companies in the Netherlands. This shows that TQM has much better effects on overall performance than ISO 9000, reducing quality costs by more than seven percent. Peleska and Zahlten [71], discussed the TQM Pitfalls, and what to do about them, they determine the pitfalls as the following; the ridiculous quality slogan, the middle management threat, the paper production machine, don’t know about quality costs, measuring the wrong quality indicators, the competing-team-pitfall, quality-means-correctness

misconception,

the

the

culture-replaces-technology

misconception, TQM measures are not appropriate for the cultural background, and spending all the money on minor quality improvements.

3.1.5 TQM National Awards Worldwide, there are several quality awards which adopt and implement the TQM philosophy, such as the European quality award (EQA) in Europe (1994), the Deming prize in Japan (1996), the Malcolm Baldrige national quality award (MBNQA) in USA (1999), and the Canadian award for excellence (CAE) is an annual award to recognize Canadian companies for performance excellence. The broad aims of these national awards are described as follows [19]:  Increase awareness of TQM because of its important contribution to superior competitiveness,  Encourage systematic self-assessment against established criteria and market awareness simultaneously,  Stimulate sharing and dissemination of information on successfully deployed quality strategies and on benefits derived from implementing these strategies,  Promote understanding of the requirements for the attainment of quality excellence and successful deployment of TQM, and  Encourage firms to introduce a continuous improvement process. 33

Chapter Three


Each award model is based on a perceived model of TQM. The award models do not focus solely on either product or service perfection or traditional quality management methods, but consider a wide range of management activities, behavior and processes that influence the quality of the final offerings. They provide a useful audit framework against which firms can evaluate their TQM implementation practices, seek improvement opportunities, and the end results [19]. The European Quality Award (EQA) EQA is officially launched in 1991. The primary purpose of the award is to support, encourage, and recognize the development of effective TQM by European firms. The model of the EQA is divided into two parts, enablers and results. The enablers are leadership, people management, policy and strategy, resources, and processes. These five aspects steer the business and facilitate the transformation of inputs to outputs. The results are people satisfaction, customer satisfaction, impact on society, and business results (the measure of the level of output attained by the firm) [19]. With a focus on business results, this quality system is an improvement over Baldrige, but is nowhere near world-class quality requirements [72]. The EQA model (1994) consists of nine primary elements, which are further divided into a number of secondary elements. The primary and secondary elements of EQA are; leadership, policy and strategy, people management, resources, processes, customer satisfaction, people satisfaction, impact on society, and business results [19]. The Deming Prize The Deming prize is established by the board of directors of the Japanese union of scientists and engineers in 1951. Its main purpose is to spread the quality gospel by recognizing performance improvements flowing from the successful implementation of firm-wide quality control based on statistical quality control techniques. The Deming prize proved an effective instrument for spreading TQM philosophy throughout the Japanese industries [19]. This quality system emphasizes statistics, but is not focused on leadership or on the principal stakeholders— customers, employees and suppliers [72]. There are ten primary elements in the Deming application prize (1996), as well as a checklist that is used to evaluate the performance of senior executives. This checklist emphasizes the importance of top 34

Chapter Three management’s

SQA Models and Approaches: A Comparative Study. active

participation

in

quality

management

activities

and

understanding of the main requirements of quality improvement programs. It is also provides senior executives with a list of what they need to do. The primary elements in the Deming application prize are; policies, organization, information, standardization, human resources, quality assurance, maintenance, improvement, effects, and future plans. Also, the checklist used to evaluate the performance of senior executives contains; understanding, policies, organization, human resources, implementation, corporate social, and future visions [19]. The Malcolm Baldrige National Quality Award (MBNQA) In 1987, the US Congress passed the Malcolm Baldrige National Quality Improvement Act, and thus established an annual quality award in the US. The aim of the award is to encourage American firms to improve quality, satisfy customers, and improve overall firms’ performance and capabilities [19]. The award is named for Malcolm Baldrige, who served as secretary of commerce from 1981 until his tragic death in a rodeo accident in 1987. His managerial excellence contributed to long-term improvement in efficiency and effectiveness of government. While its guidelines are better than ISO-9000 or QS-9000, the Baldrige award is far from being a world-class quality system. Its guidelines touch only superficially on the customer and on leadership. And they fall short in the all important “how to” of tools to achieve quality breakthroughs [72]. The model framework can be used to; assess firms’ current quality management practices, benchmark performance against key competitors and world class standards, and improve relations with suppliers and customers. The MBNQA framework (1999) contains the following; leadership, strategic planning, customer and market focus, information and analysis, human resource focus, process management, and business results [19].

Canadian Award for Excellence (CAE) The CAE is a TQM model. The model is made up of seven elements, are; leadership, planning, customer focus, people focus, process management, supplier focus, and organizational performance [21].

35

Chapter Three


3.2 Capability Maturity Model Integration (CMMI) 3.2.1 CMMI origins and history In 1985, Ron Radice and others at IBM adapts Crosby's quality management grid for the software industry. This work is directed by Watts Humphrey who takes the concepts from IBM to SEI [66]. In 1986, SEI, a federally fund research development center at Carnegie Mellon University, set out to help organizations improve their software development processes and their maturity [8]. Humphrey recognizes that this approach can help the US Government overcome problems with poor performance of software subcontract companies. Consequently, the SEI develops the CMM for software (SW-CMM) based on the organizational and quality management maturity models developed by Likert and Crosby [66]. Scientific origins of CMMs are statistical quality control (Deming’s TQM, Juran), quality management (Crosby), and Feedback from industry and government [23]. Since 1991, CMMs have been developed for a myriad of disciplines; as shown in figure 3.1. Some of the most notable include models for systems engineering (SE), software engineering (SW), software acquisition, workforce management and development, and integrated product and process development (IPPD). The CMMI Product Suite contains and is produced from a framework that provides the ability to generate multiple models and associated training and appraisal materials. These models may reflect content from bodies of knowledge in combinations most useful to you (e.g., CMMI-SE/SW, CMMI-SE/SW/IPPD/SS) [73].

Figure 3.1: The history of CMMs [Adapted from 20].

36

Chapter Three


3.2.2 CMMI principles and characteristics CMMI® is a process improvement maturity model for the development of products and services. It consists of best practices that address development and maintenance activities that cover the product lifecycle from conception through delivery and maintenance [20]. The organization can use CMMI to help set processimprovement objectives and priorities, improve processes, and provide guidance for ensuring stable, capable, and mature processes. A selected CMMI model can serve as a guide for improvement of organizational processes [73]. According to Bernhart [16] the principles of CMMI assessment are:  Quantitative management methods increase the organization's capability to control the quality and improve the productivity.  Application of the five-level capability maturity model that enables to evaluate the achievements and determine the efforts needed to reach the next capability.  Generic process areas that define the “what” — not “how” enable the model's applicability to a wide range of implementation organizations; it allows use of any life cycle model, any design methodology, development tool and programming language, and does not specify any particular documentation standard. The CMMI provides a set of recommended practices that define key process areas (KPAs) specific to software development. The objective of the CMMI is to offer guidance on how an organization can best control its processes for developing and maintaining software, and provide a path for helping software organizations evolve their current software processes toward software engineering and management excellence [8]. 3.2.3 CMMI representations The definition of a CMM allows the community to develop models supporting different approaches to process improvement. As long as a model contains the essential elements of effective processes for one or more disciplines and describes an evolutionary improvement path from ad hoc, immature processes to disciplined, mature processes with improved quality and effectiveness, it is considered a CMM. CMMI enables you to approach process improvement and appraisals using two different representations: continuous and staged [20]. 37

Chapter Three


The continuous representation enables an organization to select a process area (or group of process areas) and improve processes related to it. This representation uses capability levels to characterize improvement relative to an individual process area. The staged representation uses predefined sets of process areas to define an improvement path for an organization. This improvement path is characterized by maturity levels. Each maturity level provides a set of process areas that characterize different organizational behaviors. Three categories of factors that may influence your decision when selecting a representation are business, culture, and legacy [20]. A maturity level is a defined evolutionary plateau for organizational process improvement. Each maturity level matures an important subset of the organization’s processes, preparing it to move to the next maturity level. The maturity levels are measured by the achievement of the specific and generic goals associated with each predefined set of process areas. There are five maturity levels, each a layer in the foundation for ongoing process improvement, designated by the numbers 1 through 5; initial, managed, defined, quantitatively managed, and optimizing. Each maturity level has a specific number of KPAs [20]. 3.2.4 CMMI criticism (advantages and disadvantages) According to Cater-Steel [66] the CMMI best practices enable organizations to; link management and engineering activities more explicitly to business objectives, more fully comply with relevant international standards such as ISO 9000 and SPICE, expand the scope of and visibility into the product lifecycle and engineering activities to ensure that the product or service meet customer expectations, incorporate lessons learned from additional areas of best practice, and implement more robust highmaturity practices and address additional organizational functions critical to its products and services. CMMI model does not fully define the interfaces, interactions, or flow between the process areas or practices, and it does not cover the specifics of engineering design for particular products. Thus, CMMIs show what kinds of activities to include in a process model, but not how to string them together at the level of execution. A CMMI is a model of a process’s capability and maturity, not a model of the process itself [74].

38

Chapter Three


Also Bernhart [16] criticizes CMMIs, illustrating their drawbacks; lack of innovation, describes what an organization should have, does not say how to get there, process assessment flaws, and may lead to ossification of software processes. On the other hand, Arya [69] CMMI criticism has the following conclusions: subjective, binary, and intrusive evaluation, different approached for different organizations, CMMI does not specify methods, just criteria, no proper integration with TQM, unrealistic and skewed levels, cost of moving from one level to another, and large companies with traditional process model.

3.3 ISO/IEC 9000-3 (ISO 9001 Standard) 3.3.1 ISO origins and history The International Organization for Standardization (ISO) is the world's largest developer of standards. ISO officially formed in 1947 and is located in Geneva, Switzerland after delegates from 25 countries met in London the pervious year with the intention of creating an international organization whose mission would be "to facilitate the international coordination and unification of industrial standards" [8]. ISO contributes to the dissemination of technology and good business practice and support the development, manufacturing and supply of more efficient, safer and cleaner products and services. It makes trade between countries easier and fairer. ISO standards also safeguard users and consumers, and make many aspects of their lives simpler. ISO develops only those standards that are required by the market [22]. ISO 9000 is a quality series and comprises a set of five documents developed in 1987 by ISO. ISO 9000 standards and certification are usually associated with nonIS manufacturing processes. However, application development organizations can benefit from these standards and position themselves for certification, if necessary. All the ISO 9000 standards are guidelines and interpretive because of their lack of stringency and rules. ISO certification is becoming more and more important throughout Europe and US for the manufacture of hardware. Software suppliers will increasingly be required to have certification. ISO 9000 is a definitive set of quality standards, but it represents as part of TQM program [6]. The ISO 9000 Family consists of the following [69]:

39

Chapter Three


ISO 9000: Fundamentals and vocabulary ISO 9001: Requirements, the only standard in the ISO 9000 family against which third-party certification can be carried. ISO 9002: Models for quality assurance in production, installation, and servicing. ISO 9003: Models for quality assurance in final specification and test. ISO 9004: Guidelines for performance improvements.

3.3.2 ISO principles and characteristics ISO 9001, ISO 9002 and ISO 9003 have been integrated into the new ISO 9001:2000. ISO 9001:2000 is a series of generic standards on quality assurance and quality management. It specifies requirements for QMS for any organization that needs to demonstrate its ability to consistently provide product that meets customer and applicable regulatory requirements and aims to enhance customer satisfaction. It has been organized in a user-friendly format with terms that are easily recognized by all business sectors. It is used for certification/registration and contractual purposes by organizations seeking recognition of their QMS [22]. ISO 9001 is an international QMS standard used to assess an organization's management approach regarding quality. ISO 9001's focus is directed internally at an organization's processes and methods and externally at managing (controlling, assuring, etc.) the quality of products and services delivered. It is a generic international standard, adopted on a country-by-country basis, and written for use by the widest possible audience. As a result, the standard provides requirements (what needs to be done) and does not issue specific prescriptive solutions (how to do it). Being so broadly focused, the ISO 9001 standard does not offer details about its application to specific domains of expertise. To assist in the application of the standard for specific domains, a series of guidelines are available; e.g., ISO 9000-3 is a guideline for the software development industry [21]. A guideline to apply ISO 9001 to software industry, which is built on the principles; customer focus, leadership, people involvement, process approach, system approach to management, continual improvement, factual approach to decision making, and mutually beneficial supplier relationships [75]. ISO/IEC 9000-3:2004 is developed by JTC 1, SC 7 (Joint Technical Committee 1, Subcommittee 7). ISO/IEC 9000-3 2004 is officially published on February 15, 2004. It replaces the old ISO 9000-3:1997 standard. It provides 40

Chapter Three


guidance for organizations in the application of ISO 9001:2000 to the acquisition, supply, development, operation and maintenance of computer software and related support services [52]. ISO 9000-3 is an international guideline which provides "Guidance" on implementing an ISO 9001 compliant set of processes (as a "Quality System"). Guidance is for software development, supply and maintenance environments. The guideline is primarily written for "custom" (contract driven) software markets. ISO 9000-3 virtually mirrors the provision of ISO 9001 and does not add to, or otherwise change, the requirements of it. It is not intended to be used as an internal/external audit tool. Its intent is to guide software organizations with their ISO 9001 implementation and process change efforts, in short, software organizations are audited against ISO 9001 (not ISO 9000-3) [21].

3.3.3 ISO representations ISO 9000-3 describes a number of elements that should be included in an organization’s quality system for software. The elements are grouped into five Categories; systematic, management, resource, realization, and remedial. it consists of 20 elements [67]. ISO 9000-3 guideline is based on the premise that “a high quality process results in the production of high quality goods and services”. Such an emphasis means that organizations receive ISO 9001 certification, not the products they produce [76]. According to Bernhart [16] the Steps for ISO 9001 Registration (see Figure 3.2) are: 1. Establishing contact with an accredited registration body (a registrar).

Informational Meeting to clarify your goals and familiarize you with the Certification Process.

Document Review of your existing Quality System.

Surveillance Audits

Corrective Action

to assure maintenance of your Quality System.

within 30 days-guaranteed (Certificate Issued).

Preliminary Assessment to aid in Audit Planning.

Certification Audit by the team that best suits your needs.

Figure 3.2: ISO 9001 certification process [Adapted from 16].

41

Chapter Three


2. The registrar evaluates the company’s quality manual and on-site practices to confirm that the company’s quality system conforms to the standards and is consistent with its own documentation. 3. If these Assessments find no serious inconsistencies, the company is granted ISO 9000 registration. 4. Surveillance Audits are conducted after registration is granted.

3.3.4 ISO criticism (advantages and disadvantages) ISO 9001 provides a framework and systematic approach to managing business processes to produce a product/service that conforms to customer expectations. For customers, the certification of suppliers to ISO standards means that they can be assured that the development of their products and services are compliant to reference documents that are globally accepted. This, of course, means that customers and suppliers are able to compete in markets around the world. The benefits of becoming certified are numerous; companies should ensure that they are pursuing certification for the right reasons; to improve business processes and save money, to qualify for new customers, and to enter global markets [77]. Other advantages exit such as, developing self discipline, worldwide recognition, improving overall competitiveness, enhancing marketing credibility, serving as a global quality model, avoiding duplicative quality audits, and creating uniform quality system. It seems clear that businesses that have achieved registration are enjoying a competitive advantage [10]. ISO 9001 is a very comprehensive standard and defines all the quality elements required to demonstrate the supplier’s ability to design and deliver a quality product [6]. According to Liao et al. [10] ISO 9001 Registration not only gives companies advantages to stay competitive in the market but also improves their QMSs. Many companies that have succeeded in obtaining an ISO registration claim reduced defects and cost of quality, increased productivity, fewer lost-time accidents, better employee morale and communication, increased customer satisfaction and decreased complaints. On the other hand the disadvantages of ISO 9001 are [10]:  The cost of setting up and the drain on management time for ISO 9001 registration can become a significant hurdle to overcome for a small to mid-size companies. 42

Chapter Three


 With fewer internal resources, the ISO 9000 series standards are very difficult for smaller companies to adopt. The main Criticisms of the use of ISO 9001 standard for systems development are [38]:  The general nature of the standard and it recognizes only one certifiable level of quality.  Little attention is given to quality measures on the level of specific development projects.  The emphasis on strict procedures and their documentation in manuals leads to a bureaucratic type of behavior.  Though throughput time may be more important in some cases than superb quality and a somewhat "Quick and Dirty" approach might then be advisable. On the other hand Raftus [30] considered the ISO 9001 - Pros and Cons are; checklists are primarily used, marketing - promotional kits, ISO 9000 is a step on the way of continuous improvement, certificate is valid for three years on annual surveillance, continually on semi annual surveillance, pass or fail criteria, ISO certification is a level 2 CMM organization, and November 2000 version - more focus on; customer satisfaction, continuous process improvement, and data analysis. For most companies, ISO 9000 registration is an expensive and time consuming process [10]. For smaller companies, the direct and indirect costs associated with registration could be an enormous strain on resources. The amount of time and effort spent in complying with the standard and undergoing an audit may take away more away than the resulting quality system's gain in efficiency and productivity provides [76].

3.4 Six-Sigma (6σ) Approach 3.4.1 Six-Sigma origins and history Six-Sigma as a measurement standard in product variation can be traced back to the 1920's when Shewhart showed that three sigma from the mean is the point where a process requires correction. Many measurement standards, such as zero 43

Chapter Three


defects, later came to the fore, but credit for coining the term "Six-Sigma" goes to a Motorola engineer named Bill Smith [78]. Six-Sigma approach is originated by Motorola (Schqumburg, Illinois) in the mid-1980s. Sigma (σ) is a Greek letter and in statistics represents the standard deviation to measure variability from the mean or average (the target set) [59]. Here is a quality initiative that had a significant role for management in its implementation. Started at Motorola but popularized in the 1990s by AlliedSignal and General Electric [79]. In organizations, variation is often the cause of defects or out-of-control processes and translates into products or services that do not meet customer needs or expectations [8]. In the early days, benefits are operational, and the focus is on defect reduction and cycle time improvement. Today, companies use Six-Sigma to grow market share, improve customer retention, develop new products and services, accelerate innovation, and manage changing customer requirements [80]. Six-Sigma is a comprehensive and flexible system for achieving, sustaining and maximizing business success. It is uniquely driven by close understanding of customer needs, disciplined use of facts, data, and statistical analysis, and diligent attention to managing, improving, and reinventing business processes. The types of “Business Success” including; cost reduction, productivity improvement, marketshare growth, customer retention, cycle-time and defect reduction, culture change, product/service development and many more [81]. 3.4.2 Six-Sigma principles and characteristics There are three key elements of Six-Sigma; customer satisfaction, defining processes and defining metrics and measures for processes, and team building and involving employees. So, the Six-Sigma philosophy is to improve customer satisfaction through the prevention and elimination of defects and, as a result, increase business profitability. Six-Sigma defines defects in terms of the customer’s (not the engineer’s) viewpoint. Therefore, defects are product, service, or process variations which prevent customers from having their needs met, or which add cost, whether or not that cost is detected. Business profitability is the central motive of Six Sigma. It is a Top-Down initiative that cuts across the entire enterprise, including areas such as engineering, sales, marketing, and research [18].

44

Chapter Three


Six-Sigma is a fact-based, data-driven philosophy of quality improvement that values defect prevention over defect detection. It applies anywhere variation and waste exist, and every employee should be involved. In simple terms, Six-Sigma quality performance means no more than 3.4 defects per million opportunities or 99.99966% probability good [23]. Six-Sigma focuses on Defects per Opportunities (DPO) as a basis for measuring the quality of a process rather than products it produces, because products may vary in complexity. A defect may be thought of as any thing that results in customer dissatisfaction. The sigma value, therefore, tells us how often defects are likely to occur. The higher the value of sigma, the lower the probability of a defect occurring, as illustrated in Table 3.1. Therefore, Six-Sigma can be viewed as a quality objective whereby customer satisfaction will increase as a result of reducing defects; however, it is also a business-driven approach for improving processes, reducing costs, and increasing profits [8]. Table 3.1: Sigma degree and defects per million [Adapted from 8]. Sigma 1σ 2σ 3σ 4σ 5σ 6σ

Defects per Million 690,000 308,537 66,807 6,210 233 3.4

Six-sigma is a management framework that, in the past twenty years, has evolved from a focus of process improvement using statistical tool to a comprehensive framework for managing a business. It is particularly appropriate when a process is repeated long time within a manageable period [24]. 3.4.3 Six-Sigma representation Six-Sigma is a methodology that provides businesses with the tools to improve the capability of their business processes. This increase in performance and decrease in process variation leads to defect reduction and vast improvement in profits, employee morale and quality of product. This approach starts with the customer by defining what is critical to quality and ends by producing superior results for the customer [78]. In its first generation, Six-Sigma process improvement methodology included four logically linked phases: Measure → Analyze → Improve → Control. In

45

Chapter Three


the second generation, a new first phase, called Define, is added (DMAIC methodology) as illustrated in figure 3.3.

Figure 3.3: High-level DMAIC improvement methodology [Adapted from 79]. The strategy of six-sigma is called Business Process Management. This strategic component is the responsibility of executive management. The second component of Six-Sigma deals with the tactics of how project teams improve a broken process. The scientific method refers to defining and measuring a problem, analyzing its root cause, and testing theories of improvement [79]. Six-Sigma differs from traditional performance improvement programs in its focus on input variables. A Six-Sigma program demands that problems be addressed at the root level, eliminating the need for unnecessary inspection and rework processes [78]. To carry out a Six-Sigma program in an organization, a significant investment in training and infrastructure may be required. Motorola adopted the following martial arts terminology to describe these various roles and responsibilities: master black belts, black belts, green belts, and champions [8]. According to Siviy et al. [18] The Six-Sigma methods are:  DMAIC is used to improve and optimize existing processes and products.  DFSS (Design for Six-Sigma) is used to design new products and processes, and to redesign existing products and processes that have been optimized but still do 46

Chapter Three


not meet performance goals. One example sequence of DFSS is Define, Measure, Analyze, Design, and Verify.  Lean combined with 6σ is an increasingly occurring variant of the 6σ movement. The Six-Sigma toolkit is dynamic, organization-specific, and adapted for the domain. The decision to adapt, add, or focus on specific methods should be made to better meet customer needs and increase business benefits [18]. The standard SixSigma toolkit is a collection of techniques and statistical tools that include; regression, process mapping, histogram, scatter diagram, pareto chart, control chart, affinity diagram, run chart, cause-effect diagram, design of experiments (DOE), analysis of variance (ANOVA), statistical process control (SPC), critical-to-quality (CTQ) tree, quality function deployment (QFD), measurement system evaluation (MSE), process summary analysis worksheet, and failure mode and effects analysis (FMEA) [79]. Six-sigma is about cost reduction by eliminating defects. It is not team driven like TQM, but it is a management approach. Defect metrics are the main tool for leader to impact cost and margin. The QFD matrices are case-effect mappings between topics. The QFD matrices originate from Ishikawa- diagrams that are used to visualize root cause analysis. Thus they always have a direction from the CTOs to its impact on the deliverables, not on the contrary. The matrices can be considered as linear mappings that map the solution topic’s profile on the goal topics. When selecting a particular solution profile then this yields a corresponding goal’s profile, by applying the linear mapping defined by the matrix. The result is called “Effective Profile”. This compares with the original profile, the weights that we have set for the goal topics. The combinatory metrics (see figure 3.4) studies the difference between the solution profile and the effective profile. The two profiles coincide if the solution is ideal, that is, if the goal can be exactly reached with the given solution topics. This normally not the case. In all other cases profiles differ; the Convergence Factor being a metric for the difference. The indicator is the convergence factor, when zero it means complete convergence; up to one it is considered acceptable. Convergence factors greater than on indicate a significant difference between the deployed weight and the measured weight of the topic profile. Defects for the combinatory metrics network are defined as differences between convergence factors and effective deviations that exceed 1 [106]. 47

Chapter Three


Figure 3.4: The combinatory metrics in six-sigma [Adapted from106].

3.4.4 Six-Sigma criticisms (advantages and disadvantages) Benbow and Kubiak [23] provide differing opinions on the definition of 6σ approach; it is a Philosophy, a set of tools, and a Methodology. Lopez-Ona [82] finds that there are three Key characteristics of 6σ approach; leadership commitment, managing decisions with data, and training and cultural change. According to Motorola [80] there are six reasons why 6σ fails: 1. Lack of visible senior leader sponsorship. 2. Lack of alignment to a clear organizational strategy. 3. Lack of performance tracking and accountability. 4. Failure to link projects to bottom-line impact. 5. Insufficient or ineffective allocation of human resources. 6. Over-emphasis on rigid approach and technical tools. Also, Slavich [83] finds that there are some reasons for abandoning six-sigma approach such as; lack of executive management involvement, poor managerial leadership and high costs, poorly defined problem statements, six-sigma not integrated with business structure, and projects not meeting ROI expectations.

48

Chapter Three


The usefulness of 6σ lies in the conscious and methodical way of achieving customer satisfaction through the improvement of current processes and products and their design [8]. Using 6σ will guide the firm to; improve customer satisfaction, increase profit margins, shorten cycle times, and reduce costs. On the other hand, according to Slavich [83] the 6σ strengths are: ● Customized processes, process transparency and control. ● Executive involvement and favorable to change management. ● Measurable ROI and limited expenditures. ● Incremental implementation process. ● Evaluation of success confidence level. ● Evaluate → Diagnose → Treat → Validate. ● Clear problem definition & factual data for 6σ projects. ● Identification of cost-drivers/wastes & added value activities.

3.5

Bootstrap

Methodology, the European Software

Process Assessment and Improvement Method. 3.5.1 Bootstrap origins and history In 1988 the commission of the European community (CEC) runs a study on the performance of its major IT programme ESPRIT, finding out that technology transfer in particular in software engineering is not as successful as expected and as compared to other areas like CIM or office automation. The BOOTSTRAP project is defined and accepted as one of the pathfinders for CEC's 'European System and Software Initiative' (ESSI). The mission statement BOOTSTRAP chose at the time of its kick-off in 1989 is; 'the BOOTSTRAP project shall fertilize the grounds for introducing modern software technology into industry. The means are through analysis of the current state of software engineering in industry, identification of the potentials of change and motivation for accepting new contexts for software engineering [25]. Starting in 1990, ESPRIT, adapt Humphrey and Sweet's lead and develop what is called BOOTSTRAP methodology. The original BOOTSTRAP partners are from Germany, Italy, Finland, Austria, and Belgium [84]. 3.5.2 Bootstrap principles and characteristics Bootstrap's hypothesis is that before any investments are made in true technology (T) upgrade through products like tools and infrastructural computer

49

Chapter Three


support, the existentially critical questions on how to build solutions, i.e. methodology and methods (M), and on how to organize (O) development and maintenance of software have to be solved. Our formula of priorities in level and schedule therefore is,

O>M>T The scientific method explicitly applied in BOOTSTRAP is first to create

models, then to use, adapt and improve them. The aspects and scope had to be covered are; the target area of interest, the structure and behavioral description of this area, (i.e. its organization), the scaling of this organization using a reference model, the metrics for 'measuring' the organizations, and the process of changing the organization towards a 'better' state [25]. BOOTSTRAP is SPICE, ISO 9000 and CMM

®

compliant and designed to

help European commercial applications, and it concentrated on small- and mediumsize software systems and put more emphasis on rating individual software practices (especially technical practices) rather than global technical and management processes [84]. Its principle is to determine by assessment the gap between the current process state and the desired process state for a particular aspect of the business, and then to develop an improvement plan from that analysis. The model is developed and maintained by the Bootstrap Institute, based in Brussels [85]. BOOTSTRAP can be interpreted as providing a method and the instruments problem identification, i.e. for precisely describing where an organization stands and what gradual changes are to be recommended in the next steps [25]. According to ESSI-SCOPE [26] the objectives of the BOOTSTRAP are: 

Provide support for the evaluation of process capability against a set of recognized software engineering best practices.



Include internationally recognized software engineering standards as sources for identification of best practices.



Support the evaluation of how the reference standard has been implemented in the organization and assure the reliability and repeatability of the evaluation.



Identify, in the assessed organization, process strengths and weaknesses and Support improvement planning with suitable and reliable results.



Support the achievement of the organization's goals by planning improvement actions.



Help increase process effectiveness while implementing standard requirements in the organization.

50

Chapter Three


3.5.3 The BOOTSTRAP representation The basic process model which BOOTSTRAP chooses is an entity extended by feedback loops which refer to a comparison process calculating the difference between the measured characteristics and the desired or intended characteristics which one foresees, or in summary for a total organization modeled by many processes such as a software engineering department [25]. BOOTSTRAP combined its analytical methodology with a questionnaire aimed at developing an action plan for process improvement. Attempting to combine the 1987 SEI questionnaire, the European Space Agency life cycle model ESA-PS005, and ISO 9000-3, the BOOTSTRAP Questionnaire asked more (and more specific) questions than the 1987 SEI questionnaire. Problems remained, however, including a limited version of a working SPI development paradigm [84]. ESSISCOPE considers the main features of the BOOTSTRAP as the following; the assessment process, the process model, questionnaires, scoring, rating and result presentation, process improvement guidelines, and the bootstrap database [26]. BOOTSTRAP methodology version 3.2 (see figure 3.5) is fully compliant to ISO/IEC TR 15504 (SPICE). The core of the methodology consists of an assessment model and method [86].

ISO 12207

ISO 9001 ISO 9000-3

ISO/IEC TR 15504 (SPICE)

BOOTSTRAP 3.2

ESA

CMM

Figure 3.5: BOOTSTRAP and the methodologies it is based on. The BOOTSTRAP attribute tree (see figure 3.6) forms the basis for a software process measurement and a methodology subsequent assessment procedure to

51

Chapter Three


determine the software process quality. This classification later became the ISO 15504 standard (SPICE). BOOTSTRAP includes the following components [87]:

BOOTSTRAP V3.2

Organization

Life Cycle Dependent

Methodology

Technology

Life Cycle Independent

Management

Support

Process Related

Customer-Supplier

Figure 3.6: BOOTSTRAP attribute tree [Adapted from 26].  Reference Model against which the process capability is evaluated,  Assessment Method that defines the assessment procedure, and  Improvement Method that includes guidance on how the assessment results are used for process improvement. The Assessment Model of the methodology version 3.0 is updated to align with the ISO 12207 life-cycle and SPICE reference model requirements. In addition to the Process and Capability dimensions, it contains a technology dimension. The process dimension contains 33 different processes organized in six clusters; organization, life cycle dependent, management, support, customer-supplier, and process related. The capability dimension consists of six levels, each level consisting of one or more process attributes, adopted from SPICE [86]. Process capability is measured based on the following capability levels; Level 0: Incomplete Process, Level 1: Performed Process, Level 2: Managed Process, Level 3: Established Process, Level 4: Predictable Process, Level 5: Optimizing Process. The BOOTSTRAP method widens the assessment scope to the organizational level. Furthermore, assessment is more fine-grained than in CMM; the five capability levels are refined into quarters. Using the quarter-divisions helps managers and assessors to identify how far the organization is from the next

52

Chapter Three


level. This also motivates improvement. The underlying philosophy is that the improvement should be driven by organizational needs. The process dimension provides a framework for improving the individual processes. Each process is assessed based on its goals, scope and purpose. Processes are composed of base practices, which produce certain work products. These process inputs and outputs, in turn, should have certain characteristics to meet capability requirements [88]. The final report of a BOOTSTRAP assessment is a detailed analysis of the capability level of the company and the project(s). It includes [89]:  Where the organization stands (maturity profiles).  Analysis of strengths and weaknesses (ISO 9001 gap analysis).  Improvement guidelines (action plan, improvements for the short and long term). According to Koch [25] the goals of a BOOTSTRAP assessment are: (a) Measure and develop an SPU (Software Engineering/Producing Unit) quality profile analytically, discovering strengths and weaknesses of the SPU assessed, (b) Derive the steps for improvement from the strengths - and – weaknesses profile in terms of a plan of recommended actions to be taken immediately, and (c) Transform the action plan into a series of 'mini-projects' implementing the recommended improvement steps. 3.5.4 BOOTSTRAP criticisms (advantages and disadvantages) The main advantage of BOOTSTRAP is that it is SPICE compliant, but it also allows the selection of the processes to be assessed. This makes it easier to evaluate only a portion of the development organization, and to target improvement

at the

most critical operations. Managing and

understanding the improvement initiative is also more straightforward [88]. On the other hand, according to Raftus [30] the BOOTSTRAP pros and cons are; European initiative, Bootstrap database, more technology oriented than other models, process for deriving an action plan from the results is also provided. Also, BOOTSTRAP methodology proved to be a very efficient and effective means not

53

Chapter Three


only to assess a current status of software process quality and to initiate appropriate improvement actions [90].

3.6 Trillium Model,

Telecommunications Product development and

Support Capability Model.

3.6.1 Trillium origins and history Since 1982, Bell Canada corporate quality assurance has been assessing the software product development process of prospective suppliers as a means to minimize the risks involved and ensure both the performance and timely delivery of purchased software systems. Since April 1991, a new assessment model, subsequently named Trillium, has been developed in partnership with northern telecom and Bell northern research. Inspired by SEI-CMM, Trillium aims to benchmark an organization's product development and support capability in a commercial context. Bell Canada has been using Trillium since the summer of 1991. The model and its application method have been already improved many times based on this experience and feedback from suppliers [29]. Trillium is used to assess the product development and support capability of prospective and existing suppliers of telecommunications or IT-based products. It can also be used as a reference benchmark in an internal capability improvement program [28]. The Trillium model is based on the SEI-CMM v1.1 (development process maturity). Additional sources are; ISO 9001 and ISO 9000-3 (QMS), Bellcore's TRNWT-000179 and TA-NWT-001315 (telecom practices), MBNQA criteria (TQM), IEEE standards and IEC 300 (best practices), and professional and technical references [28].

54

Chapter Three


3.6.2 Trillium principles and characteristics The Trillium model has been developed from a customer perspective, as perceived in a competitive, commercial environment. In this context, capability is defined as the ability of a development organization to consistently deliver a product or an enhancement to an existing product: that meets customer expectations, with minimal defects, for the lowest life-cycle cost, and in the shortest time. A telecommunications product typically includes hardware, software, documentation, training and support services. The Trillium characteristics are [28]: 

Telecommunications Orientation.



Provides a product perspective and a customer focus.



Covers ISO, Bellcore, MBNQA, IEEE and IEC standards.



Includes additional Trillium-specific practices and technological maturity.



Provides a roadmap approach which sequences improvements by maturity. The Trillium goal is to provide a means to initiate and guide a continuous

improvement program. The model is used in a variety of ways, they are [28]: 

To benchmark an organization’s product development and support process capability against best practices in the industry,



In self-assessment mode, to help identify opportunities for improvement within a product development organization, and



In pre-contractual negotiations, to assist in selecting a supplier.

55

Chapter Three


3.6.3 Trillium representation The Trillium scale spans levels 1 through 5. The levels can be characterized in the following way; level1: unstructured, level2: repeatable and project oriented, level3: defined and process oriented, level4: managed and integrated, level5: fully integrated [28]. The Trillium model architecture consists of capability areas, roadmaps and practices, as shown in Figure 3.7. There are eight capability areas within the Trillium model. Each capability area contains practices at multiple Trillium levels. Each capability area incorporates one or more roadmaps. A roadmap is a set of related practices that focus on an organizational area or need, or a specific element within the product development process. Each roadmap represents a significant capability for a software development organization. The span of each capability area in Trillium model is illustrated in table 3.2 [28].

Figure 3.7: Architecture of the Trillium model [Adapted from 28]. Table 3.2: The span of each capability area in Trillium model [Adapted from 28]. -------------------------------------------------------------|Contains practices | Trillium Capability Area | at level | | 2 | 3 | 4 | 5 | --------------------------------------------------------------

56

Chapter Three


Organizational Process Quality |_X_|__X__|__X__|___| Human Resource Development & Management |_X_|__X__|__X__|___| Process |_X_|__X__|__X__|_X_| Management |_X_|__X__|__X__|___| Quality |_X_|__X__|__X__|_X_| System Development Practices |_X_|__X__|__X__|_X_| Development Environment |_X_|__X__|__X__|_X_| Customer Support |_X_|__X__|__X__|___| --------------------------------------------------------------

Bell Canada provides four different ways in which the Trillium Model is typically applied in the organization in [28]: 

Capability evaluation to benchmark an organization’s product development and support process capability against best practices in the industry,



Capability joint-assessment is a capability evaluation performed by a combined customer-supplier team. The conclusions and recommendations of this program represent a team consensus.



Capability self-assessment is more than an assessment of the processes, tools, techniques, etc. of the organization, because of the emphasis on their effectiveness in achieving business objectives.



Continuous improvement program is based on the Deming PDCA (Plan, Do, Check, Act) cycle. The Trillium model should always be used in a pragmatic fashion keeping in

mind that the bottom-line in all Trillium based activities or program is improving customers' (and shareholders') satisfaction. All Trillium activities should thus be context driven. This means taking into account factors such as; the nature of the product, the context in which the product will be used, the perception that the customer(s) has of the product (or product line) and its evolution, and the pertinent development and support organizations [29].

3.6.4 Trillium criticisms (advantages and disadvantages) For Customer Organizations, a higher capability means that [28]: 

The development organization is more responsive to customer and market demands,



The life-cycle cost of the product(s) is minimized, and



End-user satisfaction is maximized.

57

Chapter Three


For the Development Organization, achieving a higher capability can result in [28]: 

Lower development and maintenance costs, and Shorter cycle time and development intervals,



An increased ability to achieve content and schedule commitments due to effective project risk analysis and effort estimation, and



An increasing ability to meet quantifiable design and quality objectives at all stages of the development process. According to Raftus [30] the Trillium pros and cons are; telecom specific,

maturity profile, includes technology, product perspective, improvement road maps, and assessment methodology less advanced.

3.7 TickIT scheme 3.7.1 TickIT origins and history The TickIT Scheme is developed in 1990. It is launched in 1991, along with the first issue of its companion Guide, to assist in the expert interpretation of the ISO 9001 requirements when applied to software and IT [9]. The TickIT initiative came about as a result of a report commissioned by the British DTI to review the state of software quality and development in industry. This report showed that there is reluctance on the part of the software producers to adopt ISO 9000 as it is pitched at a high level of generality, the terminology is difficult to interpret for software and the guidance documentation is confusing. As a result of the findings of this report, the British Government decided to appoint BCS to lead an initiative called TickIT. The aim of this initiative is to create a detailed method for organization, procedures and rules for a Software Sector Certification Scheme (SSCS) which would cover the assessment and certification of an organization's software quality management scheme to ISO 9000/BS5750 [21]. The British software industry and the BDTI worked toward unified certification with TickIT ("tick" = "check" in British English and "IT" for Information Technology). TickIT is a guide to the implementation of ISO 9001 requirements for the IT industry [91]. Since May 1993 the responsibility of the TickIT scheme is transferred from DTI to DISC. DISC is committed to working on the maintenance and enhancement of 58

Chapter Three


the TickIT scheme, so that the principles of the scheme continue to be observed in the face of external changes; to the requirement of suppliers and customers, in development technology, in the content of the ISO 9000 series, and in the boundaries of its application. In 1992 TickIT V2.0 is released, and in 1998 TickIT V4.0 is released [21].

3.7.2 TickIT principles and characteristics TickIT is a certification scheme developed to apply ISO 9001, but with the advantage of having been "tuned" to deal with the special requirements of software development. The main principles of TickIT scheme are [26]: 

The interpretation of ISO 9000 for the sector.



The need to ensure continuing conformity for certified suppliers.



The necessity to perform assessments with experienced and skilled assessors, as witnessed by their ability to satisfy examiners.



The benefit of accredited training and examination for entrants on the assessor register. However, since the scope of TickIT is identical to ISO 9001, there is virtually

no content relating to process improvement, except in the context of changes resulting from corrective and preventive actions [26]. An important purpose of TickIT scheme, which is supported by the UK and Swedish software industries, has been to stimulate software system developers to think about; what quality really is in the context of the processes of software development, how quality may be achieved, and how QMSs may be continuously improved. Although certification of compliance to ISO 9001 is a contractual requirement for software suppliers in certain market areas, it should be a by-product of the more fundamental aims of quality achievement and improvement, and the delivery of customer satisfaction [92]. With regard to Certification itself, the Objectives are to [92]: 

improve market confidence in third party quality management system certification through accredited certification bodies for the software sector,



improve professional practice amongst quality management system auditors in the software sector, and

59

Chapter Three 


publish authoritative guidance material (TickIT guide) for all stakeholders. TickIT includes a guide provides guidance in understanding and applying ISO

9001 in the IT industry. It gives a background to the TickIT scheme, including its origins and objectives. Furthermore, it provides detailed information on how to implement a quality system and the expected structure and content relevant to software activities. The TickIT guide also assists in defining appropriate measures and/or metrics. The DISC TickIT office has just recently released the TickIT guide (Issue 5.5-November 2007). It has been extensively revised and updated to reflect the many changes in the industry. The TickIT guide includes the following sections: customer guide, supplier guide, and auditor guide [26].

3.7.3 TickIT representation A successful audit by a TickIT-accredited certification body results in the award of a certificate of compliance to ISO 9001:2000, endorsed with a TickIT logo. TickIT gives software developers an accredited quality system certification scheme that meets the special needs of the industry, enjoys the confidence of professional staff and commands respect from purchasers and suppliers [9]. By implementing a quality system and gaining certification, an organization is demonstrating management's commitment to quality and is sending a clear signal to users and competitors that the delivery of quality is a principal management objective. The TickIT logo indicates that the certificated organization is involved in software development and that TickIT guidance and certification procedures have been followed. TickIT applies to all types of information systems supply which involve software development processes. Typical systems suppliers include system houses, software houses and in-house developers. TickIT disciplines are also relevant to the development of embedded software. A full definition of the TickIT scope is included in the TickIT guide [92]. According to Hari and Sanne [9] the TickIT scheme applies to; softwarerelated services, peripheral services, facilities management, software replication, computer operations services, systems integration services, software product or service development, and software archiving, storage, and subcontracting.

60

Chapter Three


In the UK, TickIT is recognized by all government departments and major purchasers and it is compatible with European requirements for accredited quality system certification. World-wide, some 1400 TickIT certificates have now been issued. Currently, 50% of all new certificates are being granted to organizations outside the UK [92]. 3.7.4 TickIT criticisms (advantages and disadvantages) TickIT, which is completely revised to align with BS EN ISO 9001:2000, provides clause-by-clause guidance for software. According to Hari and Sanne [9] it has the following benefits:  Reduction in number of second party audits, due to acceptance by customers around the world of internationally-recognized accredited certification schemes,  Marketing advantage, an objective basis for process improvements, and a natural move towards improved organizational performance,  Improved in-house QMS expertise for assessing potential suppliers,  Orientation with continual improvement, which results in improved product quality and repeatability, increased process efficiencies, and increased customer satisfaction,  Enhanced efficiency of the organization through streamlining of its operations, reduction in failure cost; it is cheaper to get it right the first time,  Building customer confidence as their requirements are met resulting in repeat orders and/or new clients, and  Where the organization needs demonstrable management controls and product management, then TickIT provides the advantage. On the other hand, there are some disadvantages of TickIT scheme [9]: 

Implementation

might

become

difficult

with

frequent

changes

in

models/standards as well as new practitioners, 

Pressure to improve further even before the full benefits of current process improvement initiatives have realized.

61

Chapter Three


3.8 SPICE

(or ISO/IEC 15504) Software Process Improvement and Capability dEtermination Standard.

3.8.1 SPICE origins and history In 1991, an international effort is undertaken to refine key parts of ISO 9000-3, Bell Canada's Trillium, and Bootstrap by incorporating perspectives from the CMM. This effort officially begins at the June 1991 ISO plenary meeting of the joint technical committee of the ISO and IEC [84]. In June 1992 in London (UK), ISO/ IEC JTC1/SC7 approved at its plenary meeting a resolution recommending the creation of a new working group (WG10) to develop an international standard on SPA [93]. In January 1993 a program of work is approved by ISO/IEC JTC1 for the development of an international standard for SPA. The SPICE Project Organization completed its task of producing the set of working drafts in June 1995 and the user trials commenced in January 1995 [31]. The SPICE assessment framework has been developed using input from several other standards and assessment models and methods (see figure 3.8) [105].

Figure 3.8: SPICE relationships with other standards and models [Adapted from 105]. The SPICE framework for SPA can be used by organizations involved in planning, managing, monitoring, controlling, and improving the acquisition, supply, development, operation, evolution and support of software. SPICE standard provides a structured approach for SPA (see figure 3.9) for the following purposes [31]: 1. an organization with the objective of understanding the state of its own processes for process improvement; 62

Chapter Three


2. an organization with the objective of determining the suitability of its own processes for a particular requirement or class of requirements;

Figure 3.9: SPICE framework structure [Adapted from 105]. 3. one organization with the objective of determining the suitability of another organization's processes for a particular contract or class of contracts. One can most easily describe the SPICE effort by saying that it intended to modify the CMM's primary focus on total organizational capability by incorporating Bootstrap's emphasis on particular process elements. In the CMM, control over the organization's process is achieved through a step-by-step progression in which focus shifts as an organization climbs the maturity ladder from controlling an individual project's software process to controlling the system-wide organizational software processes. SPICE, however, attempted to evaluate the "capability" level of individual processes rather than the "maturity level" of the organization as a whole [84].

3.8.2 SPICE principles and characteristics The SPICE framework includes a reference model for process assessment. The reference model has two dimensions: the process dimension, which provides a strong mapping to the international standard for software life cycle processes (ISO/IEC 12207), and the capability dimension, which defines a measurement scale for the evaluation of the process [88]. According to Bernhart [16] the principles of the SPICE assessment model are the following:

63

Chapter Three 


Harmonize the many existing “independent” assessment methodologies by providing a comprehensive framework model (“What” has to be accomplished rather than “How” it has to be done).



Be universal to serve all or almost all categories of software suppliers, customers and software categories, also be highly professional.



Aim at reaching international acceptance as world standard. To save suppliers' resources by eliminating the need to perform several different capability assessments in response to different customer requirements. The characteristics of the process assessment framework are; encourages self-

assessment; takes into account the context in which the assessed processes operate; produces a set of process ratings (a process profile) rather than a pass/fail result; through the generic practices, addresses the adequacy of the management of the assessed processes; and is appropriate across all application domains and sizes of organization. SPICE has been designed to satisfy the needs of acquirers, suppliers and assessors, and their individual requirements from within a single source [31]. SPICE deals with software processes (e.g. development, management, customer support, quality, etc.) and also is concerned with People and Technology. It is intended to be applicable to different business sectors, application domains, and organization sizes, as well as being culturally independent [93].

3.8.3 SPICE representation SPICE standard is Europe’s implementation of CMM. It has an assessment framework by which an organization may establish and subsequently improve its process capabilities in the acquisition, supply, development, operation, evolution and support of software [85]. According to Hoffman [94] and Drouin [93] organizations are able to use the SPICE standard in (see figure 3.10):  Capability determination mode, to help determine the capability of a potential software supplier,  Process improvement mode, to help improve their own software development process,

64

Chapter Three


 Process assessment, self-assessment mode, to help determine their ability to undertake a new project.  Qualification and training of assessors. Process is examined by identifies changes to

Process Assessment leads to

Process Improvement

identifies capability and risks of leads to

motivates

Capability Determination

Figure 3.10: Software process improvement framework [Adapted from 31]. SPICE framework defines a reference model of processes and process capability that forms the basis for any model to be used for the (singular, no plural) purpose of SPA [95]. The SPICE architecture is two-dimensional; One dimension consists of the processes that are actually assessed (process dimension) that are grouped into five categories; customer-supplier, engineering, management, support, and organization [31]. The second (the capability dimension) consists of the capability levels as shown in Figure 3.11, which is used to evaluate the process capability. The capability dimension defines a six-level capability scale (from Level 0: incomplete to Level 5: optimizing) [16].

65

Chapter Three


Figure 3.11: The SPICE capability levels [Adapted from 16].

3.8.4 SPICE criticisms (advantages and disadvantages) According to El Emam et al. [95] the SPICE standard can be used for SPI as the following: 

SPICE assessment costs are lower than for CMM, and also SPICE fully contains QA and is TQM based. Also, there is more emphasis on objective evidence in SPICE assessments than in CMM.



SPICE provides specific guidelines for fairly intangible processes, and offers the most generic and flexible method for determining improvement opportunities. The central SPICE document (the base practices guide or BPG), says that

a "capability level is a set of common features...that work together to provide a major enhancement in the capability to perform a process" (emphasis added). SPICE calls this rating of an organization's processes individually a "process assessment approach," which emphasizes a "Continuous" framework of assessing individual processes [84]. SPICE identifies the whole range of processes that goes into a function and evaluates each process. Thereby, it succeeds in giving a profile of the organization giving out lot more information [96]. The very fact that SPICE allows for the evaluation of individual processes such as project management and configuration management in an organization is a big advantage. SPICE's treatment of acquisition is better than CMMs. Unfortunately; the fact that SPICE gives different numbers for various processes becomes a handicap in the eyes of some executives. SPICE is inexpensive and much less resource intensive. It is a handy method to assess the capability of the software for small companies [96].

66

Chapter Three


The SEI is the U.S.’s representative on the SPICE project, and has been integrating some of the SPICE features into the development of the CMMI. One of the key differences between SPICE and the CMM has been the portrayal of the assessment results. Whereas the CMM presents levels of maturity from (1 to 5), SPICE presents a profile of capability levels (0 to 5) in each of four process categories and within their associated processes, this is known as the continuous representation [85]. There is growing interest in SPICE, it has been estimated that approximately 1,260 SPICE-based SPAs are conducted during the 22 months from September 1996 to June 1998 [66]. Over 2,000 assessments have been done all over the world with Japan and Australia being the prominent countries. Some companies opt for SPICE for specific assessments and CMMIs for others [96].

3.9 The Comparative Study of SQA Models 3.9.1 The objectives of the proposed comparative study 

There are organizations without a SQA model that would like to learn about different models because of competitive pressure or certification requirements. Such organizations usually have only limited general SQA knowledge and even less knowledge about specific models. This requires the comparison to be on a high level with few details.



There are organizations with an organized and systematic model that would like to learn about other models. Many organizations find themselves in a situation where they have to use more than one SQA model or approach. As an aid in the process of learning about other models one would obviously like to utilize already existent SQA knowledge. A discussion of the lesser-known models from the point of view of the well-known model could therefore be beneficial [37].



The resulting comparative study will be used as a first step for building an intelligent system for the SQA models and approaches selection and implementation.

3.9.2 The proposed comparison framework Many companies are finding themselves in a position where they must assess their QMS using more than one assessment model or they have to integrate two models. This is causing an increase in the cost of conducting business, which can be 67

Chapter Three


minimized by understanding where these assessment models overlap and how they differ from one another [32]. The goal of this study is to make the comparison of SQA models easier and more objective for the software development organizations in order to help in adopting the more suitable SQA model according to their requirements. This research compares between the selected eight quality models using the Characteristics comparison technique in addition to the textual description for each quality model or approach illustrating its origins, history, principles, characteristics, representation, advantages, and disadvantages as described in the previous sections. The Characteristics comparison technique is used as it is well suited for a general overview of the quality models and it can be used as a basis for other comparison methods. A lot of information can be inferred from a table listing the characteristics of several models. Therefore from the analysis and study in details of the SQA models and approaches, and the detailed comparative study of these available models and the modification of the previous related works, this research presents a list of 30 characteristics, i.e. areas of interest, relevant for discussing differences and similarities between SQA models. These characteristics are collected from the previous comparisons in addition to new characteristics mentioned in this study, such as "Integration Ability". There are so many characteristics, so, they have been grouped in five categories to enhance comprehensibility and readability as shown below in Table 3.3. Table 3.3: Categorization of comparison characteristics. [Collected from References 27, 32, 34, 35 and 40]. General Geographic origin / spread Scientific origin Development/stability Popularity Software specific Prescriptive/descriptive Scope Adaptability

Process

Organization

Assessment Assessor Process Improvement method

Improvement initiation Improvement focus Analysis techniques Evaluation Documents Document Management Requirements Management

C1. General Category

68

Actors/roles/ stakeholders

Organization size Coherence

Quality Quality perspective

Progression Causal relation Comparative

Results Goal Process artifacts Certification Cost of implementation

Validation Integration Ability

Chapter Three


This category of characteristics describes general attributes or features of SQA models. Such attributes are often specific to each model and frequently related to how the model is constructed or designed.

1. Geographic Origin/Spread This characteristic describes where in the world the model or approach originated as well as where it is commonly used today [32].

2. Scientific Origin The scientific origin of a model is the philosophical background on which it is based. Some models have originated from purely philosophical principles, while others are based on mathematical or statistical rules. Some models can even be said to originate from previous models, by borrowing heavily from them in terms of concepts, content and focus [32].

3. Development/Stability One should expect that the stability of a model increases as experience is gained over a number of years. Many models depend on feedback from users to evolve as time goes by, e.g. identification of parts in the model that are difficult to use or especially useful. Changes in the software engineering discipline may be reflected as well, for example new technologies or the introduction of new SQA models and standards. Naturally, it is desirable to employ a model that is both evolved and relatively stable [32].

4. Popularity Popularity is also important for an organization selecting to employ a certain process improvement approach. One would like to be sure that support and further development of the model is continuing in the foreseeable future. Those models with a greater mass of supporters also have a tendency to receive better support [32].

5. Software Specific Whether the model is software specific or not reveals something about its scope. Some of the approaches are specifically directed towards software

69

Chapter Three


development and maintenance, while others are more general. The latter must be adapted to the software development and this requires an extra effort [32].

6. Prescriptive/Descriptive A prescriptive model, prescribes requirements and processes that are mandatory. In contrast, a descriptive model does not assign specific actions to be taken by the organization. Instead, it describes a state or certain expectations to be met without stating how they should be accomplished [32].

7. Scope The specific area in which the SQA model or approach can be adopted. For example, the model scope may be organization improvement, software development, telecommunication, or trade, etc [34].

8. Adaptability The degree of flexibility varies among the process improvement approaches. An adaptive approach has support for tailoring and is customizable for specific uses. A study is indicated that the use of comprehensive SQA approaches is still quite low, partly because of their rigidity. This suggests that adaptability is an important quality of a model [32].

C2. Process Category The process category concerns characteristics that describe how the SQA model can be used or implemented.

9. Assessment The improvement work is often initiated by an assessment of the organization’s current software process capabilities. Models that include such assessment methods specify techniques for assessing software processes, practices and infrastructure, with the result being an identification of strengths and weaknesses. The assessment often includes recommendations for process improvement and it can therefore be used as an improvement road map. There are at least two dimensions to assessments, namely what is assessed and who performs the assessment. The latter part is covered by the next characteristic 70

Chapter Three


called assessor. This investigation shows substantial differences between the models in what their assessments are meant to cover. For example, some models assess organizational maturity, others process maturity and then again others may assess customer satisfaction [32].

10. Assessor The second dimension of assessments is who it is carried out by, i.e. the assessor. Generally, this is the organization itself or an external group depending on the purpose of the assessment. Some studies show that different assessors reach different conclusions when the same organization is assessed. This challenge should be addressed by the models in question and a few initiatives have been made in this direction, e.g. TickIT. In this study, assessments are characterized as

either “external” or “internal”, or

a combination of these [27].

11. Process Improvement Method Some models are purely assessment models, while others include guidelines or process improvement methods used to implement and institutionalize the process improvement. A guideline like this is often called a process improvement life cycle. It “defines the strategy and the process by which improvement is pursued” [35]. This characteristic indicates the type of improvement method included in the model, if any [32].

12. Improvement Initiation Improvement initiation is related to the previous characteristic and indicates where the improvement work is initiated in the organization. In other words, it is the approach used to achieve improvement. The approach can for example be Top-down or Bottom-up. However, other approaches exist as well, e.g. SPICE standard initiates the improvement work at the level of process instances. A combination of these approaches could also have been used [32].

13. Improvement Focus Some SQA models or approaches focus primarily on experience reuse, whereas others concentrate on management processes or other areas. They differ in improvement focus, i.e. which SQA activities they regard as essential for

71

Chapter Three


improvement. The difference between the focus and the goal characteristic, which will be described later, might not be clear and it deserves some extra attention. Improvement focus is related to activities and deals with the means of achieving the end result. It is therefore placed in the process category. In contrast, the goal characteristic is only concerned with the end result not how it is achieved [34].

14. Analysis Techniques Their use of analysis techniques can also be employed to classify process improvement models. Some models use quantitative techniques, others qualitative and still others might not use any defined techniques at all. Examples of analysis techniques are questionnaires, statistical process control, various measurements, etc [32].

15. Evaluation Documents The documents used for the evaluation process by external consultants may be “Normal” documents (as in use in the company), or special “quality” documents (prepared specially for the assessment/audit) [27].

16. Requirements Management To which degree the SQA model or approach support the management of the organization and business requirements. The degree may be fully supported, partially supported, or not supported [40].

17. Document Management To which degree the SQA model or approach support the management of the documentation process. The degree may be fully supported, partially supported, or not supported [40].

C3. Organization Category The characteristics in the organization category are directly related to attributes of the organization and environment in which the SQA model or approach is used, i.e. who is involved and for what kind of organization is the model applicable.

18. Actors/Roles/Stakeholders 72

Chapter Three


Each model has its own primary actors, which are the people, groups or organizations affected by or taking part in the improvement process. Generally, each such actor has a predefined role in the improvement process [32].

19. Organization Size This characteristic indicates the organization size for which the SQA approach is suitable. Some models are so comprehensive that only large corporations have the resources to use them. Other approaches are easier for smaller companies to employ [32].

20. Coherence Coherence is the logical connection between engineering factors and factors related to the business or organization. In an article by Cattaneo et al. [35] this is pointed out as extremely important for a successful SQA implementation, yet still insufficient in many process improvement models. There are two types of coherence and both should be maximized for the greatest benefits: Internal coherence – The fit among the different parts of a company, such as organization, technology, human resources, strategy and culture. External coherence – The fit between the organization and its environment including market, customers and competitors [35].

C4. Quality Category The quality category deals with characteristics related to the quality dimension by pointing out aspects such as how progression is measured, whose quality perspective is employed and what that means in terms of quality indicators and causal relations.

21. Quality Perspective When looking at quality, it is necessary to establish a clear understanding of whose quality perspective the approach adopts. The concept of quality will naturally be different depending on who is asked. For example, the management will have a different perspective compared to what a customer might have [32].

73

Chapter Three


22. Progression If the SQA model is looked upon as a road map, progression along the road can be characterized as “flat”, “staged” or “continuous” as shown in Figure 3.12. ISO 9001 is a typical flat model with only one step, namely certification. One cannot be more or less certified. On the other hand, in the CMMI there is a discrete ladder of staged maturity levels, each indicating a specific level of process quality. There are also approaches that proclaim a continuous and evolutionary quality progression, e.g. TQM and QIP [32].

Figure 3.12: The progression characteristic [Adapted from 32].

23. Causal Relation SQA models assume that product quality is affected by the quality of the process used to produce it. However, process quality is difficult to determine and therefore most models measure some quality indicator instead. This suggests the following causal relation; F’’ (SQA model) F’(Quality indicator) Quality(Process) Quality (Product) None of the models explicitly states a causal relation like those above. However, they are implicit and this paper proposes this explicit representation [32].

24. Comparative Some models are comparative in nature, meaning that they can be used to compare certain aspects of different organizational units, either internally or externally. The aspect compared may vary between the models, but it is supposed to represent an objective comparison criterion. For example, two organizations assessed using the CMMI can be compared on the basis of their maturity levels, i.e. the one with the higher maturity level is assumed to have the higher process quality as well. In contrast, when using ISO 9000 the only comparison that can be made is whether the organizations are certified, a very

74

Chapter Three


coarse characterization. Models that are not comparative are only used as guidelines within a single project or organization [32].

C5. Result Category This category treats characteristics that describe the results of employing an SQA model, but also the costs of reaching these results and the methods used to validate them.

25. Goal The goal of the model is its primary objective or end result, i.e. what is achieved by using the model. As discussed previously it differs from the improvement focus characteristic by only treating the end result, not the means of getting there [32].

26. Process Artifacts Process artifacts are the “things” created as a result of adopting an SQA model. The process-related artifacts can be tangible, such as documents containing process models, improvement plans, experience data etc. However, the artifacts can also be intangible, e.g. increased knowledge in the organization. In either case, the process artifacts are created in addition to the actual product being produced [32].

27. Certification The International Organization for Standardization (ISO) and a variety of national standard bodies have developed standards to which an organization can be certified. Only conformance with the minimum requirements is measured in the certification process. A standard to which one can be certified is comparative by nature. Certification is closely related to the flat quality progression previously discussed, indicating that not all SQA approaches lead to certification [32].

28. Implementation cost The implementation cost is naturally an important characteristic, because it often dictates whether the approach is financially feasible. There are considerable cost differences among the models, making some impossible for smaller organizations to adopt. El Emam et al. [95] state that “the implementation of certain processes or process management practices may not 75

Chapter Three


be as cost-effective for small organizations as for large ones”. However, the exact impact of organization size on cost is not obvious. Because the cost is affected by a number of factors outside the SQA model, it will be impossible to provide exact numbers in this characteristic. However, rough estimates should be provided where such numbers can be obtained [27].

29. Validation Validation is needed to evaluate whether the improvement efforts have actually worked. Such validation should distinguish between improvements resulting from SQA efforts and improvements resulting from other factors. The question we want answered, is whether the same benefits would have been achieved without a comprehensive SQA effort. In other words, the difficulty lies in establishing a direct causal relation between the SQA efforts and the achieved improvements [32].

30. Integration Ability This characteristic illustrates which models can be integrated in order to recover the weaknesses of each other, for example, Six Sigma and CMMISM, are complementary and mutually supportive. In order to present the comparative study in a simplified and a general overview, the following Tables 3.4 (a) and (b) summarize the Comparative study of the selected eight models and approaches according to the proposed framework.

76

Table 3.4 (a): The comparison of SQA models and approaches [TQM, CMMI, ISO 9000-3, and Six Sigma]. Category

C1. General

C2. Process

Characteristic

C4. Quality

C5. Result

Chapter Three

Geographic Origin/Spread

Scientific Origin

Quality Control Management

Development/Stability

Entire post-war era (Since 1950)

Popularity Software Specific

High (esp. in Japan)

Prescriptive/Descriptive Scope Adaptability

Descriptive

Assessment Assessor

Organization Performance Internal and External PDCA cycle Top-Down Management processes

Process Improvement Method

Improvement Initiation Improvement Focus Analysis Techniques

C3. Organization

TQM Japan / World-wide

and

NO Organization Improvement.

Yes

CMMI

SIX SIGMA SQA ModelsISO and9000-3 Approaches: AMotorola Comparative Study. (U.S.A) / World

U.S.A / World-wide TQM 'Total Quality Management", SPC "Statistical Process Control" Since 1986

Europe / World-Wide

Top (esp. in U.S.) Yes Both Software Development Limited Organization Maturity Internal and External IDEAL Top-Down

Quality Management Systems

Advanced Business Statistics and Engineering and Project Management.

Since 1987

Since the mid-1980s.

High (esp. in Europe)

High

NO

NO Both

Both Trade (General) Limited Process External ISO 9000-3 Guidelines

Organization Improvement.

Yes Process Performance Internal and External

Documentation Review of QS.

DMAIC – DFSS Top-Down

Management Processes

Quality Control


Assessment Questionnaires

ISO Guidelines and Checklists

SPC

Evaluation Documents

7QC, 7MP, SPC, QFD New

Old

New

New

Requirements Management Document Management

Partially Supported Partially Supported

Partially Supported Not Supported

Not Supported Fully Supported

Fully Supported Partially Supported

Actors/Roles/Stakeholders Organization Size Coherence

Customer, Employees, Management

Management Large Internal

Customer, Supplier Large

Management, Customer, Employees

Quality Perspective

Customer

Customer

Progression Casual Relation

Continuous (Flat for Awards)

Management Staged and Continuous

All Internal and External Customer

Flat

Continuous

F1’(Business Performance) F2'(Improvement Plans) Q(Process) Q(Product)

F1’(Key process areas) F2'(Maturity level) Q(Process) Q(Product)

F1’(Quality elements) F2'(Certification) Q(Process) Q(Product)

F1’(Measurements) F2'(SPC, σ Level) Q(Process) Q(Product)

Comparative

No

Yes, maturity level

Goal

Customer Satisfaction

Yes, Certification Establish Core Management Processes

Yes, Sigma level

Process Artifacts

Improvement Plans, Diagrams

Process Improvement, Supplier Capability Determination Process Documentation, Assessment Result

Process Documentation, Certificate

Diagrams, Plans

Certification Implementation Cost Validation

NO ( Award) Expensive Case studies

Yes Very expensive Survey

NO Expensive Case Studies and Survey.

CMMI or SIX SIGMA

ISO 9001 Certification

Integration Ability

Large Internal and External

NO Expensive Surveys and case studies

ISO 9001 Certification ISO 9001 Certification

77

Internal and Limited External

Customer Satisfaction & Business Success

Table 3.4 (b): The comparison of SQA models and approaches [Bootstrap, Trillium, TickIT, and SPICE]. Category

C1. General

Characteristic

TRILLIUM SQA Models and TICKIT SPICE Approaches: A Comparative Study.

Bell Canada / World -Wide

London (UK) / World -Wide

Scientific Origin

ISO, Bellcore, MBNQA, SEI-CMM V1.1, IEEE and IEC Standards.

Interpretation of ISO 9001 for the Software field.

Development/Stability

Since 1989

Since 1991

Since 1993

High (esp. in Europe)

High (esp. in Canada )

Since 1991 High (esp. in Europe[UK 73.8%])

Software Specific

Yes

Yes " Telecommunication Orientation, Product Perspective"

Yes

Yes

Prescriptive/Descriptive Scope Adaptability

Both

Both

Both

Both

Software Development

Telecommunication

Trade

Software Development

Yes

Yes

Yes

Yes

Assessment Assessor

Organizational Maturity External

Organizational Maturity Internal and External

Process External

Process Improvement Method

PDCA cycle, O>M>T

CMM v1.1, PDCA cycle

ISO 9001:2000 Requirement.

Process Maturity Internal and External SPICE Guidance- Doc. Part 7

Improvement Initiation Improvement Focus Analysis Techniques

Process Improvement Guidelines

Top-Down Process + Technology

Documentation Review of QS.

Process Instance

QS Definition/Improvement


Assessment Questionnaires

Roadmaps and Practices

TickIT Guide-Issue5.5 Nov.2007

Several (Manual & Automated) Required.

Evaluation Documents

Old Not Supported

Old Fully Supported

Old Fully Supported

Popularity

C2. Process

BOOTSTRAP Chapter Three

Europe / World -Wide ISO 9000, CMM, ESA Standards

Geographic Origin/Spread

Organization, Methodology, Technology

London (UK) / World -Wide CMM, ISO 9000, Bootstrap, Trillium, SPQA.

High

Partially Supported

Partially Supported

New Not Supported Fully Supported

Actors/Roles/Stakeholders

Management

Management, Customer, Supplier

Customer, Supplier

Management

Organization

Organization Size

All

All

All

All

Coherence

Internal

Internal

Internal and Limited External

Internal

C4. Quality

Quality Perspective

Management

Customer

Customer

Management

Progression

Continuous

Staged

Flat

Continuous (Staged at process instance level)

Casual Relation

F1’(Process Attributes) F2'(Capability level) Q(Process) Q(Product)

F1’( Roadmaps ) F2' (Maturity level) Q (Process) Q (Product)

F1’(Quality elements) F2'(Certification) Q(Process) Q(Product)

F1’(Process attributes) F2'(Capability level) Q(Process) Q(Product)

Comparative

Yes, Capability Levels

Yes, Maturity Profile

Yes, Certification

Yes, Maturity Profile

Software Development Capability, Process Improvement Capability Profile, Improvement Plans.

Process Improvement, Supplier Capability Determination Organization Profile, Process Documentations.

Customer Satisfaction, Establish core Management Processes

SPA and SPI

Process Documentation, Certificate.

Process Profile, Assessment & Capability Reports

Certification Implementation Cost Validation

NO Moderate Surveys and Case Studies.

NO Moderate Surveys and Case Studies.

Yes Moderate Survey

NO Moderate

Integration Ability

According to Organization need

ISO 9001Certification

CMMI or SIX SIGMA

According to org. needs

Requirements Management Document Management

C3.

C5. Result

Goal Process Artifacts

78

Fully Supported

Document Review, Trials (Case Studies & Surveys)

CHAPTER FOUR A KNOWLEDGE-BASED SYSTEM FOR SQA MODELS SELECTION AND IMPLEMENTATION

Chapter Four: A knowledge-based Expert System for SQA Models Selection and Implementation

CHAPTER FOUR A KNOWLEDGE-BASED SYSTEM FOR SQA MODELS SELECTION AND IMPLEMENTATION

This chapter describes the development of a prototype intelligent knowledge-based system for the selection and implementation of software quality assurance models (SQAMSI Advisory System) using COM technology.

4.1 The Proposed SQAMSI Advisory System The proper selection and implementation of the SQA models is often a difficult, a time consuming, and a costly task for software organizations especially small and medium ones. In this research, a knowledge-based system for software quality models selection and implementation (SQAMSI advisory system) is developed by modifying the SQA advisory system developed by Eldrandaly [52]. The proposed system has two main objectives: 1. Assisting the user in selecting the more suitable SQA model. 2. Playing the role of a “Virtual Software Quality Editor”. The proposed system achieves its objectives via the following two phases: Selection phase: in this phase an expert system is used to identify the more suitable SQA model or approach that meets the desired individuals’ / organizations’ objectives, and requirements. Implementation Phase: in this phase an expert system is used to play the role of a “Virtual Software Quality Editor”. The proposed system supports individuals and software companies in implementing three different software quality models (CMMI, ISO, and SPICE). By identifying the GAP (Gauging Absence of Prerequisites) between the prerequisites and what actually exists in the present 78

Chapter Four: A knowledge-based Expert System for SQA Models Selection and Implementation environment, the SQAMSI advisory system provides assessment results and suggestions to the companies so that successful implementation of these quality models can be achieved.

4.2

Development System

of

the

Proposed

SQAMSI

Advisory

A prototype of the proposed system is developed using component object modeling (COM) technology. A number of COM-compliant software packages are used to develop the proposed system. The Visual Rule Studio® is used to develop the expert system component (http://www.RuleMachines.com). Microsoft® Access 2003 is used to build the assessment documents database module. The Microsoft® Visual Basic 6.0 is used to develop the system’s user interface, and to provide the shell for the COM integration. The proposed system is developed as three-tire architecture. This software industry standard architecture provides a framework for logical components of the software to interact and enables flexibility in managing changes and updates in system components. This architecture consists of three layers as follows: Presentation

Layer (Client-tier) -- This tier is

responsible for the

presentation of data, receiving user events and controlling the user interface. Business Layer (Application-server-tier) -- This tier contains a number of

software modules/components that use the user interface data to perform required tasks. Database Layer (Data-server-tier) -- This tier is responsible for data storage

(Models Guidelines, Assessment Documents and Reports, Improvement Recommendations, etc.). Figure 4.1 schematically shows the three-tier architecture of the proposed system

and

the

following

sections

components/modules of the proposed system.

79

describe

the

different

software


4.3 Expert System Module A prototype expert system is developed to assist the decision maker in selecting and implementing the more suitable SQA model.

User Interface Presentation Layer

SQA models selection ES

ISO ES

CMMI ES

SPICE ES

Business Layer

ISO

CMMI

SPICE

Database Layer

COM Figure 4.1: The three-tier architecture of the proposed system. 4.3.1 Knowledge Acquisition Knowledge acquisition is a key point in building a successful expert system. Acquiring the knowledge needed for an expert system and structuring that knowledge into a usable form is one of the main tasks in expert system development. Knowledge in an expert system can originate from many sources, such as previous research, reports, databases, case studies, empirical data, and personal experience. In this work three main sources of knowledge have been used; domain experts, models documents and reports, and available literature. The process of knowledge acquisition is described as follows: 1. Define the task domain of the system. In this system, the selection criteria of the more suitable SQA model for a specific organization are defined which are

80

Chapter Four: A knowledge-based Expert System for SQA Models Selection and Implementation extracted from the characteristics in the comparative study in this research. Also the SQA model implementation guidelines are defined from the models documents and reports. 2. Review published material about the domain of quality management and expert system from 1991 until 2009. 3. Get the facts, documents, guidelines, and technical reports about the available SQA models and approaches. 4. Determine the knowledge representation model. In this system, as the problem is well defined, and the knowledge is available in the form of recommendations, rule-based knowledge representation (production rule system) is selected. 5. Construct the rules. 6. Build a prototype system using an expert system development environment. Visual Rule Studio, an object-oriented expert system development environment for windows, is used to develop the proposed expert system as an ActiveX DLL component to achieve the required software interoperability. 7. Test, analyze, expand and redesign this system. 4.3.2 Expert system building tool ®

Visual Rule Studio (VRS) is used to develop the proposed prototype expert system. It is an object-oriented expert system development environment for windows. The Visual Rule Studio’s object-oriented rules technology is a new adaptation of rulebased expert system technology. It is based on the Production Rule Language (PRL) and the Inference Engines of LEVEL5 Object®. The rich PRL Language and powerful Inference Engines of VRS are compatible in language representation and engine behavior to LEVEL5 Object. With the use of Visual Rule Studio, it is no longer necessary that an entire application be developed wholly within a proprietary expert system development tool to realize the benefits of rules programming and knowledge management (KM) applications. By isolating application rules as component objects, separate from application objects and application logic (as shown in Figure 4.2), VRS allows analysts and developers to leverage the proven productivity of today’s component-oriented

81

Chapter Four: A knowledge-based Expert System for SQA Models Selection and Implementation development tools, such as Visual Basic. With VRS, rule development becomes a natural part of the component architecture development process. The complex and time consuming problems of integrating multiple development tools and managing incompatible object models are issues of the past.

Client User Interface

Application Logic

Application Objects

Application Rules

Back End Database

Figure 4.2: Rules as part of the component development process [adapted from 97]. Visual Rule Studio solves the problem of software interoperability by allowing the developers to package rules into component reusable objects called RuleSets. By fully utilizing OLE and COM technologies, RuleSets act as COM automation servers, exposing RuleSet objects in a natural COM fashion to any COM compatible client. RuleSets are presented not by a static, cumbersome API, but are exposed as COM automation objects derived from the objects created within each individual RuleSet. Thus, each RuleSet exposes a unique object signature, which directly models the object behavior of the specific RuleSet. Visual Rule Studio installs as an integral part of MS Visual Basic 6.0, Professional or Enterprise Editions, and appears within the visual Basic as an ActiveX Designer as shown in Figure 4.3. This allows the developers to add rule objects to their existing or new Visual Basic application in much the same manner they would extend their application with a new form or ActiveX control. RuleSets can be complied within MS Visual Basic. EXE, .OCX, or .DLL executables and used in any of the ways the developers normally use such executables [97].

82

Chapter Four: A knowledge-based Expert System for SQA Models Selection and Implementation 4.3.3 Architecture of the proposed expert system As the problem of SQA models selection and implementation is well defined, and the knowledge is available in the form of selection criteria, recommendations, and assessment guidelines; the rule-based knowledge representation technique (Visual Rule Studio‘s production rule system) is selected to implement the proposed prototype expert system (SQAMSI). The architecture of Visual Rule Studio‘s production rule system is comprised of five primary components: 1. Data -- The facts and values of the knowledge problem, 2. Rules --The If Condition-Then Action representation of the knowledge, 3. Inference Engine --The underlying executor that matches data and rules, 4. Knowledge Representation Language -- The representation grammar for data and rules, and 5. Knowledge Component Objects --The packaging of rules and data into a component reusable entity for use by an application. A Visual Rule Studio package of data, rules, and inference engine is called a RuleSet.

Figure 4.3: Visual rule studio ruleset designer.

83

Chapter Four: A knowledge-based Expert System for SQA Models Selection and Implementation 4.3.3.1 Production system data The data of a knowledge problem e.g., defining the SQA model or approach selection criteria can be considered the inputs and outputs for the set of rules used in solving the problem. Visual Rule Studio models the problem domain data as objects, which represent the things and events of the problem. The set of all data objects and their current values is collectively referred to as the context. Data values are frequently generated through the following channels: 1. The data is provided to the RuleSet by the activating application. 2. The data is changed or created by the rules of the RuleSet. 3. The RuleSet requests the data in an ad hoc fashion from either the controlling application or an interactive user. 4.3.3.2 Production system rules Rules in a production system consist of a collection of If Condition-Then Action statements. Each rule has a left-hand-side, or IF part, and a right-hand-side, or THEN part. The IF part of a rule comprises the conditions or antecedents of the rule. The THEN part is the action part of a rule and is often called the rule’s consequent or conclusion. The IF part of a rule is comprised of one or more clauses, or expressions, combined with the Boolean operators AND, OR, and NOT. An expression or antecedent usually is a restriction, or relational expression, on some value or attribute of some object of the context data. The object may, for example, be an object of the problem domain such as an industry type or criteria group, or internal variable used for directing problem solution strategy, such as a goal. Rules in a production system are thought of as having direction. Direction being implied by whether the IF part or the THEN part is the means by which the rule becomes active. Rules that are activated by the IF part are said to be data driven or forward chaining rules and rules that are activated by the THEN part are said to be goal-driven or backward chaining rules. The rules of the proposed system (SQAMSI) are forward chaining rules. The following are some examples of the proposed system production rules: Rule SixSigma If (SQA.CmpLoc = "europe" Or SQA.CmpLoc = "usa" Or SQA.CmpLoc =

84

Chapter Four: A knowledge-based Expert System for SQA Models Selection and Implementation "japan" Or SQA.CmpLoc = "canada" Or SQA.CmpLoc = "world") And (SQA.CusLoc = "europe" Or SQA.CusLoc = "usa" Or SQA.CusLoc = "japan" Or SQA.CusLoc = "canada" Or SQA.CusLoc = "world") And (SQA.CmpScope = "orgimprvmnt" Or SQA.CmpScope = "swdvlpmntprcs") And(SQA.CmpAim = "prcsimprvmnt") And(SQA.CmpSize = "medium" Or SQA.CmpSize = "large" ) And (SQA.EstimCost = "moderate" Or SQA.EstimCost = "high") And (SQA.QultPrsp = "customer") And (SQA.Coherence = "internalandexternal") And (SQA.Assessment = "prcsmturty") And (SQA.AssResp = "internalandexternal") And (SQA.Progression = "prgstaged") And (SQA.AnlysTech = "statistical") And (SQA.ImprvFocus = "mngmntprcs") And (SQA.ImprvInitiat = "topdown") And (SQA.ImprvPeopl = "mngcustempl") Then SQA.Result:= "The Six-Sigma Approach" Rule 2_1_1_P [SPICE Standard] If exp_advice.level_subitem_id=" 2.1" And exp_advice.question_id=1 And exp_advice.user_ans ="PARTIALLY" Then exp_advice.advice := " Adequate Resources (including people) should be allocated for performing the process. The company should manage resources and schedule. " And exp_advice.eva :=" The Practice is Partially Implemented ! There are no adequate Resources allocated for performing the process." And exp_advice.jump :=0 4.3.3.3 Inference Engine The inference engine of VRS’s production system acts as the "Unseen Hand" or executor which causes processing to take place. Processing here is defined as the combining of supplied data with rules to create inferred data. It is the inferred data that is the desired end result of the production system processing.

85

Chapter Four: A knowledge-based Expert System for SQA Models Selection and Implementation In a VRS RuleSet, control is based on cyclical reevaluation of the state of the context, not on any explicit ordering of instructions. There is no explicit transfer of execution control between rules as there is in traditional program statements. Rules communicate with each other only by way of the data values of the context. This complete separation of execution flow, provided by the executor, from problem knowledge represented as rules, is the key to the representational and processing power of production systems. The author of a RuleSet need only be concerned with the representation of knowledge explicitly as rules, constructing the small units or steps one would naturally use when solving complex problems. The process of choosing, ordering and connecting the steps for problem solution is carried out by the executor, or inference engine. The Visual Rule Studio inference engine provides two primary problem-solving engines relevant to production systems; the forward chaining engine and the backward chaining engine. In the proposed SQA expert system forward chaining engine is used. Forward chaining systems (see figure 4.4) process from initial information that is provided to a final state or goal. Starting from an initial or current set of data, the forward chaining inference engine makes a chain of inferences until a goal is reached. In forward chaining the data values of the context are matched against the IF parts, or left-hand-sides, of rules. If a rule’s IF side matches the context, then the inference engine executes the Then part, or right-hand-side of the rule. If the execution of the Then part of a rule changes the data values of the context, then the inference engine repeats the entire match-execute cycle again using the new state of the context data values as a new initial set of data.

New or Changed Value? Match Rules

Fire Rules

Figure 4.4: Forward chaining inference processing [adapted from 97].

86

Chapter Four: A knowledge-based Expert System for SQA Models Selection and Implementation 4.3.3.4 Knowledge Representation Language The rule language of Visual Rule Studio, known as production rule language or PRL, is a high-level grammar for business problem representation and abstraction designed specifically for the specification and processing of business rules. A ruleset may contain class declarations and methods, forward-chaining rules, backwardchaining demons, and an agenda. Thus the major components of a production system language are: Objects in a knowledge base are created via Class declarations. Class declarations define the structure of the objects contained in the knowledge base. As instances, classes hold and retain the application’s data values. An object’s structure is defined by its class and attributes declarations. Rules and Demons describe the operational logic, rules-of-thumb, and cause-andeffect relationships needed to make decisions and trigger certain events or actions during processing. An agenda schedules the main events and the sequences of events or procedures in pursuit of goals. The grammar of the PRL uses an object-referencing notation that is the same as that of all popular language environments, such as, C++, Java, and Visual Basic. This familiar notation not only facilitates ease of learning, but also provides for the easy mapping of client business objects to the objects of VRS rulesets. VRS objects are used to encapsulate knowledge structure, procedures, and values. An object’s structure is defined by its class and attributes declarations within a RuleSet. Object behavior is tightly bound to attributes in the form of facets, methods, rules, and demons. The breakdown of a VRS object’s structure is outlined in Figure 4.5. The knowledge base of this system consists of four different RuleSets. The Selection Criteria RuleSet, the ISO 9000-3 RuleSet, the CMMI RuleSet, and the SPICE RuleSet. Figure 4.6 gives a typical example of the class and rules used in the SPICE RuleSet. Each one of these RuleSets represents a separate knowledge source (KS). These knowledge sources are independent chunks of knowledge and do not directly communicate with each other.

87


Figure 4.5: The structure of a Visual Rule Studio object [adapted from 97]. Instead, they participate in the problem solving process by writing messages on a global database called blackboard and reading messages from other knowledge sources. This type of architecture is called blackboard architecture and is shown in figure 4.7.

Figure 4.6: A fragment of the RuleSet (the class and rules) of the proposed system.

88

Chapter Four: A knowledge-based Expert System for SQA Models Selection and Implementation The blackboard architecture is intended to support development of systems in domains characterized by interaction between diverse sources of knowledge and hence provides a framework for integrating knowledge from several sources. The blackboard serves as a global data structure, which facilitate this interaction. Usually, in typical blackboard architecture, the inference mechanism consists of the agenda and the monitor. The agenda keeps track of all events in the blackboard and calculates the priority of execution for KSs that are generated as a result of the activation of other KSs. The monitor takes the element with the highest priority and executes it. However, there is no fixed agenda and monitor in the current blackboard architecture. Knowledge Engineer

Human Domain Experts

Knowledge from Documents

Windows Environment Visual Basic Programming Environment Virtual Rule Studio

Inference Mechanism Forward Chaining

Knowledge Bases Selection Criteria

ActiveX Data Object (ADO)

ISO CMMI SPICE

User Interfaces

Microsoft Access Databases (ISO) (CMMI) (SPICE)

COM

User Figure 4.7: Blackboard architecture of the SQAMSI system. Since different solution steps of this system are explicitly seen on the main screen display, the sequences of the different processes are primarily selected by the 89

Chapter Four: A knowledge-based Expert System for SQA Models Selection and Implementation user. Without fixed agenda, the user is free to change input data and check intermediate results given by the system during the consultation session [52]. For a problem solving RuleSet to contribute effectively to an application it must be packaged and made available to the tools being used for creating applications. A VRS RuleSet packages problem solving rules by means of the industry standard COM. RuleSets based applications and components can be deployed in one of four main ways: 1. Deploying as a stand-alone, distributable EXE application. 2. As a stand-alone, distributable OCX or DLL component. 3. As an Internet web server component. 4. As a server component of a client server application. The first two deployment options require that the VRS Runtime Libraries be distributed along with your EXE, OCX, or DLL executable. This is appropriate when the knowledge of the RuleSets is relatively static. When RuleSets are updated, the revised application or component would have to be re-distributed to each user. The proposed expert system’s RuleSet is deployed as an ActiveX DLL component. The following is an outline of the steps followed to develop and deploy RuleSet as an ActiveX DLL component (Figure 4.8): 1. Creating an ActiveX DLL Visual Basic project. 2. Adding a RuleSet to the project. 3. Setting build options for the project. 4. Creating an installation task for the component. 5. Deploying the component.

4.4 Database Module Microsoft® ActiveX® Data Objects (ADO) is the Microsoft strategic application-level programming interface to data and information. ADO provides consistent, high-performance access to data and supports a variety of development needs, including the creation of front-end database clients and middle-tier business objects, using applications, tools, languages or Internet browsers. ADO is designed to be the one data interface needed for one-to-multitier client/server and Web-based data-driven solution development [98].

90


Figure 4.8: The ActiveX DLL VB project of the proposed ES component. Microsoft® Access 2003 is used to develop the database module. This module contains three different databases, the first one contains the questions and the assessment results of ISO 9000-3, the second one contains the questions and the assessment results for the CMMI model and the third one contains the questions and the assessment results of the SPICE standard. ADO is designed as an easy-to-use application level interface to Microsoft’s newest and most powerful data access paradigm, OLE DB. OLE DB is Microsoft’s strategic system-level programming interface to data across the organization. OLE DB is a set of COM interfaces that provide applications with uniform access to data stored in diverse information sources, regardless of location or type. These interfaces enable the creation of software components that implement such services. OLE DB components consist of data providers, which contain and expose data; data consumers, which use data; and service components, which process and transport data (such as query processors and cursor engines). To make the advantages of OLE DB available to higher- level development environments, Microsoft has implemented an Automation-based object model called ADO [98, 99]. To use ADO, the first step is to create a connection to the database. This requires a reference to the name and location of the database file and information on the data provider as shown in the following lines: 91

Chapter Four: A knowledge-based Expert System for SQA Models Selection and Implementation Dim dbcon_spice As Connection Set dbcon_ spice = New Connection dbcon_ spice.Open"PROVIDER=Microsoft.Jet.OLEDB.4.0; Data Source= spice.mdb The first line of the above code declares a Connection object where dbcon_ spice is name of the Connection object (any generic name could be used), and the next few lines of the above code illustrate how the Connection String and Provider properties and the Open method of the connection object are used to open the connection. The Connection String property must begin with "Data Source=" and be followed by the path and name of the database file. The provider property is the name of data provider. In this research a personal database is used which is implemented as a Microsoft Access database so the Microsoft Jet 4.0 OLE DB Provider is used. The Open method opens the connection once it is no longer needed. Once a connection is formed, the next step is usually to open a recordset based on the database. A recordset is a temporary database table (that is, it resides only on the memory). The following lines indicate how to open a recordset: Dim DBRS_ spice As Recordset Set DBRS_ spice = New Recordset DBRS_ spice.Open "SELECT * From users", dbcon_ spice, adOpenStatic, adLockOptimistic The first two arguments of the Open method in the above code are the table name (users) and the connection object that has already been opened (dbcon_ spice). As with a connection object, a Recordset object should be closed with the Close method when it is no longer needed. Then the close method of the connection should be called. After the recordset is opened, it is easy to go through its records with a Do loop and retrieve information from any of its fields. A common practice is to code such as the following: With rs Do Until DBRS_ spice.EOF If (UName.Text = HName.Text And Password.Text = HPassword.Text) Then user_id = HID.Text spice _open_project.Show Unload Me

92

Chapter Four: A knowledge-based Expert System for SQA Models Selection and Implementation Exit Do Else DBRS_ spice.MoveNext End If Loop End With In the above code, rs is a recordset object variable. One of its properties is EOF (end of file), and one of its methods is MoveNext. These lines of code tell the program to go through the records of the recordset, one at a time, taking some actions in the part called Statements, and quitting when the end of the file is reached. The ADO is used in this research in the Implementation Phase of SQAMSI to read the self-assessment projects using the implemented SQA models or approaches and to produce the final assessment and evaluation reports and improvement recommendations. Figure 4.9 shows a fragment of the VBA code written to access the different SQA model's levels, sub items, assessment questions …etc. from the projects, level_subitem, questions tables, … etc. in the database (ISO, CMMI, and SPICE) using ADO and write them in the forms of the assessment questions and evaluation reports.

Figure 4.9: A Fragment of the VBA code written to implement ADO.

93


4.5 SQAMSI User Interface The proposed system’s user interface (as shown in Figure 4.10) is developed using Visual Basic 6.0. The main purpose of this interface is to control and integrate the software modules/ components of the proposed system using COM technology.

Figure 4.10: The proposed system’s user interface.

94

CHAPTER FIVE SYSTEM VALIDATION AND IMPLEMENTATION

Chapter Five

System Validation and Implementation

CHAPTER FIVE SYSTEM VALIDATION AND IMPLEMENTATION This chapter describes the verification and validation process of the proposed system. Also, the chapter presents the results of implementing the proposed system SQAMSI on a real case study, such that the proposed system was tested on the E-SmartSoft Information Systems (A software Development Company in Heliopolis, Cairo, Egypt) The following sections describe the verification and validation, the case study, the data used for testing the proposed system, and present the system implementation based on the proposed SQA models selection and implementation approach that was discussed in chapter four.

5.1 System Verification and Validation Verification determines correctness of the system, i.e. whether the product satisfies the specification standards set at the beginning of the project. In the expert system’s context, verification ensures that the compile time and runtime errors are eliminated. The debugging utilities of the Visual Basic 6.0 and Visual Rule Studio are utilized throughout the development cycle to ensure error-free execution of the system. Individual modules are evaluated individually by executing a series of predetermined test cases [100]. The complete system is then verified to ensure that the integrated modules behaved as expected. Verification aspects common to knowledgebased processing, as noted by Medsker and Liebowitz [101], are also applied. Validation is the process of determining that the system completely and accurately represents the problem domain, and that it achieves acceptable performance levels [102]. Based upon their research methods approach, Ayel and Laurent [103] as well as Libertore and Stylianou [102] present a common procedure for establishing content validity. These guidelines are utilized throughout the development of SQAMSI system. According to Medsker and Leibowitz [101],

95

Chapter Five


validation using independent experts reduces potential bias in the results, and lends credibility to the validation process. Content validity was established through face validation by several external experts. These experts are selected for their in-depth knowledge and experience in the field of software quality assurance, particularly with respect to the ISO 9000, CMMI, and SPICE standards. All of the domain experts consider the system to be satisfactory, and that only some minor modifications are required. Any disparities between the evaluators’ opinions are resolved, and their suggestions for modification are incorporated into the final version. Also, the proposed system is validated by the quality experts in the Software Engineering Competence Center (SECC) - www.secc.org.eg. The Consultation and Assessment Unit Manager go through the proposed system and consider it represents when answering the questions the following: 1. Selection for the different types of process improvement initiative 2. Checking the documents readiness for the appraisal without exploring the contents of these documents, the system can be used as a checklist for preparation for the appraisal.

5.2 Case Study The proposed system SQAMSI is tested on a real case study in the ESmartSoft Information Systems Company (http://www.esmartsoft.com.eg). The E-SmartSoft Company is a software development company. The company vision is becoming the regional leader for: 

Development of readymade software tools.



Offshore development.



Implementation of state-of-the-art software technology. The mission of the E-SmartSoft company is delivering and accumulating the

best practices in: 

Development and packaging of software tools.



Offshore development using IBM rational, IBM rational business developer (EGL), and Java 2EE.

96

Chapter Five 


Data-Management using IBM Informix, IBM enterprise content management, and IBM BI. The E-SmartSoft company achieves Capability Maturity Model Integration

(CMMI) accreditation Maturity level П on 14 March, 2007 from the SECC. Currently, the company registers again in order to get the CMMI Maturity level Ш.

5.3 Test Data for the Case Study The main purpose of the proposed system SQAMSI is to assist the software development organization in selecting the more appropriate SQA model or approach to adopt. Then the proposed system will help these organizations in the implementation process of the selected model (definitely implementation of ISO, CMMI, and SPICE) as a self assessment tool by identifying the GAP (Gauging Absence of Prerequisites) between the prerequisites and what actually exists in the present environment. The different types of data required for testing the proposed system are obtained from several sources such as domain experts, quality models documents, etc. the researcher interviewed several experts from the SECC center and the E-SmartSoft company.

5.4 System Implementation The user starts the implementation of the proposed system SQAMSI from the welcome screen as shown in Figure 5.1, by pressing the Start Program button. The welcome screen explains the program goal to the user. After that the user can start the program by selecting the needed program phase to be implemented from the screen displayed in Figure 5.2. The SQAMSI program contains two phases the selection phase which suggests the appropriate model and the implementation phase which implements the suggested model.

97

Chapter Five


Figure 5.1: The SQAMSI welcome screen.

Figure 5.2: The SQAMSI phase selection.

98

Chapter Five


5.4.1 Selection phase If the user chooses to implement the Selection phase (as shown below in Figure 5.3), he will have about 15 questions to be answered about his organization requirements and needs. The user should answer these questions carefully and according to the organization actual status and requirements in order to help the SQAMSI suggests the more appropriate SQA model or approach.

Figure 5.3: The SQAMSI selection phase. Figures from 5.3 to 5.11 provide a sample of the selection phase questions needed to be answered by the system user. As shown in the following figures of the selection phase question, the SQAMSI program provides the user with an explanation for each question in order to help the system user understands the question and answers it correctly.

99

Chapter Five


Figure 5.4: The customer location question.

Figure 5.5: The company scope question.

100

Chapter Five


Figure 5.6: The company aim question.

Figure 5.7: The company size question.

101

Chapter Five


Figure 5.8: The assessment type question.

Figure 5.9: The progression type question.

102

Chapter Five


Figure 5.10: The improvement focus question.

Figure 5.11: The improvement people question.

103

Chapter Five


When the user finishes answering the selection questions, the system will provide him with the suggested SQA model or approach according to his requirements and needs. The suggested SQA model according to the previous answers in our real case study is shown below in Figure 5.12.

As shown in Figure 5.12 the user can exit the program or start the implementation phase in order to implement the suggested model.

Figure 5.12: The final result of the SQAMSI selection phase.

The Help of the SQAMSI program provides the user with the needed information about the suggested SQA model or approach in order to know its definition, usage, stages, …etc. Figure 5.13 shows the SQAMSI Help Screen which contains the different SQA models and approaches and describes the SPICE standard.

104

Chapter Five


Figure 5.13: The SQAMSI help screen.

5.4.2 Implementation phase The user starts the SQAMSI implementation phase by pressing the start button in the screen shown in Figure 5.14. After that the user chooses the SQA model needed to be implemented from the available models (ISO, CMMI, or SPICE) as shown in Figure 5.15. As the SQAMSI suggests the CMMI model in the selection phase in the previous section, then the user will choose the CMMI model to be implemented in his organization (The E-SmartSoft Company).

105

Chapter Five


Figure 5.14: The SQAMSI implementation phase.

Figure 5.15: The quality model selection.

106

Chapter Five


To implement any of the quality models, the user should create or load his own account by entering the user name and password as shown in Figures 5.16 and 5.17.

Figures 5.16: Loading or creating a new account.

Figures 5.17: Creating the username and password.

107

Chapter Five


After that the user creates or opens his assessment project as shown in Figure 5.18.

Figure 5.18: Loading or creating an assessment project.

Figure 5.19: Choosing the CMMI evaluation style. 108

Chapter Five


Figure 5.20: Choosing the CMMI maturity level. After the user specifies the needed level to be implemented, he should answer the assessment questions according to the actual documents in his organization.

Figure 5.21: A sample of CMMI assessment questions. 109

Chapter Five


After the user answers all the assessment questions, the SQAMSI provides him with the assessment report by pressing the Report button which will appear at the end of the assessment questions as shown below in Figure 5.22.

Figure 5.22: The end of the assessment questions in CMMI Level 2. The Assessment Report which appears in Figure 5.23 provides the user with the SQAMSI evaluation for his company and the appropriate advices and improvement recommendations in case of any shortages. Using the suggested advices the user can improve his company performance and also enhance the company activities and work in order to get the maturity level fully implemented in his company to have the high quality level for his work.

Also, the user gets the assessment and evaluation report in variety ways as shown in Figure 5.23. The user can select the report type he wants to get for his company assessment and evaluation whatever in textual representation (printed

110

Chapter Five


format) or graphical representation. The SQAMSI program contains three types of evaluation reports:

(1) Text Report, (2) Bar Graph Report, and (3) Pie Graph Report.

Figure 5.23: The assessment and evaluation report. If the SQAMSI user chooses to display the Text Report for the assessment and evaluation project, it will appear in a new window, as shown in Figure 5.24.

If the SQAMSI user chooses to display the Bar Graph report for the assessment and evaluation project, it will appear in a new window. Figure 5.25 represents the implementation of CMMI Level 2: Managed for the E-SmartSoft company in the bar chart format.

111

Chapter Five


Figure 5.24: The SQAMSI text assessment and evaluation report.

Figure 5.25: The SQAMSI bar graph assessment report.

112

Chapter Five


If the user of SQAMSI program chooses to display the Pie Graph report for the assessment and evaluation project, it will appear in a new window. Figure 5.26 represents the implementation of CMMI Level 2: Managed for the E-SmartSoft company in the pie chart format.

Figure 5.26: The SQAMSI pie graph assessment report.

5.5 Results and Discussion Applying the case study on the proposed SQAMSI system shows that the proposed system suggests the E-SmartSoft company the CMMI model as the more appropriate quality model to adopt according to its requirements and needs. This result compromises with the actual model implemented in the company. On the other hand, according to the self assessment reports of the E-SmartSoft company on the CMMI Level 2 that resulting from the implementation phase of the proposed system, the E-SmartSoft company can get the CMMI Level 2. This result compromises with the actual status of the company, as it already has the CMMI Level 2 and currently registered in order to get CMMI Level 3. These results achieve the validation and verification of the proposed SQAMSI system.

113

CHAPTER SIX CONCLUSIONS AND RECOMMENDATIONS

Chapter Six

Conclusions and Recommendations.

CHAPTER SIX CONCLUSIONS AND RECOMMENDATIONS This chapter presents a summary of the research findings and its significance. Suggestions for improvement and future work are also discussed.

6.1 Summary and Conclusions There are several models and approaches for software quality assurance, such as TQM, CMMI, ISO/IEC 9001, TRILLIUM, SPICE ... etc. However, the proper selection and implementation of these quality models or approaches is often a difficult, a time consuming and a costly task for software-development organizations especially small and medium ones. This research provides a comparative study between eight quality models and approaches (TQM, CMMI, ISO, SIX-SIGMA, BOOTSTRAP, TRILLIUM, TICKIT, and SPICE) in a textual description illustrating the history, origins, principles, characteristics,

Representation,

advantages,

and

disadvantages

…etc.

using

a proposed comparison framework of thirty characteristics in five categories. The resulting comparative study provides practical insight and guidance when selecting which SQA model or approach to adopt in a software-producing organization. Such guidance is needed because of the multitude, diversity and comprehensiveness of existing models and approaches. Using the expert system technology and the resulting SQA models comparative study, the dissertation provides an intelligent system for the software quality assurance models selection and implementation [SQAMSI]. The Component Object Model (COM) technology is successfully used in designing the proposed SQAMSI system to strengthen the software interoperability. 114 Software Quality Assurance Models: A comparative Study_Master thesis.

Chapter Six

Conclusions and Recommendations.

Based on the work carried in this research, the following major conclusions are drawn: 1. A comparison framework is developed which contains thirty characteristics in five categories. In this aspect five characteristics are added in order to provide a more general comparative study which covers most of the quality models aspects. 2. A comparative study is conducted between eight selected models using the proposed framework. 3. A prototype knowledge-based system is developed for the selection and implementation of SQA models. This system could benefit individuals, organizations, and software companies who desire to implement these quality models. 4. The COM technology is successfully used in designing the system to strengthen the software interoperability. 5. The presented system is considered a prototype because the knowledge base supporting the expert system contains only a limited range of software quality models.

6.2 Recommendations This study is regarded as the first step in the long term research agenda of the researcher in the area of SQA models selection and implementation. Therefore, suggestion and recommendations for improvement and issues for future work can include the following research topics: 1. Expanding the comparative study to include more SQA models or approaches. 2. Modifying the proposed system to include more SQA models.

115 Software Quality Assurance Models: A comparative Study_Master thesis.

REFERENCES

References

REFERENCES [1] Ur Rehman, A., (2008), "Quality Cost Analysis", Feditec Enterprise. [2] Fitzpatrick, R., (1996), Software Quality: Definitions and Strategic Issues, MSc Computing Science (ITSM), Advanced Research Module. [3] Kan, S. H., (2002), Metrics and Models in Software Quality Engineering, Second Edition, Addison Wesley. [4] Ashrafi, N., (2003), "The Impact of Software Process Improvement on Quality: in Theory and Practice", the journal of Information and Management, Vol. 40, pp 677–690. [5] Rosenberg, L. H., (2002), "What is Software Quality Assurance", NASA, Available at http://www.stsc.hill.af.mil/crosstalk/2002/05/rosenberg.html Accessed in October 25, 2007 at 3:10 pm. [6] Lewis, W. E., (2005), Software Testing and Continuous Quality Improvement, Second Edition, Auerbach Publications. [7] Frankovisch, J., and Pan D., (1997), "Software Quality Assurance", available at http://sern.ucalgary.ca/courses/seng/621/W97/johnf/sqa-deta.htm Accessed in December 15, 2006 at 12:20 pm. [8] Marchewka, J. T., (2006), Information Technology Project Management, second edition, John Wiley and Sons. [9] Hari, S., and Sanne, S., (2004), "A Statistical Approach for Comparing TickIT and CMM", TickIT International 1Q04, Firm Focus on behalf of BSI. [10] Liao, H., Enke, D., and Wiebe, H., (2004), "An Expert Advisory System for ISO 9001 Quality System", Expert Systems with Applications Journal, vol. 27, pp. 313 -322. [11] Dumke, R., Braungarten, R., Blazey, M., Hegewald, H., Reitz, D., and Richter, K., (2006), "Software Process Measurement and Control", Faculty of Computer Science, Otto von Guericke University Magdeburg. [12] Niessink, F., (2000), Perspectives on Improving Software Maintenance, SIKS Dissertation Series No. 2000-1 [13] Wang, Y., and Bryant, A., (2002), Process-Based Software Engineering, Annals of Software Engineering, Volume 14. [14] Rubey, R. J., and Brewer, A. C., (1991), "Software Quality Assurance StandardsA Comparison and An Integration", SofTech Inc., Fairborn, OH; IEEE May. [15] Crane, M. L., (2002), Introduction to Software Standards, CFICSE 2002, ST01. [16] Bernhart, M., (2007), "Software Quality Assurance Standards", INSO - Industrial Software.

116

References [17] Herndon, M. A., and Salars, S., (2005), Two Case Studies (Journeys) in Implementing Model Based Process Improvement in Small Organizations, International Research Workshop for Process Improvement in Small Settings, Pittsburgh, PA. [18] Siviy, J., Penn, M. L. and Harper, E., (2005), "Relationships between CMMI® and Six Sigma", Software Engineering Measurement and Analysis, CMU/SEI2005-TN-005. [19] Zhang, Z., (2001), "Implementation of Total Quality Management, An Empirical Study of Chinese Manufacturing Firms", Dissertations, University of Groningen. [20] Carnegie Mellon University Software Engineering Institute CMU-SEI, (2006), "CMMI® for Development, Version 1.2", CMU/SEI-2006-TR-008, ESC-TR2006-008, Pittsburgh, USA. [21] Tantara, (2009), Available at http://www.tantara.ab.ca/ Accessed in February 15, 2009 at 10:35 am. [22] ISO (2009), "Selection and Use of the ISO 9000 Family of Standards", ISO Press. [23] Benbow, D. W., and Kubiak, T.M., (2005), The Certified Six Sigma Black Belt Handbook, ASQ Quality Press. [24] Albeanu, G., Thyregod, P., Madsen, H., and Popentiu-Vladicescu, F., (2006), "On Using Six Sigma Methodology for Software Quality Assurance", the sixth Annual ENBIS Conference in Wroclaw, Poland. [25] Koch, G. R., (1993), "Process Assessment: the BOOTSTRAP Approach", Information and Software Technology, vol. 35, no 6/7, pp. 387-403. [26] ESSI-SCOPE, (2009), "Software Process Improvement Approaches", available at http://www.cse.dcu.ie/essiscope/sm5/approach.html Accessed in May 21, 2009 at 2:40 pm. [27] Haase, V.H., (1996), "Software Process Assessment Concepts", Journal of Systems Architecture, Vol. 42, pp. 621-631. [28] Bell Canada, (1994), "Trillium Model for Telecom Product Development and Support Process Capability", Trillium Release 3.0. [29] Coallier, F., (1995), "TRILLIUM: A model for the Assessment of Telecom Product, Development and Support Capability", IEEE Computer Society TCSE, Software Process Newsletter: No. 2. [30] Raftus, M., (1999), "Audit and Assessment Methodologies", GRafP Technologies inc., Ottawa SPIN, Meeting November 18. [31] SPICE, (1995), "ISO/IEC Software Process Assessment", SPICE Working Draft V1.00. [32] Halvorsen, C. P., and Conradi, R., (2002) "A Taxiomatic Attempt at Comparing SPI Frameworks", Springer Berlin / Heidelberg. 117

References [33] Sørumgård, S., (1997), "Verification of Process Conformance in Empirical Studies of Software Development", Doctoral thesis 1997:14, The Norwegian University of Science and Technology. [34] Sheard, S. A., (2001), "Evolution of the Frameworks Quagmire ", IEEE Computer, Volume 34, Issue 7, Page(s): 96 – 98, July. [35] Cattaneo, F.; Fuggetta, A., and Sciuto, D., (2001), "Pursuing Coherence in Software Process Assessment and Improvement Research Section", Software Process Improvement and Practice, Vol. 6, pp 3-22. [36] Tingey, M. O., (1997), "Comparing ISO 9000, Malcolm Baldrige, and the SEI CMM for Software: A Reference and Selection Guide", Prentice Hall. [37] Wang, Y.; Court, I.; ROSS, M.; King, G. and Dorling, A., (1997), "Quantitative Evaluation of the SPICE, CMM, ISO 9000 and BOOTSTRAP", IEEE. [38] Van der Piji, G.J., Swinkels, G.J.R, and Verrijdt, J.G., (1997), "ISO 9000 versus CMM: Standardization and Certification of IS Development", Information and Management Journal, Vol. 32, pp. 267-274. [39] Kuilboer, J. P., and Ashrafi, A., (2000), "Software Process and Product Improvement: An Empirical Assessment." Information and Software Technology, vol. 42, pp. 27-34. [40] Williams, D., and Kennedy, M., (2000), "A Framework for Improving the Requirements Engineering Process Effectiveness", Software Quality Journal. [41] Paulk, M. C., (1995) "How ISO 9001 Compares with the CMM", 0740-7459/94 IEEE software, pp.74-83, January. [42] Bamford, R. C., and Deibler, W. J., (1993), "Comparing, Contrasting ISO 9001 and the SEI Capability Maturity Model", IEEE Computers, Vol. 26, No. 10, pp 68-70, October. [43] Paulk, M. C., (1994), "A Comparison of ISO 9001 and the Capability Maturity Model for Software", Technical Report, CMU/SEI-94-TR-12, ESC-TR-94-12, SEI, Pittsburgh, July. [44] Oskarsson, Ö., and Glass, R. L., (1995) "An ISO 9000 Approach to Building Quality Software", Prentice Hall. [45] Saiedian, H., and Mcclanahan, L. M., (1996), "Frameworks for Quality Software Process: SEI Capability Maturity Model versus ISO 9000", Software Quality Journal vol. 5, pp. 1-23. [46] Messnarz, R., (1997), "A comparison of BOOTSTRAP and SPICE", IEEE Computer Society TCSE, Software Process Newsletter No. 8, winter. [47] CEO Issue Sheet, (2002), "Baldrige, Six Sigma, and ISO: Understanding Your Options", Baldrige National Quality Program, NIST, Technology Administration, U.S DoC, summer.

118

References [48] Bishop, B., (2008), "Why ISO 9000 Should Be A Company's Guidepost, And Not TQM", available at 2009 at 11:20 am.

http://tqmcintltqm.blogspot.com/ Accessed in May 21,

[49] Vasiljevic, D., and Skoog, S., (2003), "SPI Framework for Small Organizations", Thesis, Software Engineering, Thesis no: MSE-2003:28, Blekinge Institute of Technology, Sweden, August. [50] Yoo, C., Yoon, J., Lee, B., Lee, C., Lee, J., Hyun, S., and Wu, C., (2006), "A Unified Model for the Implementation of both ISO 9001:2000 and CMMI by ISO-Certified Organizations", the Journal of Systems and Software, vol. 79, pp. 954–961. [51] Dey, P., (2008), "How to Complement ISO 9001:2000 with Six Sigma", EXCEL Partnership, Inc., USA. [52] Eldrandaly, K., (2008), "A Knowledge-Based Advisory System for Software Quality Assurance", The International Arab Journal of Information Technology, vol. 5, no. 3, pp. 304 -310. [53] Crossfield, R. T., and Dale, B. G., (1991), "The Use of Expert Systems in Total Quality Management: An Exploratory Study", Quality and Reliability Engineering International, vol. 7, pp. 19-26. [54] Cheng, L. L., Tsang, W. M., and Yeung, K. L., (1993), "An Expert System for ISO 9000 Quality Management System", IEEE TENCON (Bei jing, Hong Kong Productivity Council), vol.2, pp. 726-729. [55] Chan, F. Y., Willborn, W., Xiao, H., and Li, H. (1996), " Research Report: An Expert System for Quality Management Auditing", International Journal of Quality and Reliability Management, vol. 10, issue 6. [56] Oppermann, R., Reiterer, H., (1997), "Software Evaluation using the 9241 Evaluator", Behaviour and Information Technology, Vol. 16, No. 4/5, pp. 232 – 245. [57] Bayraktar, D., (1998), "A Knowledge-based Expert System Approach for the Auditing Process of Some Elements in the Quality Assurance System", International Journal of Production Economics, vol. 56/57, pp. 37 – 46. [58] Khan, M. K., and Hafiz, N., (1999), "Development of an Expert System for Implementation of ISO 9000 Quality Systems", Total Quality Management, Vol. 10, No. 1, pp. 47 – 59. [59] Paladini, E. P. (2000), "An expert system approach to quality control", Expert Systems with Applications, Volume 18, Issue 2, Pages 133-151.

119

References [60] Maroukhine, O. V., and Berestneva, O. G., (2003), "Expert Support System for Making Decision by The Results of Computer-Based Testing within the Ends of Teaching Quality Evaluation", IEEE, The 7th Korea-Russia International Symposium on 28 June-6 July, vol. 2, pp. 416 – 419. [61] Herrera, E. M., and Ramirez, R. A., (2003), "A Methodology for Self-Diagnosis for Software Quality Assurance in Small and Medium-Sized Industries in Latin America", EJISDC, vol.15 (4), pp.1-13. [62] Reznik, L., and Dabke, K. P., (2004), "Measurement Models: Application of Intelligent Methods", Measurement Journal, vol. 35, pp. 47– 58. [63] Goldberg, Y., and Shmilovici, A., (2005), "An Expert System Approach for Quality Assurance Auditing", International Journal of Advanced Manufacturing Technology, vol. 26, pp. 415 – 419. [64] Keller, N. L., and Grif, M.G. (2005), "The Expert System for Quality Management in Higher Educational Establishments Based on ISO 9000: an Outline of the Knowledgebase Structure", IEEE, Science and Technology, KORUS, Proceedings. The 9th Russian-Korean International Symposium on 26 June-2 July, pp. 1084 – 1086. [65] Lau, H. C. W., Ho, G. T. S., Chu, K. F., Ho, W., and Lee, C. K. M., (2009), "Development of an Intelligent Quality Management System using Fuzzy Association Rules", Expert Systems with Applications, Vol. 36, Issue 2, Part 1, pp. 1801-1815. [66] Cater-Steel, A. P., (2004), "An Evaluation of Software Development Practice and Assessment-Based Process Improvement in Small Software Development Firms", Ph.D Dissertation, Faculty of engineering and Information Technology, Griffith University, December. [67] Kenett, R. S., and Baker E. R., (1999), "Software Process Quality-Management and Control", Marcel Dekker, Inc. [68] Patton, R., (2005), Software Testing, 2nd edition, Sams Publishing. [69] Arya, A., (2003), "Software Project Management, Process Improvement", Available at www.raminsoftworx.com/elec443/lectures/improve.pdf Accessed in August 17, 2008 at 2:35 pm. [70] Brecker Associates, (2001), "TQM Tools". http://www.brecker.com/quality.htm Accessed in January 25, 2009 at 4:15 pm. [71] Peleska, J., and Zahlten, C., (2008), "TQM Pitfalls - and What to do About them". Available at http://tqmcintltqm.blogspot.com/ Accessed in November 2, 2008 at 1:45 pm. [72] Bhote, K. R., (2003), The Power of Ultimate Six Sigma, AMACOM, USA.

120

References [73] Carnegie Mellon University Software Engineering Institute CMU-SEI, (2002), "CMMISM for Software Engineering" (CMMI-SW, V1.1: Staged Representation). Pittsburgh, USA. [74] Browning, T. R., Fricke E., and Negele, H., (2006), "Key Concepts in Modeling Product Development Processes", Wiley Periodicals, Inc., System Engineering, vol. 9, pp. 104–128. [75] Sandahl, k., (2007), "Software Quality Management", Department of Computer and Information Science, Linköping University, Sweden. [76] May, D., (1997), "ISO 9000-3 and Compliance Risks for Organizations", SWSE 625, April 21. [77] Isixsigma, (2009), "The Benefits of ISO Certification", Available at http://www.isixsigma.com/library/content/c000917b.asp Accessed in May 23, 2009 at 3:15 pm. [78] Six Sigma Qualtec, (2009), "Origins of Six Sigma", Six Sigma Qualtec Inc., Available at http://www.ssqi.com/six-sigma-library/origins.html Accessed in May 21, 2009 at 9:30 am. [79] Eckes, G., (2003), Six Sigma for Every One, John Wiley and Sons, Inc., Hoboken, New Jersey. [80] Motorola, (2004), "Six-Sigma Lessons", Motorola University, Motorola Inc, US. [81] Pande, P. S. , Neuman, R. P., and Cavanagh, R. R., (2000), The Six Sigma WayHow GE, Motorola and other Top Companies are Honing their Performances, McGraw-Hill, New York, USA. [82] Lopez-Ona, J., (2006), "Successful Six Sigma: The Importance of Leadership Involvement", A white paper, Six Sigma Qualtec. [83] Slavich, J., (2004), "Six Sigma Implementation Process for Engineering Processes", SPI LINK Ottawa, Spin Inc. [84] Bush, M., and Dunaway, D., (2005), CMMI® Assessments: Motivating Positive Change, Addison Wesley Professional. [85] Sadauskas, L., (2002), "Survey of Candidate Organizational Capability Assessment and improvement Methods and Tools for Employment in RiskBalanced Management", DoD Information Technology (IT/NSS) Acquisition Management Transformation Rapid Improvement Team (RIT), 27 September. [86] Kuvaja, P., (1999), "BOOTSTRAP 3.0- A SPICE1 Conformant Software Process Assessment Methodology", Software Quality Journal, vol. 8(1), September. [87] Järvinen, J., (2000), "Measurement based Continuous Assessment of Software Engineering Processes", ESPOO, Technical Research center of Finland, VTT Publications 426. [88] Harjumaa, L., (2005), "Improving the Software Inspection Process with Patterns", University of Oulu, Finland. [89] Robben, D., Sogeti Nederland, B. V., and The Netherlands, (2000), "TPI, BOOTSTRAP and Testing", V.1., January 26, Sogeti Nederland B.V., Vianen, The Netherlands. 121

References [90] Stienen, H., Engelmann, F., and Lebsanft, E., (1997), "Bootstrap: Five Years of Assessment Experience," step, pp.371, 8th International Workshop on Software Technology and Engineering Practice (STEP '97) (including CASE '97). [91] TickIT, (1992), "Guide to Software Quality Management System Construction and Certification Using ISO 9001", EN 29001/BS 5750, TickIT Project Office. [92] TickIT, (2009), Available at http://www.tickit.org/ Accessed in April 17, 2009 at 7:15 pm. [93] Drouin, J-N., (1995), "The SPICE Project: An Overview", IEEE computer society TCSE, SPN: No. 2. [94] Hoffman, D., (2004), "Software Quality Methods ", CSQE exam preparation, LLC. [95] El Emam, K., Drouin, J-N., and Melo, W., (1998), "SPICE: The Theory and Practice of Software Process Improvement", IEEE Computer Society Press. [96] Mathew, P. K., (2001), "SPICE is an effective model for small companies". Available at http://www.ciol.com/content/news/interviews/101021901.asp Accessed in October 27, 2008 at 11:15 am. [97] RuleMachines, (2002), "Visual Rule Studio Developer's Guide", Canada: On Demand Manuals. [98] Microsoft, (2003), Microsoft Developer Network Online Documentation (MSDN Library). [99] Eddon, G., and Eddon, H., (1998), "Programming Components with Microsoft Visual Basic 6.0", Redmond, Microsoft press. [100] Laudon, K., and Laudon, J., (2001), Management Information Systems: New Approaches to Organization and Technology, Upper Saddle River, NJ: Prentice Hall. [101] Medsker, L., and Liebowitz, J., (1994), Design and Development of Expert Systems and Neural Networks, New York: Macmillan [102] Libertore, M., and Stylianou, A., (1993), "The Development Manager’s Advisory System: A Knowledge Based DSS Tool for Project Assessment", Decision Sciences, vol. 24 (3), pp. 953–976. [103] Ayel, M., and Laurent, J. (1991), Validation, Verification and Test of Knowledge-Based Systems, John Wiley and Sons, Chichester, England. [104] IEEE Std 730™ (2002), "IEEE Standard for Software Quality Assurance Plans", IEEE computer society, sponsored by the software engineering standards committee. [105] ESI (1996), SPICE Relationships with Other Standards and Models, European Software Institute SPICE-TI-95-44/01. [106] Fehlmann, T. M., (2009), “Six Sigma for Software”, Available at http://www.ep-o.com/Bibliographie/SixSigmaForSoftware.pdf Accessed in October 14, 2009 at 3:15 am.

122

SQA Models: A comparative Study

SQA Models: A comparative Study

Suggest Documents

A comparative study of thermochemistry models for

Software Quality Models: A Comparative Study

Software Quality Models: A Comparative Study

Software Quality Models: A Comparative Study

A Comparative Study of Stochastic Volatility Models

A comparative study of cavitation models in a Venturi

Comparative Study of Risk Assessment Models ...

Comparative Study of Bankruptcy Prediction Models

Comparative Study of Software Quality Models

A comparative study of models for predation and parasitism

a comparative study of software engineering process models for ...

A Comparative Study of Data Mining Process Models ...

a comparative study on different statistical models for calculating the

A Comparative Study of Power-based Health Assessment Models

A comparative study of models for the incident duration prediction ...

Comparative study of traffic models: a concrete mass evacuation ...

A comparative study of survival models for breast cancer ...

A comparative study of two trunk biomechanical models ... - IRSST

A Comparative Study of SVM Models for Learning ... - WSEAS

A Comparative Study of Logistic Models Using an ...

Time series forecasting models: A comparative study ...

A Comparative Study to Noise Models and Image

Stochastic Resonance: a Comparative Study of Two-State Models

A Comparative Study of 3D Web Integration Models ...