The Application of a Lean Software Development Methodology within the Regulated Domain of Medical Device Software Oisín Cawley Ph.D. 2013 Supervised by:
Dr. Ita Richardson Dr. Xiaofeng Wang
Submitted to the University of Limerick, November 2013
Confidentiality Agreement The organisations, company documentation, interview excerpts and other sensitive material contained in this thesis have been provided under an agreement of confidentiality and anonymity. Readers are therefore asked not to reproduce this data or use it in any form without the consent of the author.
i
Acknowledgements This thesis was supported through funding from Science Foundation Ireland for the Medi-SPICE project through Lero (the Irish Software Engineering Research Centre). Although a culmination of a number of years of personal dedication, this work would not have been achievable without the support and guidance of some wonderful people. I would like to specifically and sincerely thank them here. Firstly I would like thank my supervisors, Dr. Ita Richardson and Dr. Xiaofeng Wang. To use a nautical analogy, I may have powered the ship but they certainly helped steer. I would specifically like to thank Dr. Richardson for her encouragement, patience, time and judgment throughout. I would also like to gratefully acknowledge the support and camaraderie of my fellow researchers within the Lero research centre, and the wider academic community in both the University of Limerick and the Dundalk Institute of Technology. To my parents Sean and Valarie I say a heartfelt thank you. Dad, your advice was heeded more than you know. Ma, I’m convinced your prayers were answered. You are both irreplaceable. To my brother Ciaran and his wife Clare, both of whom have shared this journey on the way to their own Ph.Ds. Many thanks for the advice and encouragement and the odd glass of vino! The company has helped more than you know. Finally and most importantly, to my wife Geraldine I must express the deepest gratitude and love for her endless patience, encouragement and support. Geraldine, you continue to be my rock and inspiration. I dedicate this Thesis to you and our wonderful children Aoife, Éadaoin, Eóin and Aisling. What a trip! ii
Contents Acknowledgements ....................................................................................................................................... ii Abstract ........................................................................................................................................................... viii Declaration........................................................................................................................................................ix Glossary ............................................................................................................................................................... x List of Tables ................................................................................................................................................. xiii List of Figures .................................................................................................................................................xv Chapter 1: Introduction .............................................................................................................................. 1 1.1
Background ...................................................................................................................................... 1
1.2
Motivation ......................................................................................................................................... 2
1.3
The Problem Area ......................................................................................................................... 3
1.3.1
Medical Devices.................................................................................................................... 3
1.3.2
Medical Device Software ................................................................................................. 5
1.4
Research Problem ......................................................................................................................... 6
1.5
High-level Research Overview ............................................................................................... 9
1.6
Thesis Layout ............................................................................................................................... 10
Chapter 2: Literature Review ................................................................................................................ 13 2.1
Introduction .................................................................................................................................. 13
2.2
Traditional Software Development .................................................................................. 14
2.3
Iterative Software Development ........................................................................................ 15
2.4
Lean Software Development ................................................................................................ 18
2.4.1
Lean Manufacturing ........................................................................................................ 19
2.4.2
Lean Concepts .................................................................................................................... 20
2.4.3
‘Leaning’ Software Development ............................................................................. 23
2.4.4
Lean or Agile ....................................................................................................................... 27
2.4.5
Summary .............................................................................................................................. 28
2.5
Regulating Software Development ................................................................................... 29
2.5.1
Domain Specific Software Regulations ................................................................. 30
2.5.2
Medical Device Software Regulation ..................................................................... 30
2.5.3
Summary .............................................................................................................................. 35
2.6
A Systematic Review................................................................................................................. 35
2.6.1 2.7
The Mapping Study ......................................................................................................... 36
Software Development Methodology Adoption ......................................................... 39
iii
2.7.1 2.8
The Methods-in-Action Framework ....................................................................... 46
Discussion ...................................................................................................................................... 48
2.8.1
Safety-Critical Software Development .................................................................. 49
2.8.2
Regulatory focus on validation and verification .............................................. 52
2.8.3
Hardware/Software interdependencies .............................................................. 53
2.8.4
Regulatory focus on full traceability ...................................................................... 54
2.8.5
Organisational Challenges ........................................................................................... 56
2.9
Conclusion ...................................................................................................................................... 57
2.9.1
Research Questions ......................................................................................................... 58
2.9.2
A Conceptual Framework for Lean and Regulated SD .................................. 59
2.9.3
The Initial SaLes-4-MD Model ................................................................................... 75
Chapter 3: Research Approach ............................................................................................................. 87 3.1
Introduction .................................................................................................................................. 87
3.2
A Philosophic Reflection ......................................................................................................... 88
3.3
Research Methodology ............................................................................................................ 89
3.3.1
Design Science as a Methodology ............................................................................ 91
3.3.2
Artefact Construction ..................................................................................................... 92
3.3.3
Empirical Method Selection ........................................................................................ 93
3.3.4
Summary............................................................................................................................ 102
3.4
The Research Process ........................................................................................................... 102
3.4.1
Overview ............................................................................................................................ 103
3.4.2
Research Scope ............................................................................................................... 106
3.4.3
Research Validation ..................................................................................................... 106
3.4.4
PHASE 0 - “Defining The Problem Area” ........................................................... 108
3.4.5
PHASE 1 – “The Literature Review” .................................................................... 108
3.4.6
PHASE 2 – “Iterative Model Development” ..................................................... 109
3.4.7
PHASE 3 – “Validation”............................................................................................... 125
3.5
Data Analysis Considerations ........................................................................................... 132
3.5.1
Data Overload ................................................................................................................. 133
3.5.2
Researcher Bias .............................................................................................................. 134
3.5.3
Time demands of coding data................................................................................. 135
3.5.4
The adequacy of sampling with small numbers of cases.......................... 135
3.5.5
The Generalisability of the findings..................................................................... 136
3.6
Research Validity ..................................................................................................................... 137 iv
3.7
Research Quality ...................................................................................................................... 139
3.7.1
Quality Attributes and Principles ......................................................................... 140
3.7.2
Quality Building Techniques ................................................................................... 142
3.8
Conclusion ................................................................................................................................... 146
Chapter 4: Development of the Models......................................................................................... 149 3.9
Introduction ............................................................................................................................... 149
3.10
PHASE 2 – “Iterative Model Development” .......................................................... 149
3.10.2
Regulations and Standards ................................................................................. 151
3.10.3
Case Study Observations ...................................................................................... 154
3.10.4
Expert Opinion .......................................................................................................... 176
3.10.5
Conceptual Framework Changes ..................................................................... 177
3.10.6
Enfolding Literature ............................................................................................... 189
3.10.7
Summary ...................................................................................................................... 207
Chapter 5: Validation of the SaLeS-4-MD Model ...................................................................... 210 5.1
Introduction ............................................................................................................................... 210
5.2
PHASE 3 – “Validation”......................................................................................................... 210
5.2.1
The Medi-SPICE Assessment ................................................................................... 210
5.2.2
Expert Interviews ......................................................................................................... 212
5.3
Conclusion ................................................................................................................................... 228
Chapter 6: Safe and Lean Software for Medical Devices (SaLeS-4-MD Described) 230 6.1
Introduction ............................................................................................................................... 230
6.2
Process Overview .................................................................................................................... 230
6.2.1
Research and Development ..................................................................................... 234
6.3
The Iteration .............................................................................................................................. 235
6.4
The SaLeS-4-MD Practices .................................................................................................. 237
6.4.1
SP1. Define and establish a unit verification strategy................................ 237
6.4.2
SP2. Define and establish software unit verification criteria ................. 238
6.4.3
SP3. Define and establish software unit acceptance criteria.................. 239
6.4.4
SP4. Develop software units .................................................................................... 239
6.4.5
SP5. Ensure consistency and bilateral traceability to system and
software requirements ........................................................................................................................ 248 6.4.6
SP6. Ensure consistency and bilateral traceability to detailed design 248
v
6.4.7
SP7. Ensure consistency and bilateral traceability to test specification
of software units ..................................................................................................................................... 249 6.4.8
SP8. Verify software units......................................................................................... 250
6.4.9
SP9. Document software unit verification results ....................................... 255
6.4.10
SP10. Conduct risk analysis ................................................................................ 256
6.4.11
SP11. Conduct risk evaluation and risk control activities .................. 258
6.5
Conclusion ................................................................................................................................... 258
Chapter 7: The Conceptual Framework Described ................................................................. 260 7.1
Introduction ............................................................................................................................... 260
7.2
The Framework Structure .................................................................................................. 262
7.3
Situated Lean Software Development Practices ..................................................... 262
7.4
Regulatory Requirements ................................................................................................... 263
7.5
Lean Thinking............................................................................................................................ 264
7.6
Organisation ............................................................................................................................... 266
7.7
Project Characteristics.......................................................................................................... 268
7.8
Human Engagement ............................................................................................................... 269
7.9
Conclusion ................................................................................................................................... 269
Chapter 8: Summary and Conclusions........................................................................................... 272 8.1
Introduction ............................................................................................................................... 272
8.2
Research Objective ................................................................................................................. 272
8.3
Summary of Research Approach ..................................................................................... 273
8.4
The Research Questions Answered ............................................................................... 273
8.5
Contributions............................................................................................................................. 281
8.5.1
Theoretical Contributions: ....................................................................................... 282
8.5.2
Practical Contributions: ............................................................................................. 283
8.6
Limitations of the Research ............................................................................................... 285
8.7
Research Quality Review ..................................................................................................... 287
8.8
Potential for Future Research ........................................................................................... 289
8.9
Concluding Remarks .............................................................................................................. 291
References .................................................................................................................................................... 292 Appendix A – The Mapping Study Process .................................................................................. 308 Appendix B – The Mapping Study Protocol ................................................................................ 312 Appendix C – Selected Literature from Mapping Study ....................................................... 321 Appendix D – Data Extraction Sheet (truncated version) ................................................... 323 vi
Appendix E – Case Study Protocol ................................................................................................... 324 Appendix F – Participants’ Information Sheet .......................................................................... 332 Appendix G – Case Study Interview Guide .................................................................................. 334 Appendix H – Industry Questionnaire ........................................................................................... 336 Appendix I – The SaLeS-4-MD Model Detail ............................................................................... 337 Appendix J – The Industry Report ................................................................................................... 363 Appendix K – The SaLeS-4-MD Model Version 1 ..................................................................... 379 Appendix L – The SaLeS-4-MD Model Version 2 ...................................................................... 391 Appendix M – Analysis of the Literature Review ..................................................................... 423
vii
Abstract Developing software for Medical Devices (MD) is quite different to that of “normal” software development – for example where the software is intended for use as a desktop application. Within the MD domain the most important factors influencing the development of software is the safety concerns. For this reason regulations governing the development of such software were introduced. These regulations refer to the processes that must be in place and the burden of proof that must be met before such devices can be passed fit for use. Agile software development methodologies offer the promise of reduced development cycles, lower cost of development, better quality software and more satisfied customers. Lean software development, with a philosophy derived from the Lean manufacturing domain, asks us to consider it from a customer-value perspective. A combination of lean thinking with agile practices offers the potential to be the equivalent of moving from mass production to lean manufacturing within software development. Despite a lot of evidence in support of the agile claims, and a growing movement towards lean, very little evidence (either academic or industrial) exists for the application of these methodologies within the MD domain. By investigating how a lean software development methodology might be applied within the MD domain, this research developed a framework for conceptualising the challenges and opportunities of applying lean/agile philosophies and practices within such a regulated environment. This was achieved through a research approach which combined academic and industry expertise with real-life case study analysis. What emerged is a conceptual framework and process model for lean software development for Medical Devices. As an adaptation of the Methods-In-Action model, the conceptual framework portrays the software development life-cycle as being a context sensitive collection of processes and practices, and illustrates how regulations and a lean mindset influence the other framework components. The process model, again adapting existing frameworks for software process improvement, presents an empirically grounded process for MD software development. Advocating an iterative methodology, it follows a lean approach suggesting a selection of practices which may be adopted by MD companies in order to improve their SDLC. Both the conceptual framework and process model keep at their core the need for safety, compliance with regulations, and the need to give companies a mechanism to make their software development processes as efficient and flexible as possible.
viii
Declaration The work presented in this thesis is entirely my own work. It has not been submitted previously to this or any other institute for this or any other academic award. Where use has been made of the work of other people, it has been acknowledged and referenced.
Signed: _______________________________
Date: ________________________
Oisín Cawley
ix
Glossary
Medical Device Standards: Basel II Accord – Intended to create an international standard for banking regulators to control how much capital banks need to put aside to guard against the types of financial and operational risks banks (and the whole economy) face. EU Council Directive 93/42/EEC - The Medical Device Directive (Council Directive 93/42/EEC of 14 June 1993 concerning medical devices is intended to harmonise the laws relating to medical devices within the European Union. FDA 21 CFR Parts 808, 812, and 820 – In the United States, the Food and Drug Administration (FDA) revised the current good manufacturing practice (CGMP) requirements for medical devices and incorporated them into a quality system regulation. ISO/IEC 12207 – An international standard establishing a common process framework for describing the life cycle of man-made systems. ISO 13485:2003 - An ISO standard, published in 2003, that represents the requirements for a comprehensive quality management system for the design and manufacture of medical devices. ISO 14971:2007 - An ISO standard, published in 2007, that represents the requirements for a risk management system for medical devices. IEC 62304:2006 – An ANSI/AAMI/IEC standard, published in 2006, which specifies life cycle requirements for the development of medical software and software within medical devices. It is harmonized by the European Union and the United States, and therefore can be used as a benchmark to comply with regulatory requirements from both these markets. ISO/IEC 15504 – Also known as SPICE (Software Process Improvement and Capability dEtermination) is a framework or reference model for the assessment of software and system development processes. Medi-SPICE – An initiative to define a conformity assessment approach for the medical device software industry to support first, second and/or third party assessments which may be recognised by the regulatory bodies and pave the way for a more streamlined pathway towards compliance.
x
Sarbanes-Oxley Act - A United States federal law that set new or enhanced standards for all U.S. public company boards, management and public accounting firms
Lean Terms: Andon - A device that calls attention to defects, equipment abnormalities, other problems, or reports the status and needs of a system typically by means of lights – red light for failure mode, amber light to show marginal performance, and a green light for normal operation mode. Flow - The progressive achievement of tasks and/or information as it proceeds along the value stream, flow challenges us to reorganize the Value Stream to be continuous. Gemba - Literal translation is "the real place." where the actual services are provided or where the work is done. Heijunka - Production leveling process that attempts to minimize the impact of peaks and valleys in customer demand. It includes level production volume and level production-variety. Jidoka - A form of automation in which machinery automatically inspects each item after producing it, ceasing production and notifying humans if a defect is detected. Just-in-Time - A system of managing production processes that result in linebalancing, one-piece flow, and little or no excess material inventory on hand. A strategy that concentrates on making quality products, in the quantity needed, when needed. Kaizen- Meaning “change for the better”. Applied to business organizations, it implies continuing improvement involving everyone in the workforce. Kanban - A card or sheet used to authorise/signal production or movement of an item. Lean Manufacturing/Lean Production - A production practice that considers the expenditure of resources for any goal other than the creation of value for the end customer to be wasteful, and thus a target for elimination. Muda - A traditional general Japanese term for activity that is wasteful and doesn't add value or is unproductive. Removing waste is an effective way to increase profitability. Mura - A traditional general Japanese term for unevenness. It is the waste of variation in the production process. Muri - A traditional general Japanese term for overburden, unreasonableness or absurdity. Can be eliminated with the employment of standardised work. Poka-yoke - Any mechanism in a lean manufacturing process that helps an equipment operator avoid (yokeru) mistakes (poka). Its purpose is to eliminate product defects by preventing, correcting, or drawing attention to human errors as they occur. Pull - Principle that no upstream function or department should produce a good or service until the customer downstream asks for it.
xi
Value: When a product or service has been perceived or appraised to fulfil a need or desire--as defined by the customer--the product or service may be said to have value or worth. Components of value may include quality, utility, functionality, capacity, aesthetics, timeliness or availability, price, etc. Value Stream: All the activities (both value-added and non-value added) required within an organization to deliver a specific service; “everything that goes into” creating and delivering the “value” to the end-customer. Waste/Muda - Any operation, activity, or process step that takes time and resources but does not add value to the product or service sold to the customer.
xii
List of Tables Table 2-1 Lean Principles Relevant to Software Development ................................ 24 Table 2-2 Waste in Lean Software Development...................................................... 25 Table 2-3 Sample Agile Practices within Lean Software Development .................... 26 Table 2-4 Search Results summarized as Publication Type by Study Type ............. 37 Table 2-5 A sample of the high level categories from literature review analysis .... 60 Table 2-6 The Distilled List of Lean Software Development Principles ................... 77 Table 2-7 Sample of the SaLeS-4-MD Model Version 1............................................. 83 Table 3-1 Philosophical Assumptions of Three Research Perspectives (Vaishnavi and Kuechler, 2007) ........................................................................................................ 89 Table 3-2 Research ‘Fit’ (Edmonson and McManus, 2007) ...................................... 94 Table 3-3 Research Approach for each Research Question .................................... 102 Table 3-4 ... Interviewees ........................................................................................ 111 Table 3-5 Sample of the Coding Matrix from the Data Analysis Process................ 121 Table 3-6 Exploratory Survey Participant Details .................................................. 124 Table 3-7 Expert Panel Members ............................................................................ 129 Table 3-8 Research Quality Attributes (Miles & Huberman, 1994) ....................... 139 Table 4-1 The Iterative Practice Validated through the Case Study....................... 157 Table 4-2 New “Heijunka” Practice added to the Model ......................................... 163 Table 4-3 the ... Risk Analysis Matrix ...................................................................... 166 Table 4-4 New Risk Analysis Practices added to the Model ................................... 168 Table 4-5 ... Documentation Scaling Matrix ............................................................ 172 Table 4-6 New Regulatory Focused related Practices following the ... Case Study 175 Table 4-7 Sample table of the data analysis results ................................................ 179
xiii
Table 4-8 Data analysis for ‘Regulatory Requirements’ component ...................... 182 Table 4-9 Data analysis for ‘Organisation’ component ........................................... 183 Table 4-10 Data analysis for ‘Project Characteristics’ component ......................... 185 Table 4-11 Data analysis for ‘Human Engagement’ component ............................. 186 Table 4-12 Practices in Support of Medi-SPICE Model Specific Practice 3 ............. 198 Table 4-13 SaLes-4-MD Model Changes incorporating Vogel’s Concepts .............. 203 Table 4-14 Summary of SaLeS-4-MD Changes leading to Version 2....................... 208 Table 5-1 SaLeS-4-MD with Sample Validation Points from the Medi-SPICE Assessment .................................................................................................................... 212 Table 5-2 Summary of the Validation Process Results ........................................... 227 Table A-1 The Mapping Study Literature Sources .................................................. 310 Table A-2 Researcher’s Questions to Solicit Quality of Paper ................................. 311 Table B-1 Search Sources and Results .................................................................... 315 Table B-2 Quality Assessment Questions ................................................................ 318 Table B-3 Sample of Data Extraction Fields ............................................................ 318 Table B-4 Data Extraction Template ....................................................................... 320 Table I-1 The SaLeS-4-MD Model Detail ................................................................. 337 Table K-1 The SaLeS-4-MD Version 1 ..................................................................... 379 Table L-1 The SaLeS-4-MD Version 2 ...................................................................... 391 Table M-1 The recurring themes and high level categories from the literature review ............................................................................................................................ 423
xiv
List of Figures Figure 1-1 High-Level Research Overview ............................................................... 10 Figure 2-1 Relationship between Value, cost and Waste (Hines et al., 2004) .......... 20 Figure 2-2 Foundations of Lean Software Development .......................................... 27 Figure 2-3 Overview of Medi-SPICE Process Groups................................................ 34 Figure 2-4 Reported domains from the Mapping Study ........................................... 37 Figure 2-5 Methodologies reported from the Mapping Study.................................. 38 Figure 2-6 Agile ‘flavour’ being reported from the Mapping study .......................... 38 Figure 2-7 Polar Chart showing Dimensions affecting Method selection (Boehm & Turner, 2003a) ................................................................................................................ 41 Figure 2-8 The Crystal Methodology Grid (Cockburn, 2000) ................................... 42 Figure 2-9 The 3-Stage Agile adoption Process (Sidky & Arthur, 2007) .................. 44 Figure 2-10 The NIMSAD framework (Jayaratna, 1986) .......................................... 46 Figure 2-11 The Methods-in-Action Framework (Fitzgerald et al., 2002) ............... 47 Figure 2-12 The SDM Spectrum (Adapted from Boehm, 2002) ............................... 48 Figure 2-13 The Initial Conceptual Framework for Lean and Regulated SD ........... 61 Figure 2-14 Mindmap Diagram of Categorised Lean Principles and Practices ........ 80 Figure 2-15 Mapping of Lean Principles to ISO 12207 Life Cycle Process Areas ..... 84 Figure 3-1 Kolb’s Learning Cycle (Mullins, 2004) .................................................... 90 Figure 3-2 The Research Process ............................................................................ 105 Figure 3-3 Case Study Process – Adapted from Verner et al., 2009 ....................... 110 Figure 3-4 Sample 1 of an Interview Memo Sheet .................................................. 118 Figure 3-5 Sample 2 of an Interview Memo sheet .................................................. 119 Figure 3-6 Snapshot of NVivo Application .............................................................. 120
xv
Figure 3-7 Example of Quantitative Summary of Expert Responses ...................... 131 Figure 4-1 Phase 2 Empirical Sources ..................................................................... 151 Figure 4-2 Influences on the ... SDLC as derived from the Case Study Analysis...... 154 Figure 4-3 The ... Software Development Life-Cycle ............................................... 155 Figure 4-4 The Detailed ... SDLC .............................................................................. 155 Figure 4-5 An Iterative Nature added to the High Level Process Model ................ 157 Figure 4-6 An NPD Framework (Sperry and Jetter, 2009) ..................................... 158 Figure 4-7 SaLeS-4-MD Process Model Overview with Possible R&D Phase ......... 160 Figure 4-8 A Value-Stream Map of the Validation Process ..................................... 162 Figure 4-9 A Visual Display Board in ... “Heijunka” ................................................. 163 Figure 4-10 Extract from ... Training Manual .......................................................... 164 Figure 4-11 Sample Lean Project Overview within ... ............................................. 165 Figure 4-12 The ... Life-Cycle showing Risk Analysis throughout ........................... 166 Figure 4-13 Risk Analysis Process Overview within ... ........................................... 167 Figure 4-14 A Graphic Representation of the ... Life-Cycle showing Entrance and Exit Criteria .................................................................................................................... 172 Figure 4-15 The Conceptual Framework Components ........................................... 181 Figure 4-16 Lean Thinking exemplars .................................................................... 182 Figure 4-17 Regulatory Requirements exemplars .................................................. 183 Figure 4-18 Organisation exemplars....................................................................... 184 Figure 4-19 Project Characteristics exemplars ....................................................... 185 Figure 4-20 Human Engagement exemplars........................................................... 188 Figure 4-21 The updated conceptual framework ................................................... 189 Figure 4-22 Process Categories of the SPICE Software Process Model (Simon, 1996) ........................................................................................................................................ 191 xvi
Figure 4-23 The SaLeS-4-MD Model Overview ....................................................... 191 Figure 4-24 The SaLeS-4-MD Process Model Overview with an R&D Phase ......... 195 Figure 4-25 The Medi-SPICE PRM showing completed Process Areas .................. 196 Figure 4-26 The Specific Practices of the Medi-SPICE Software Construction Process Area................................................................................................................................ 197 Figure 4-27 Extract from the SaLeS-4-MD Model Layout Version 2 ...................... 200 Figure 4-28 Mapping IEC 62304 Activities into an Agile Incremental Life-Cycle (AAMI, 2011) ................................................................................................................. 206 Figure 6-1 The SaLeS-4-MD Process Model Overview ........................................... 232 Figure 6-2 The SaLeS-4-MD Process Model Overview with a R&D Phase.............. 235 Figure 6-3 The SaLeS-4-MD Iteration ..................................................................... 236 Figure 7-1 A Conceptual Framework for Situated Lean Software Development ... 261 Figure 8-1 The Planning Spectrum (Adapted from Boehm, 2002) ........................ 275 Figure 8-2 Foundations of Lean Software Development ........................................ 276 Figure 8-3 The Conceptual Framework for Lean and Regulated SD ...................... 277 Figure 8-4 A Conceptual Framework for Situated Lean Software Development ... 279 Figure 8-5 The SaLeS-4-MD Process Model Overview ........................................... 281 Figure A-1 The Mapping Study Process Steps ........................................................ 308 Figure A-2 Planning the Mapping Study ................................................................. 308 Figure A-3 The two Stage Review Process .............................................................. 311 Figure B-1 The Data Extraction Process ................................................................. 317 Figure D-1 Snapshot of the Data Extraction Sheet from the Mapping Study ......... 323 Figure E-1 The Lean Framework as a Focusing Tool.............................................. 327 Figure J-1 Software Development Philosophies and Methods ............................... 367
xvii
Chapter 1: Introduction
1.1
Background
The question “What is the best way to develop software?” sounds like a straight forward question and yet new approaches continue to be trialled in various contexts with differing levels of success (Boehm, 2006), (Hneif and Hock Ow, 2009), (Jalali and Wohlin, 2010), (Gary et al., 2011). The key to understanding this phenomenon is the realisation that software so pervades our lives that it is being used in more and more situations which were not previously envisioned. As a result, the software development processes designed in the past are finding it difficult to deliver the demands of the modern business environments, for example, higher quality, faster time-to-market, and higher customer satisfaction levels (Abrahamsson et al., 2002), (Boehm and Turner, 2003a), (Graaf et al., 2003a). When developing software for mission or safety critical systems such as aviation, automotive or medical devices (MD), software developers are faced with the additional task of not only satisfying the customer, but ensuring adherence to the various regulations which have been imposed. In such contexts, these regulations have a primary role of minimising the risk to human safety through software failure or misuse (RTCA, January 1992), (ISO, 2009), (ANSI/AAMI/IEC, 2006). Particularly within the medical device industry, the growing reliance on software has introduced new sources of risk to safety that were not experienced before (CDRH, 2002), (IEEE, 2008). In response to this growing pervasiveness of software in our everyday lives, and the consequent increase in risk of harm, various software development standards and guidelines have been drawn up by regulatory bodies at both national and international levels in an attempt to reduce the risk of harm to the lowest acceptable level (FDA, April 2009), (EU, 1993). Within the MD domain
1
such regulations can be seen as being quite restrictive in their processes, and incompatible with certain software development approaches, similar to what has been experienced in other domains such as automotive (McCaffery et al., 2008). In addition, since many of the regulations are not meant to be prescriptive, the wording of the regulations are open to interpretation and as a result some confusion exists about the precise requirements expected of the manufacturer (CDRH, 2002). As the research presented here revealed, this can lead companies to unnecessarily employ heavyweight methodologies in order to “be sure” that compliance is achieved.
1.2
Motivation
My own personal interest in Software Development stems from having worked in industry for 17 years. During this time I had observed that the Information and Technology (IT) departments were getting separated to a certain degree, from the rest of the organisation. IT, and especially the software development component, was seen as a black box by many in the business community who were not willing, able, or interested in understanding how applications were being developed. A majority of my work experience was with ModusLink, a leader in global supply chain business process management. NASDAQ listed, it has over 25 centres in 14 countries and has a diverse and interconnected collection of information systems which are developed and maintained by a combination of in-house and outsourced IT. It was within this context that I first learned about the concepts of Lean Manufacturing (Womack et al., 2007) and Six Sigma (George, 2002). However, while these concepts were being adopted within the business, there was very little if any involvement of the IT department. This was something that was peculiarly just accepted. However, in carrying out research into this phenomenon, I identified various approaches on how to improve business processes and, in turn, profitability. Such improvements came in the form of strategies such as identifying and concentrating on core competencies (Prahalad and Hamel, 1990), breaking 2
into/creating new market segments (Bower and Christensen, 1995), innovating new products (Hargadon and Sutton, 2000), or connecting with and partnering with other companies (Huston and Sakkab, 2006). These were achieved in many different ways including: modified organisational configurations, capitalising on the innovative abilities of the workforce, and streamlining the product development process through Lean initiatives. A common theme running through these success stories was their collaborative nature, in other words, getting people to work better together as a more cohesive whole. This initial research invigorated my interest in this subject area, and led me to researching the origins of lean manufacturing and the application of these lean principles within the area of software development. I could see the potential that these concepts held for integrating the sometimes opposing objectives within an organisation. It offered a unified approach through working closely together towards a common objective driven by underlying value-driven principles. What remained unclear was how applicable this approach was when faced with the perceived constraints imposed by regulatory controls. Such regulatory controls can be perceived as being quite prescriptive and leading to longer development cycles, therefore an interesting question emerged as to how lean software development could be applied in such contexts. As this was also something industry was expressing an interest in, it was certainly an area which warranted some detailed investigation.
1.3
The Problem Area
1.3.1 Medical Devices The definition of a medical device is quite a protracted one. The European Medical Device Directive (EU, 2007) article 1 paragraph 2 defines a medical device as: “any instrument, apparatus, appliance, software, material or other article, whether used alone or in combination, including the software intended by its
3
manufacturer to be used specifically for diagnostic and/or therapeutic purposes and necessary for its proper application”. Indeed, as technology progresses and software and hardware become more interwoven and easier to connect, the above definition continues to evolve. Because there are such a wide variety of medical devices, the European Commission in their guidance document (EU, 2010) on the classification of medical devices states that: “It is not feasible economically nor justifiable in practice to subject all medical devices to the most rigorous conformity assessment procedures available. A graduated system of control is more appropriate.” This is only common sense when you consider the difference between, for example, a stethoscope and an implantable cardiac pacemaker. Both are considered medical devices, but clearly the level of risk of harm to a patient is higher if there is an issue with a pacemaker, and consequently one would expect a higher level of care to be taken in its design and manufacture process. In order to facilitate such an approach, medical device classification has been introduced. The classification is a risk based approach and takes a number of criteria into account. These criteria include duration of contact with the body, degree of invasiveness, and local versus systemic effect. Within the European Union there are 4 classes of device and 18 rules which help a manufacturer determine the appropriate class (EU, 2010). These classes are I, IIa, IIb and III with an increasing level of risk to patient or user with each class. In the United States there is a similar system of three device classes: I, II, and III (FDA, 2011). This again is a risk based grading system with class III containing devices with the greatest risk. This classification of devices is important since it sets the type of governmental submission/application a company must follow before marketing. It also determines the conformity assessment they must perform in order to demonstrate compliance with the requirements of the directives. The choice of picking the Medical Device domain for this research came about for two reasons. Firstly, this research was part of a larger research 4
project developing a software process improvement model for the medical device industry (Medi-SPICE). The specific differences in this domain from other software development are mainly three-fold: 1. Human safety is a critical concern 2. Software typically has to integrate with some hardware component(s)1, and 3. Regulations exist governing the development of such devices. The second rationale for choosing the MD sector as opposed to another safetycritical domain was due to the desire to pick an area which had potential benefit in an Irish context. Fourteen of the world’s top MD companies have a presence in Ireland (McCaffery and Casey, 2010). However, an Irish Government report has highlighted that very little of the Information and Communications Technology (ICT) associated with this industry takes place in Ireland (Forfás, 2008). Consequently, it made good sense to focus around Medical Devices (MD) and possibly achieve some useful collaboration which could assist my research. The first two points above are in fact typical of safety-critical domains in general, for example Aviation, Automotive and Nuclear, due to the possible risk to humans, but it is in their respective regulations where we see domain specific differences.
1.3.2 Medical Device Software As technology advanced, software started to play a more important role in MDs. In the late 1980s, a radiation therapy machine (the Therac-25), was involved in massive overdoses of radiation in cancer patients, some of whom died as a result. The problem was attributed to the software: “The basic mistakes here involved poor software engineering practices and building a machine that relies on the software for safe operation” (Leveson, 1995).
In 2007 the European Medical Device Directive was amended to include stand-alone software to be possibly classified as medical devices in their own right. This came into effect in Ireland in March 2010. 1
5
This marked an important point in the history of regulation of medical devices and subsequently led to a concerted effort to introduce tighter controls specifically around the software components. In the United States, the Food and Drugs Administration (FDA), through its Centre for Devices and Radiological Health (CDRH), has issued a guidance document to industry and staff on what companies should submit, in terms of documentation, as part of their premarket submission for software contained in MDs (CDRH, 2005). Within this they describe the need for a manufacturer to determine the “Level of Concern” which will then guide them in determining what they should include in their submission. There are three levels of concern: Major, Moderate and Minor, based on: “an estimate of the severity of injury that a device could permit or inflict, either directly or indirectly, on a patient or operator as a result of device failures, design flaws, or simply by virtue of employing the device for its intended use”. A major level of concern is assigned where “... a failure or latent flaw could directly result in death or serious injury to the patient or operator...”, whereas a minor level of concern is where they are unlikely to cause injury. Depending on what level is assigned, the manufacturer will be expected to furnish quite different levels of detailed documents. For minor levels of concern some documentation is not required at all. Since the determination of classification and level of concern is a somewhat subjective exercise, many companies are having issues as a result of implementing a heavy and costly software development life-cycle (SDLC) to ensure they remain on the safe side of complying with the regulations.
1.4
Research Problem
Software Engineering has recently been compared to the fashion industry where the software development methodologies come in and out of fashion (Jacobson, 2010). Another way of looking at this continuing changing methodological landscape is that we are living in an ever evolving and 6
advancing technologically connected world (Friedman, 2005) and, as a result, the software engineering domain has never had adequate time to stabilise before the next innovation or application arrived (Bertelsen, 1997). The pervasiveness of software has led to hardware/software combinations which may not have been foreseen when the methodologies were created. Indeed as we continue to advance technologically and socially, it is difficult to know with any certainty what we should be planning for. In 2009, the United States Networking and Information Technology Research and Development program (NITRD, 2011) produced a report into high-confidence medical devices for the 21st century and one statement reads: “As the architecture of medical systems evolves from the domain of macro- and micromechanical systems to nanoscale CPS (Cyber-Physical Systems) and biological systems, development methods must fundamentally change”. In addition to these technological changes there has also been an increased focus on the costs associated with the development of MDs. In the past this was less of an issue, but with growing competition and increased demand for low cost devices, the industry is examining ways to reduce costs and the SDLC is once such associated cost which is being scrutinised. The philosophy of Lean is to eliminate wasteful process steps and concentrate on creating customer value, and so adopting a lean software development methodology may be a way of reducing cost within the SD aspect of a MD. However, the research presented in this thesis has shown that, due to the relatively new rejuvenation of lean SD, there has been very little published either academically or in the popular literature. Lean SD is the application of lean principles, concepts and values, derived from the domain of lean manufacturing (Womack et al., 1990), to the practice of software development. Although lean SD is an intriguing concept, its nascent nature makes it difficult to decide how best to investigate its applicability in a particular context. Many lean SD practices are considered to overlap with those in agile methodologies (Poppendieck and Poppendieck, 2003), and it is therefore important that we 7
include in our examination the pros and cons which have been highlighted with agile software development, for example faster feedback loops, shorter development life-cycles, inability to scale and lack of applications within safetycritical contexts
(Abrahamsson et al., 2003), (Boehm and Turner, 2005),
(Fairley and Willshire, 2005), (Ambler, 2008b), (Abrahamsson et al., 2010). From this we can gain an insight as to the issues lean SD might also experience in similar contexts. For example, from a regulatory perspective it would seem counterintuitive that practices based around a manifesto which values “working software over comprehensive documentation” (AgileAlliance, 2001) would easily satisfy regulatory controls which are very much documentation centric. Understanding how this principle can be upheld when developing life-critical systems would assist in our examination of lean SD in such a context. An initial literature review had revealed the following two knowledge gaps: -
a lack of understanding of lean SD, and
-
how to apply lean SD within the safety-critical domains.
Due to this unexplored nature of lean software development within the strict regulated medical device domain, our understanding of its suitability has been lacking. Consequently what would be beneficial is a conceptual representation of this area through which we might theoretically and logically examine it in more detail. Therefore, one of the expected outputs from this research was: -
A conceptual framework describing lean software development within a regulated context.
In addition, if we are interested in implementing a lean SD approach within the MD domain, what would such a process look like and what sort of practices would be supported? Therefore an additional expected output was: -
A model for software development which could provide the benefits of a lean philosophy while maintaining MD regulatory compliance.
A final consideration was to ask how might companies wishing to embark on adopting a lean approach, be supported. The final expected output was therefore:
8
-
A reference tool to assist companies in this domain to embrace a more lean approach to software development.
By addressing these problems and generating these outputs, this research could assist foreign Irish based and indigenous Irish companies see the potential benefit in a lean SD approach and consider developing their ICT capabilities in Ireland, and in the case of non medical device Irish software companies, to consider extending/transitioning into this space.
1.5
High-level Research Overview
Motivated by the above, this research examines the contemporary processes and practices for software development within safety-critical domains in detail, with a specific focus developed around the Medical Device software industry. The research takes an interpretative approach, with a conceptual framework and software development process model being built up in an iterative fashion. A high-level overview is shown in Figure 1-1. The large central arrow indicates the progression of the research, starting with a detailed review of the problem area and the formulation of the specific research questions, through to the final validated version of the Conceptual framework and SaLesS-4-MD model. The circulating arrows between ‘Model Development’ and ‘Validation’ depict the iterative approach taken to the empirical data collection and analysis and the evolutionary construction of the models. The resulting process model is named ‘Safe and Lean Software for Medical Devices’ or SaLeS-4-MD for short. The large central arrow is shown to continue after the validated models have been generated, indicating that there is a need to continue research within this area.
9
Figure 1-1 High-Level Research Overview
1.6
Thesis Layout
Chapter 2, a detailed literature review, provides a description of the different approaches to software development. Beginning with the traditional ‘Waterfall’ and ‘Vee’ models, it mentions the issues which the more modern ‘Agile’ methods were created to address, and then gives a detailed description of the emergence of ‘Lean’ software development. This is followed by an examination of the issues with developing software within regulated industries, and also a review of methodology adoption within safety-critical domains. It concludes with a problem description and a set of research questions which the thesis addresses. Chapter 3 introduces the research environment. A brief review of scientific enquiry is followed by an examination of research methods and the decision process around the choice of methods adopted for this study. The overall research process adopted is then presented in detail, describing how each process step was performed and how they influenced the evolving models. Chapter 4 describes in detail the empirical data collection and analysis processes that led to the incremental versions of the SaLeS-4-MD process model and Conceptual Framework. The model’s progression through different 10
versions is explained, and the structure of both the high-level and detailed SaLeS-4-MD model steps is presented. In addition it explains how the final Conceptual Framework evolved out of the initial Conceptual Framework discussed in Chapter 4, into its final configuration following the empirical data analysis. Chapter 5 explains how the second version of the SaLeS-4MD model was validated through a combination of assessment feedback and expert opinion. This produced the final SaLeS-4-MD model, and a summary of the complete validation results is presented. Chapter 6 is a complete description of the final SaLeS-4-MD model, including both the overview and detailed elements. Chapter 7 is a detailed description of the Conceptual Framework. It describes the framework structure, each of the components and how they are inter-related. Chapter 8 concludes the thesis, presenting the research findings and a review of the research objectives. It includes the contributions this research has made, discusses its limitations and how related future research might be shaped.
11
12
Chapter 2: Literature Review 2.1
Introduction
The field of study for this thesis is software development within medical device companies. It has some common characteristics with other safety-critical industries such as aviation and automotive, but is unique in terms of the specific regulations which govern the industry. The industry has traditionally followed a heavy plan-based software development approach but economic pressures are driving a search for efficiencies in all areas. In other industries, the software development methodological evolution has moved from the traditional waterfall approach to a more agile one, and now lean software development is seen by some as the next evolutionary step. It is unclear however to what extent these modern SDMs can be applied in safety-critical contexts including the medical device domain. The over-arching research question can therefore be stated as:
-
How applicable is a lean software development methodology within the medical device software industry?
Therefore, this chapter begins with a brief look at the evolution of software development methodologies, from the traditional to iterative and lean. Since within the literature there existed a lack of clarity and differing opinions of what constitutes lean software development, a detailed description is provided and in particular how it relates to and differs from agile software development. This is followed by an examination of software compliance in regulated settings, including a focused look at the regulations governing the medical device industry and the suitability of various agile and lean practices in this context. A brief review of software methodology adoption is presented, showing the lack of guidance available for lean SD within safety-critical environments.
13
The chapter is concluded with a discussion section and a conclusion highlighting the absence of guidance for industry and the need for a lean model of software development for medical device software.
2.2
Traditional Software Development
Contemporary
thinking
has
software
development
methodologies
categorised into two main groups: Plan-based and Iterative (DeMarco and Boehm, 2002), (Boehm and Turner, 2003a), (Black et al., 2009). Indeed there is much debate about the pros and cons of the various alternatives. The long time favourites of many, the ‘Waterfall’ (Royce, 1970), (Boehm, 1976) or ‘Vee’ (Forsberg and Mooz, 1991) (plan-based models), have been around since the beginning of the practice of developing software and get their names from the visual depiction of the software development process that is typically used to represent them. The waterfall method is a sequential set of process steps typically depicted as flowing downwards, each step flowing from the preceding one. The ‘Vee’ model or more commonly written V-model (Pfleeger and Atlee, 2010), can be viewed as an extension of the waterfall model whereby the process steps turn back up following the implementation stage in order to demonstrate the alignment of the various testing steps with their corresponding development steps. While this provides a more visible and intuitive view of the development process, there is some doubt about the validity of its underlying approach. Gilb suggested that: “The ‘waterfall model’ may be unrealistic, and dangerous to the primary objectives of any software project”. Brooks wrote that: “Much of present-day software acquisition procedure rests upon the assumption that one can specify a satisfactory system in advance, get bids for its construction, have it built, and install it. I think this assumption is fundamentally wrong, and that many software acquisition problems spring from that fallacy” (Brooks, 1987). 14
Similarly, Boehm, while describing his Spiral development methodology (Boehm, 1988), highlights the primary source of difficulty with this traditional approach – the insistence in creating complete requirements and specification documents before beginning development – and therefore the inability to effectively handle changing requirements during the development lifecycle. Addressing this is a cornerstone of many of the more modern SDMs.
2.3
Iterative Software Development
In the 1990s it was becoming apparent that the traditional software development methodologies were hindering competitive advantage due to their long lifecycles. For example, within the field of satellite systems development, there was a call to halve the duration of the development schedule (ESA, 1998). In order to reduce development schedules from 3-4 years down to 18-24 months with more software-intensive systems, (Vardanega and van Katwijk, 1998) suggested that practices such as more concentrated testing earlier in the process, among others, could help speed up the process, something which would become one of the cornerstones of the agile software development movement. In 2001, the various advocates of alternatives to the traditional waterfall approach met and agreed to an umbrella term called ‘Agile’ and to a set of underlying values and principles, defined in their manifesto (AgileAlliance, 2001). “Agile implies being effective and manoeuvrable. An agile process is both light and sufficient” (Highsmith, 2002). Common characteristics of such agile SDMs include: collaborative, flexible, and iterative (Dyba and Dingsoyr, 2009) which are achieved through practices such as face-to-face communication, small teams, small releases, and minimal documentation (Highsmith, 2002). There are many agile methods in existence today such as: eXtreme Programming (XP) (Beck, 1999), Scrum (Schwaber and Beedle, 2001), Crystal (Cockburn, 2000), Feature Driven Development (Coad et al., 1999), Dynamic Systems Development Method (Stapleton and Constable, 1997), and each emphasises a different set of specific practices. Some are very prescriptive in terms of the 15
software development activities that should be performed, for example pair programming in XP (Jeffries et al., 2001), while others concentrate on project management and collaboration (Highsmith, 2002), (Abrahamsson et al., 2002). The introduction of agile development methodologies such as XP and Scrum held the promise of improved software quality, flexible requirements and clearer project timelines (Jeffries et al., 2001), (Schwaber and Beedle, 2001), (Schwaber, 2004). However, while much evidence does exist to support many of the agile claims, it can be argued that such methodologies are not fully suited to the rigorous environment of mission or safety-critical, embedded software development, including medical device development, due to the need for safety and the high level of regulation (Abrahamsson et al., 2002), (Grenning, 2002), (Wils et al., 2006a), (Sidky and Arthur, 2007), (Chisholm, 2007), (Lin and Fan, 2009), (Gary et al., 2011). A typical feature of agile methodologies is the practice of iterative development, a concept which was first introduced in 1975 (Basili and Turner, 1975), and similarly used by (Gilb, 1985) when he introduced his evolutionary delivery model. A fixed amount of time, between 1 and 8 weeks (Miller, 2001, Highsmith, 2002) , is allowed to develop and deliver a subset of the required functionality in a working version of the software. (Ambler, 2008a) suggests that the preferred iteration length is 2 weeks. By having the requirements for each iteration (the product backlog) prioritised or selected by the customer (Beck and Fowler, 2000), this iterative approach has the benefit of delivering the highest value features first and can help in eliminating unnecessary features which can lead to “Fat” software (Wirth, 1995). After each iteration a review of the process is held (a retrospective) to see if anything needs to be adjusted within the process (Salo et al., 2004). Barry Boehm has consistently put forward the view that a risk-based approach should be a key element in driving the software development selection process (Boehm, 1988). Each cycle of the spiral involves a progression which repeats the steps of the previous spiral gradually building up to the complete product. A risk analysis approach is used to tailor the risk-based 16
processes into an overall development strategy, by defining and addressing risks particularly associated with agile and plan-driven methods. This approach is somewhat similar to the approach suggested by (Basili and Rombach, 1987) in which they advocate the tailoring of the SD process based on a quantitative characterisation of the project goals. Academic literature has numerous anecdotal examples of successful agile implementations, but empirical support has been lacking (Dyba and Dingsoyr, 2009). In addition there has been much debate about the suitability of agile methodologies in certain circumstances (Boehm, 2002), (Lindvall et al., 2002). (Abrahamsson et al., 2003) performed a comparative study of many of the agile methodologies, and found a majority had little support for true project management. Some discussions have indicated issues when trying to apply agile methods to larger projects (Leven, 2007), (Lindvall et al., 2002), (Boehm and Turner, 2005), while others have reported success in rolling out a common set of agile practices across multiple teams (Nielsen and McMunn, 2005) or even an entire organisation (Ryan and Scudiere, 2008). (Ambler, 2008c) provides a walkthrough of one such scaling strategy. Further, issues are identified by (Abrahamsson et al., 2002) where they review many of the agile SDMs. Issues such as XP’s lack of attention to management practices, FDD solely focusing on design and implementation, and Scrum’s lack of detail around integration and acceptance testing are some of the shortcomings identified. However, there is little doubt that agile methods are still gaining in popularity. A 2008 internet survey found that 69% of organisations are using agile approaches in some form or another (Ambler, 2008c). The method of adoption can be quite varied, with some companies opting for a big-bang approach, implementing all the agile practices (Drobka et al., 2004), and others, which seems to be more usual, taking a slow and gradual approach (Aveling, 2004). A survey of some key agile practitioners (Fraser et al., 2006) solicited the following feedback:
17
1. No agile method is implemented “out-of–the-box”. Some tailoring always happens. 2. Agile methods are still evolving as enhancements are needed to cover new domains and circumstances. Point 2 is particularly interesting for this research as it indicates that as the pervasiveness of software grows, it is appearing in places that were not envisaged by the agile alliance and therefore the agile methodologies have to change to accommodate these new domains and situational factors. This appears to be borne out with the increase in papers and articles discussing the pros and cons of agile and plan based SDMs, and researchers and practitioners examining ways of combining them (Williams and Cockburn, 2003), (Boehm and Turner, 2003a), (Black et al., 2009).
2.4
Lean Software Development 2
Although it is experiencing somewhat of a rejuvenation in recent times, Lean Software Development (lean SD) has a longer history, with aspects of lean SD having been around long before the term was coined. For example, one of the main principles of being ‘Lean’, is the elimination of waste (Ohno, 1988), (Naylor et al., 1999) which when translated into software engineering terms, can mean the elimination of defects (bugs) in code (Poppendieck and Poppendieck, 2003), (Hibbs et al., 2009). This may seem an obvious goal of developing software, but not until the creation of formal mechanisms to achieve this, such as Fagan’s inspections (Fagan, 1976) did it begin to show the power of doing this in a systematic fashion (McConnell, 2000). This is one of the cornerstones of what lean software development is founded on, finding and fixing defects early in the development process. Similarly, around the 1960s, Shigeo Shingo created his Zero Quality Control concepts (Vardeman et al., 2002) which incorporated a technique which became known as Poka-yoke (mistake proofing) (Grout and Downs, 2012). The principal idea was to define processes that make it impossible for mistakes to 2
Portions of this section have previously been published (Wang et al, 2012)
18
happen, and where that is not achievable, to design the processes in such a way as to make it so defects are easily detected and corrected. The similarities between Fagan’s code inspections and Shingo’s Poka-yoke devices has been identified by (Schulmeyer, 1990), (Tiernery, 1993). What is interesting to note is that Schulmeyer, when writing about the Total Quality Concept, wrote: “Everything starts with the customer, and his or her perception of value is considered important to a zero defect software program”. As we will see, this concept of customer value is also a core element of a lean philosophy.
2.4.1 Lean Manufacturing The origins and taxonomy of lean manufacturing are generally attributed to the now well studied Toyota Production System (TPS) developed in Japan around the 1940s (Liker, 2003). The TPS is a form of what (Womack et al., 1990) coined ‘Lean Manufacturing’. The core principles underlying lean manufacturing are not confined to manufacturing processes either but have been shown to be applicable in many other disciplines too (Womack and Jones, 1996). One of the cornerstones of Toyota’s success was their ability to produce high quality cars at the end of the production line with little or no need for rework. This was achieved by the early detection of defects and the immediate focus on eliminating the cause of the defect so that it would never happen again. This principle is of course transferable to many other domains/situations. In a software engineering context this can be translated to finding and eliminating defects in your code as quickly as possible and learning not to make those same mistakes again. (Robinson, 1997a) suggests that each bug found by a developer should lead to the following two questions: How could I have automatically detected this bug and how could I have prevented this bug? However, lean is not only concerned with defect identification and eradication but equally concentrates on the surrounding processes, and getting work to ‘flow’ smoothly though the entire process. This is easier to visualise in a 19
manufacturing setting where product can be seen to flow down the assembly line, but has been applied in other areas also (Cumbo et al., 2006).
2.4.2 Lean Concepts Lean is more a way of thinking about, or a mental approach taken to, a particular process or set of processes. Lean is about achieving more with less (Christopher and Towill, 2000), or producing in one-third the time, at one-third the cost, and with one-third the defect rate (Charette, 2007). The primary focus and guiding principle is the identification and elimination of process waste in order to focus on creating customer value (Ohno, 1988), (Hines et al., 2004), (Cumbo et al., 2006), (Womack et al., 2007). An important aspect is that value is determined by the customer, not the producing company, and so any cost incurred in the process needs to be in support of activities which add value to the customer (Figure 2-1).
Figure 2-2 Relationship between Value, cost and Waste (Hines et al., 2004)
There are five main modern lean concepts (Womack and Jones, 1996): Value: It is defined by the customer and it is paramount to have a clear understanding of what that is; Value Stream: A map that identifies every step in the process and categorises each step in terms of the value it adds; 20
Flow: It is important that the production process flows continuously; Pull: Customer orders pull product, ensuring nothing is built before it is needed; Perfection: You strive for perfection in your process by continuously identifying and removing waste. However, for the research presented in this thesis, a 6 th concept is also used, Respect for People. This is highlighted by (Sugimori et al., 1977) where they describe the key facets of the Toyota Production System. This last principle is very significant and a key component of Toyota’s second basic concept: “To make full use of workers’ capabilities”. (Womack and Jones, 1994) see it as a key component in building a lean enterprise, and it has also been incorporated into the International Council on Systems Engineering’s model for Lean Enablers for Systems Engineering (INCOSE, 1990), (Oppenheim et al., 2011). (Petersen and Wohlin, 2011) summarises lean manufacturing under two important headings: -
The removal of waste in the manufacturing process
-
The analysis of the flow of material through the manufacturing process.
As these are core concepts within the lean paradigm, a brief description of these is now presented. 2.4.2.1
Lean Principle – Removal of Waste
In Japanese, process waste is referred to as muda, although other terms such as mura, and muri are also used. Lean thinking classifies work into 3 categories (Womack and Jones, 1996): -
Value-adding activities Required non-value adding activities Non-value adding activities
By mapping out a work process using a value stream map, process steps which do not contribute to creating value can be identified, thus allowing for a concentrated effort on reducing or eliminating these steps. The concept of waste can therefore be quite broad but Taiichi Ohno – the founder of the Toyota
21
production process - defined 7 types of waste (Ohno, 1988), and an eighth has been added by (Liker, 2003). These are: 1. 2. 3. 4. 5. 6. 7. 8.
The waste of over-production The waste of time on hand (waiting) The waste of transportation The waste of over-processing or incorrect processing The waste of stock on hand (excess inventory) The waste of movement The waste of making defective products The waste of unused employee creativity
It is the identification of these different types of waste which make a lean approach so powerful, since it does not suggest that any particular part of the process should be targeted, but rather waste in any form and in any place should be sought out. 2.4.2.2
Analysis of Flow
The concept of flow, from a manufacturing viewpoint, is about having a steady and continuous manufacturing process which can operate indefinitely (in theory). Specific focus is given to ensuring that the rate of production meets the demand, and that no one step in the process gets overloaded and becomes a bottleneck. When a bottleneck does occur, the entire process can come to a halt. In order to reach a state of continuous flow, some form of analysis is needed, and this is typically done by mapping out the entire process (the value chain) and measuring the times it takes product to flow from one step in the process to the next. Those steps which are deemed to be taking too long and thereby affecting the flow can be identified as possible areas of improvement. There are a number of things which are important when considering flow, these include: the build up of queues, the amount of product variability, batch sizes, delivery rate (cadence), work-in-process constraints, and centralised control. An important question to ask is how do such lean manufacturing practices typically repeating identical tasks and producing the same product as output relate to product development activities which always produce something different? The concept of flow within a manufacturing practice is easier to 22
visualise than within the more creative processes of product development. Don Reinertsen has applied lean manufacturing practices to product development (Reinertsen, 1997) and although he identifies some significant differences between the two (Reinertsen, 2009), the concept of flow is still critically important. For example, while the build up of queues is worrying within lean manufacturing, Reinertsen suggests that from a product development perspective, the application of queuing theory together with an economics focus leads to a different interpretation. Reinertsen suggests that queues are not universally bad, but that we must treat them as having a quantifiable cost to the process. In addition, he suggests that reducing batch size is the single most cost effective way to reduce queues, and in fact the goal is often to reduce the batch size to one, a state referred to as one-piece flow. Finally he suggests that blindly trying to drive up efficiencies and drive down variability without considering the economic consequences is fundamentally wrong.
2.4.3 ‘Leaning’ Software Development Looking specifically at lean from a Software Development perspective (Raman, 1998) attempted to “... see whether the basic Lean principles ... can be applied to software development”. He concluded, that with practices such as rapid prototyping, quality function deployment, continuous integration, object oriented and component-based development: “The question whether Lean Software Development is Feasible can easily be answered with “yes” ”. Similarly, (Petersen and Wohlin, 2010) suggest that: “Results of lean product development are more interesting for software engineering than the pure manufacturing part as the success of software development highly depends on an integrative view”. They conclude that lean principles may be beneficial in a software development context but that: “Further evaluation of lean principles is needed to understand how they affect the performance of the software process” (Petersen and Wohlin, 2011). 23
2.4.3.1
Lean Software Development Principles
The core intent of lean can be summarised as follows: “All we are doing is looking at the timeline from the moment a customer gives us an order to the point when we collect the cash. And we are reducing that timeline by removing the nonvalue-added wastes” (Ohno, 1988). The contemporary understanding of lean software development is largely driven by practitioners’ writings (Poppendieck and Poppendieck, 2003), (Poppendieck and Poppendieck, 2006), (Hibbs et al., 2009) and (Anderson, 2010), however the broad nature of lean means that lean software development has much in common with related domains such as lean product development (Reinertsen, 2009) and lean systems engineering (Oppenheim et al., 2011). Table 2-1 summarises the main sets of lean principles within a software development context. Table 2-1 Lean Principles Relevant to Software Development Lean Software The Principles of Product The Kanban Principles Development Principles Development Flow (Anderson, 2010) (Poppendieck and (Reinertsen, 2009) Poppendieck, 2003)
Eliminate waste Build quality in Create knowledge Defer commitment Deliver fast Respect people Optimise the whole
Use an economic view Manage queues Exploit variability Reduce batch size Apply WIP (Work in Progress) constraints Control flow under uncertainty Use fast feedback Decentralise control
Visualize the workflow Limit WIP Manage Flow Make Process Policies Explicit Improve Collaboratively (using models & the scientific method)
While these sets differ slightly in the lean principles they advocate, they are all share some core lean concepts. A common core focus of all lean software development proponents is delivering value by identifying and eliminating waste, but the interpretation of waste and how to address it can vary. For example, while (Poppendieck and Poppendieck, 2003) and (Hibbs et al., 2009) identify specific wastes which should be addressed immediately, (Reinertsen, 2009) suggests that only when
24
the proposed waste has been converted into economic terms does it become useful in deciding whether it is waste and what to do about it. Additionally (Ballard, 2000) talks about addressing what he calls the waste of negative iterations. Referring to iterative design, he says a negative iteration is one which could be eliminated without any loss in value. Table 2-2 shows the interpretation of waste which I have used for this thesis based on (Poppendieck and Poppendieck, 2003, Hibbs et al., 2009, Liker, 2003): Table 2-2 Waste in Lean Software Development
Lean Manufacturing
Lean Software Development
Over-Production Time On Hand (waiting) Transportation Over-Processing or Incorrect Processing Stock On Hand (excess inventory) Movement Making Defective Products Unused Employee Creativity
Extra Features/Code Delays Task Switching Extra Processes Partially Done Work Movement Defects Unused Employee Creativity
From a software development perspective, one interpretation of waste is the identification and elimination of defects in the software, and so effort is expended to ensure defects are found and removed as quickly as possible. Another interpretation is writing too much code such as developing features which were not requested (over production), something which (Hibbs et al., 2009) suggests that Behaviour Driven Development (BDD)3 can help address. Similarly, if defects are allowed to propagate through the system and are identified late in the project, then a considerably larger effort is required to remove them than if they were caught early. This wasted effort may be eliminated by following a test-first approach such as the agile practice of test driven development (TDD) (Beck, 2002).
BDD aims to help focus development on the delivery of prioritised, verifiable business value by providing a common vocabulary that spans the divide between Business and Technology. 3
25
Many more such practices have been mapped to form a toolset of lean SD practices by (Poppendieck and Poppendieck, 2003), (Poppendieck and Poppendieck, 2006) and a guide on implementation was written by (Hibbs et al., 2009). Table 2-3 shows a sample of these practices. Table 2-3 Sample Agile Practices within Lean Software Development
Run software tests as soon as code is written
Write code instead of more documentation or detailed planning
Propose user interfaces and get feedback instead of more detailed requirements
Test the top 3 tools instead of trying to pick the right one first time
Develop in short iterations
Features that are too big for 1 iteration need to be broken down
The highest priority feature should be developed first
High risk items should be addressed earlier rather than later
Make progress visible
Automate software builds and build tests
A more scientific approach is taken by (Reinertsen, 2009) and (Anderson, 2010), where focus is placed on measurements derived from subject areas such as queuing theory, variability and transaction cost analysis and using those measurements to shape and adjust the development processes. Lean SD can be viewed therefore, as the merging of lean concepts with modern software development practices, such as those found in agile, and other general best practices for software development (Figure 2-2).
26
Figure 2-3 Foundations of Lean Software Development
2.4.4 Lean or Agile According to Robert Charette - originator of “Lean Development” (Charette, 2007) – lean software development is a key component in building a change tolerant business (Highsmith, 2002). The key difference he sees between lean and agile is that agile is a bottom up approach while lean is a top down approach. The toolkit developed by (Poppendieck and Poppendieck, 2003) contains many practices already well established within the agile community. This has both helped the agile community to embrace lean, but also added to the confusion as to what exactly is the difference between lean and agile. Consequently the boundary between lean SD and agile SD is something that is currently being debated (Fowler, 2008), (Wang, 2011), (Wang et al., 2012). Oakleigh Consulting (2007) performed a comparison between lean concepts and generic agile practices, with specific focus on the Scrum methodology, and concluded that lean thinking can help a company analyse its software development process irrespective of the development methodology in use. Since lean is a philosophy which can be applied at an organisational level, and 27
agile is typically focused at a more practical level, agile methods can be seen as supportive practices of a lean software development philosophy. I position lean SD within a wider umbrella of ‘Lean Thinking’ which spans the entire organisation. This is important, since getting the entire value-stream working together towards a common goal in a continuously flowing fashion is the real aim of a ‘Lean Enterprise’ (Womack and Jones, 1996). Trying to optimise one piece of the process, for example, writing the software code, may lead to suboptimising the wider process (Goldratt, 1997). (Parnell-Klabo, 2006) for example, when examining their internal processes leading to software delivery, found that the actual coding accounted for only 10% of the time it took to deliver a project into production. This indicates that 90% of the time is devoted to activities outside the actual coding and overflowing into other groups, departments, and or even companies. Some lean practices, however, do not have any specific reference within agile practices. For example, Harold Thimbleby (1988) advocates delaying software decisions as long as possible to be able to quickly cater for any changes that crop up further along in the project. This later became one of the lean SD practices as described by (Poppendieck and Poppendieck, 2003). Although the agile manifesto similarly values “Responding to change”, the specific practice of delaying commitment has not been seen within the corresponding methodologies. Therefore, this research views lean software development as being a collection of practices from different sources and disciplines, and not just those within the agile community.
2.4.5 Summary In this thesis, lean SD is viewed as the integration of lean concepts, principles and values (which have been around for many years) with contemporary software development practices. The underlying lean philosophy drives a focus on waste elimination and the concentration on value creation. There is a close relationship with the agile methodologies, in that many agile 28
practices can be seen as supportive of a lean philosophy. Lean SD has a much longer history, but difficult global economic conditions are leading to a renewed interest in all things lean (LeanSSC, 2009). This would indicate that lean SD may be in need of modernisation as is the case with the agile methods.
2.5
Regulating Software Development
Because software has become so pervasive in society, we have come to rely on it more and more in our daily lives (Eggermont, 2002), (Duranton et al., 2013). Consequently, when it fails or is misused, the effects can be quite devastating. To counteract this risk, various authorities have introduced regulations which aim to govern how software is developed, secured and interacts with other systems. These diverse sources of regulations are increasing as software continues to push boundaries, be misused and get embedded in ways which were not envisioned before. For example, the Enron scandal 2001, which resulted in the loss of over $11 Billion of investors’ and employees stocks and pensions, was due to fraudulent financial reporting (BBC, 2002). In Panama, 21 patients died from overdoses of radiation during cancer treatment as a result of software failure combined with software misuse (Borrás, 2006). FDA analysis of 3,140 medical device recalls between 1992 and 1998 found that 242 (over 7%) were attributable to software (CDRH, 2002). Significantly, of the 23 recalls in 2007 of what the FDA classify as lifethreatening, 3 of them involved faulty software (IEEE, 2008). Consequently, regulations have been imposed to help reduce the possibility of such events recurring. Those in the banking business must comply with the Basel II Accord of 2004 (www.basel-2.org). Publicly listed American companies are now subject to the Sarbanes-Oxley Act (SOX) of 2002 (www.soxlaw.com). Medical device manufacturers are subject to a raft of regulations such as the United States Quality Systems Regulations 21 CFR part 820 (FDA, April 2009) or the European Medical Device Directive (EU, 2007).
29
2.5.1 Domain Specific Software Regulations Due to differing concerns between domains, different software development regulations have been created and consequently affect the development processes differently. The SOX regulations, for example, were designed to ensure accurate financial reporting. Very different concerns, however, are at the heart of regulations pertaining to safety-critical domains such as Aviation, Nuclear, or Medical Devices (MD). In such domains the obvious concern is for human safety, and the regulations are designed to minimize to an acceptable level any related risk. To be SOX compliant you are expected to demonstrate that the software has been adequately tested. However, safety-critical regulations put a much heavier emphasis on this and expect thorough verification and validation of the software (CDRH, 2002). This level of detail, as required by the international standards such as IEC 62304 (ANSI/AAMI/IEC, 2006) for MD software life cycle processes, and ISO 13485 (ISO, 2003) for MD quality management systems, is far more onerous, and compliance must be demonstrated right throughout the entire software development life-cycle. Importantly, these standards govern not only the software which is embedded in the Medical Device, but also any software which may be used during the manufacturing process.
2.5.2 Medical Device Software Regulation To understand the context of software within MDs we first need to have a clear understanding of what constitutes a medical device. This is important, since the definition is continually being extended and consequently affecting the software development processes within more and more companies. The European definition of a Medical Device is: “any instrument, apparatus, appliance, software, material or other article, whether used alone or in combination, including the software intended by its manufacturer to be used specifically for diagnostic and/or therapeutic purposes and necessary for its proper application” (EU, 2007).
30
Examples of such devices are: Bone Cements, Blood bags, Systems intended to preserve and treat blood. However, additions to this definition exist. An active medical device is defined as: “Any medical device operation of which depends on a source of electrical energy or any source of power other than that directly generated by the human body or gravity and which acts by converting this energy.” (EU, 1993) Annex IX Section 1.4. Examples of such devices are: Gas mixers with anaesthesia machines, heating/cooling pads which act by chemical action. Furthermore, an active implantable medical device is defined as: “any active medical device which is intended to be totally or partially introduced, surgically or medically, into the human body or by medical intervention into a natural orifice, and which is intended to remain after the procedure” (EU, 1990) Article 1(c). Examples of such devices are: Coronary pace makers, Implantable drug administration devices. Relevant to this research is the reference in the initial definition to the constituent software component if present. As this definition evolves we will see the regulations applying to a much broader set of ‘devices’, something which has already started to happen in the European Union. For example, the Medical Device Directive (EU, 1993) governs the manufacture of medical devices, and companies must be in conformance with the regulations in order to market their products within the EU. This directive was updated in 2007 (EU, 2007) and the update became mandatory in March 2010. Included in the update was the amended definition of a medical device which now includes the statement: “Stand alone software is considered to be an active medical device” (EU, 2007) Annex IX Section 1.4. Examples of such devices are: software used to plan cancer treatment doses, software to control the setting of oncology treatment devices (McHugh et al., 2011).
31
This has ramifications for many companies who develop software-only products for the medical domain and who now find that they must ensure their software development processes are in compliance with the regulations. The ‘intended use’ of a product as specified by the manufacturer goes a long way in deciding whether it is viewed by the regulators as a medical device, and indeed what type of device they would consider it to be (Gee, 2009). If a company markets their product with the intention to allow it be used to support a medical diagnosis, then it falls into the category of a medical device and therefore must be approved by the relevant authority before being sold. This approval process involves a detailed submission to demonstrate that the device was manufactured in accordance with the regulations. 2.5.2.1 In
ANSI/AAMI/IEC 62304:2006 2006,
the
international
standard
ANSI/AAMI/IEC
62304
(ANSI/AAMI/IEC, 2006) “Medical device software – Software life cycle processes” was published. The standard provides: “a framework of life cycle process with activities and tasks necessary for the safe design and maintenance of medical device software” (Introduction to standard). This standard, which is an update of the ANSI/AAMI SW68 standard (ANSI/AAMI, 2001), addressed some issues with the earlier version by introducing a software safety classification similar to the level of concern described earlier (Jordan, 2005). According to the standard each software system must be assigned an initial software safety class based on severity as follows (ANSI/AAMI/IEC, 2006):
Class A: No injury or damage to health is possible
Class B: Non-serious injury is possible
Class C: Death or serious injury is possible
This new software classification is important because, prior to this, compliance with SW68 was only accepted by the FDA for devices with minor and moderate levels of concern. The new standard addresses this by specifying
32
processes and tasks that should be carried out when developing software for each of the software classes corresponding to each level of concern. The standard is applicable to, but distinguishes between, the development and maintenance of medical device software. Each life-cycle process has a number of requirements, each process is further divided into a set of activities, and most activities further divided into tasks. For example, one of the processes within the standard is called the Software Development Process (process 5), which consists of 8 activities. One of those activities is called Software Unit Implementation and Verification (activity 5.5), and it contains 5 tasks aimed at ensuring that each software unit is implemented in code, and verified appropriately, for example Establish Software Unit Verification Process (task 5.5.2). The IEC 62304 standard is fast becoming the de facto standard for MD software life cycle processes, and compliance with it is already accepted by many countries as satisfying their national requirements. However, this creates a new problem for the industry, especially for new entrants, namely how to ensure their processes are compliant with the standard. Because the MD regulations are non-prescriptive (in order to encompass a large number of device types), it has led to companies having to interpret them, something which can lead to problems when it comes to regulatory audits. To overcome this typically means making use of some form of assessment tool which unfortunately does not yet exist for the MD domain at the current time. With this, a gap in our understanding is identified, namely how do the regulations affect the SD process within new entrants to the MD software industry? Indeed, as the definition of a MD evolves, how does the change in definition affect the software processes and practices within existing MD companies? 2.5.2.2
Medi-SPICE
Currently under development is Medi-SPICE, a process assessment tool for the medical device industry (Medi-SPICE, 2011). Medi-SPICE is an international
33
initiative of the SPICE user group - Software Process Improvement and Capability Determination - and is being led by Dr. Fergal McCaffrey at the Dundalk Institute of Technology in Ireland (McCaffery and Dorling, 2009, McCaffery et al., 2010). It will contain a process reference model (PRM) and process assessment model (PAM) that is in conformance with the requirements of the international standard for process assessment (ISO/IEC, 2004) but which will integrate other relevant standards such as IEC 62304 and ISO 14971 (ISO, 2007). Medi-SPICE organises software development into three life-cycle categories: Primary, Supporting, and Organisational. Each of these categories contains a number of process groups, each containing multiple process areas, and each process area containing various specific practices. Figure 2-3 is a visual map of a section of the Medi-SPICE framework with the software construction process area expanded to show the 11 specific practices. The letters at the end of the practices indicate the device class for which they are mandated, with some still to be finalised.
Figure 2-4 Overview of Medi-SPICE Process Groups
By using the Medi-SPICE reference model, it will be possible to assess companies’ software development processes to aid in supplier selection, to guide process improvement, and to facilitate new companies to embark on medical device software development.
34
2.5.3 Summary Developing software for use within a medical device context is a critical undertaking since the consequences of software failure, either through a defect or by misuse, can be life-threatening. The regulatory controls introduced at national and international levels are attempting to reduce the possibility of such occurrences to an acceptable level. The effect on software development teams is that they have to demonstrate that they have a clearly defined and adequate SD process, supportive of a risk identification and elimination approach backed up by clear and objective evidence of adherence to the process. As the industry matures and advances technologically, new sources of safety risk arise which the regulatory bodies are addressing. With the latest addition to the medical device definition now categorising software itself as a possible MD, more companies are faced with need to introduce regulatory compliant SD processes. Consequently the existing process improvement models are in need of being adapted for application specifically within this domain.
2.6
A Systematic Review
An initial literature review found very little data relating to lean or agile SD within the MD domain. Indeed from a less academic perspective, a wider search of online resources revealed little of substance on the topic, but rather a growing interest in making the medical device software development process faster and more cost effective while satisfying the regulations. Some evidence, however, had been found relating to agile SD within similar safety-critical domains such as aviation and automotive. It was therefore decided that a mapping study would help the research in two ways: 1. Assist in gaining a more comprehensive and detailed understanding of safety-critical SD domains 2. Demonstrate robustness in the literature review process and thereby assist in developing credibility in the research findings.
35
A mapping study, similar to a Systematic Literature Review (SLR) (Kitchenham and Charters, 2007), is: “... a means of identifying, evaluating and interpreting all available research relevant to a particular research question, or topic area, or phenomenon of interest”. It can be used when very little evidence is likely to exist or when a topic is very broad. It is carried out in a planned methodical fashion therefore allowing for a clear, unbiased, and mostly repeatable review of the available literature. SLRs and mapping studies are a result of what is termed the evidence based paradigm which had its beginnings in medical research (Dybå et al., 2005). The mapping study approach therefore adds credibility to the review findings, and builds confidence that the review has uncovered as much as reasonably possible on the subject matter.
2.6.1 The Mapping Study A mapping study was performed according to the guidelines detailed by (Kitchenham and Charters, 2007), and a full description of how the process was carried out can be seen in Appendix A. A synthesis of the mapping study results was published (Cawley et al., 2010). From an initial count of 160 publications, it was narrowed down to 21 relevant papers. These papers spanned the years 2002-2009 and were composed mainly of experience reports published at international conferences (42%), with only 5 classified as empirical research papers - 3 case studies and 2 surveys (Table 2-4). This is indicative of the lack of empirical research in this specific field. Appendix C contains a full list of the publications S1 – S21.
36
Table 2-4 Search Results summarized as Publication Type by Study Type
Experience Report 9 [S9],[S10], [S12],[S13], [S14],[S15], [S17],[S19], [S21] 2 [S1], [S8]
Conference Paper
Journal
Case Study 2 [S11], [S20]
Expert Opinion
Survey
1 [S5]
MetaAnalysi s 1 [S7]
2 [S6], [S16] 3 [S2],[S3], [S18]
Online Articles
1 [S4]
Thesis
Following a detailed review, I identified which domain each paper addressed (Figure 2-4) but noted that some addressed the issue of agile SD in embedded software in general [S6],[S8],[S12],[S14],[S16],[S19], and two [S2] and [S7] relate to more than one domain. While the mapping study looked at regulated
embedded
SD
in
general,
40%
of
the
papers,
namely
[S1],[S3],[S5],[S9],[S13],[S15],[S17],[S18], reported from a medical device perspective.
8 6 3
2
Figure 2-5 Reported domains from the Mapping Study
37
2
The methodologies which appeared most throughout the literature were XP and Scrum. The category ‘unclear’ refers to papers that describe an agile implementation but do not mention the specific method used. The category ‘none’ refers to papers that do not describe an implementation, for example, a survey or expert opinion. Note that one paper [S8] developed a software process selection matrix and makes reference to many different SDMs namely FDD, ASD, XP, Spiral and RUP. 12
6 4
3
2
1
1
1
1
1
Figure 2-6 Methodologies reported from the Mapping Study
One of the areas I was interested in investigating was the ‘flavour’ of agile being adopted/trialled in these domains. Whether full stand-alone agile methodologies (Agile), a combination of different agile methodologies (AgileAgile) or an agile-planned (Agile-Planned) combination was being favoured. From the data there was a slight preference for a combination of agile methods with more traditional planned-type SD practices (Figure 2-6). What is worth noting is the absence of the term ‘lean’ in any of the reviewed papers. 10
9
8 6 4
4
4
4
Agile
Agile-Agile
None
2 0 Agile-Planned
Figure 2-7 Agile ‘flavour’ being reported from the Mapping study
38
The discussion in sections 2.7 and 2.8 draws from both the initial literature review and the mapping study findings.
2.7
Software Development Methodology Adoption
Contemporary software development typically follows a plan-based or agile approach (Boehm and Turner, 2003a). However, deciding on which approach to take can be problematic since there are advantages and disadvantages for both approaches (Parnas and Clements, 1986), (Boehm, 2002), (Cohen et al., 2003). The straight-forward nature of the traditional approach with clearly identified phases occurring in strict order and feeding into each other makes it an intuitive process, and something which is: “... solidly entrenched in the traditional way of doing business” (Glass, 2001). This would therefore suggest that it fits well within or alongside existing business processes. Although agile methodologies are seen as addressing the traditional development shortcomings, a 2008 survey (Ambler, 2008a) showed that 47% of respondents didn’t know when their company would adopt an agile development process and were still practicing a traditional approach. So while agile is growing in popularity, the adoption rate has still some way to go before becoming dominant. From both an academic and practitioner’s perspective, agile practices specifically, have attracted a wide range of research interest in the past couple of decades (Dybå and Dingsøyr, 2008, Dyba and Dingsoyr, 2009). Reports which tell of improved communication and coordination, quick releases, and flexibility of design (Begel and Nagappan, 2007), continue to drive this interest. The
situational
context
(business,
organisational
or
technical
environment) within which software is being developed is an important influence on the whether an agile or plan-based approach is adopted. It may result in inertia if trying to implement an agile methodology within an organisation with established plan-based processes, or a need to satisfy the safety requirements inherent in life-critical environments. For the former, (Boehm and Turner, 2005) suggest a number of techniques to help 39
organisations integrate agile practices into traditional processes, while for the later, a risk based approach is suggested (Boehm and Turner, 2003b). In 2003 (Graaf et al., 2003b) performed a review of embedded-software development practices, and found that the existing methodologies did not take into consideration the specific needs/constraints of embedded system development nor did they indicate how to apply them to the different domain areas. Unfortunately there appears to be “no silver bullet” (Brooks, 1987), and while the agilists might argue for the universal application of their methodologies, others have suggested that more cautious and objective criteria should be used. Over the years, Boehm’s spiral model has been developed into a model which seeks to marry the discipline of plan-based development with the responsiveness and flexibility of agile development (Boehm, 1996), (Boehm, 2002). Boehm and Turner (Boehm and Turner, 2003a) identified four important characteristics of software projects: Application, Management, Technical and Personnel. Projects can be measured against these to determine their appropriate “homeground” – whether a plan-based or agile approach is best suited. They concluded that there are five critical factors which determine the suitability of a particular development methodology: size, criticality, dynamism, personnel, and culture. A very useful visual tool they use is a polar chart (Figure 2-7), whereby the closer you score to the chart’s centre, the more likely an agile development methodology will suit the project.
40
Figure 2-8 Polar Chart showing Dimensions affecting Method selection (Boehm & Turner, 2003a)
A risk-based method is used which identifies when there are some elements which favour an agile approach while for others a planned-based approach is more appropriate. It is also possible for a combination of methodologies to be the outcome. The context in which you are operating will determine what the best fit will be, and this may actually change from project to project. One relevant dimension they use is criticality (loss due to impact of defects), which in certain contexts such as medical devices could mean the loss of human life. They suggest that as this risk increases, a less agile approach to the software development is appropriate. Alistair Cockburn created his Crystal methodologies for developing software (Cockburn, 2000). In a similar approach, he concentrates on two particular attributes of a project, number of people involved and criticality. Depending on the value of these attributes, a different Crystal methodology is deemed most suitable, and by means of a grid (Figure 2-8) this selection is facilitated. Each cell within the grid is identified by a letter-number combination indicating the maximum criticality and project size for that cell. For example, C6 indicates a 41
loss-of-comfort project with a team size of up to 6 people. D40 indicates a lossof-discretionary-money project with a team size of between 21 and 40 people.
Figure 2-9 The Crystal Methodology Grid (Cockburn, 2000)
However, from the point of view of this thesis, three things are important to note in this model (Cockburn, 2002): -
He has had no experience with life-critical applications
-
It was aimed at larger projects where iterations are 1-4 months
-
He only addresses co-located teams.
As software has advanced in sophistication and pervasiveness, the contextspecific application of SDMs has come under scrutiny. Reports have been published of the success/failure of certain approaches, and what practices had to be dropped, enhanced or adopted to satisfy the different situations. Alleman et al. (2003) (Alleman et al., 2003) used a combination of earned-value measurement with agile practices to comply with the “progress-to-plan” approach expected within government contracts. At Microsoft, Anderson (2005) (Anderson, 2005) adapted their agile practices to ensure they complied 42
with CMMI level 3 (CMMI, 2010). Drobka et al. (2004) (Drobka et al., 2004) reported on trialling XP on four mission-critical projects where some of the agile processes had to be tailored. Hastie (2009) (Hastie, 2009) interviewed Suncorp’s Philip Abernathy on his experience in converting to agile within a financial institution and the culture shock which it brought on. Kane et al. (2006) (Kane et al., 2006) reported how they had to augment some of the agile practices to suit their biomedical-software environment. As more companies from varied backgrounds make the decision to look at agile methodologies to improve their SD process, they face the problem of deciding how to begin. Boehm and Turner (2005) (Boehm and Turner, 2005) report on a series of annual workshops which aimed at identifying those barriers (real or perceived) which hindered the adoption of agile SDMs. From these they identified three areas that they say are critical challenges in large organisations: 1. Development process conflicts 2. Business process conflicts 3. People conflicts In fact they make the point in relation to point 3 that: “People issues are by far the most critical in improving management of engineering and development personnel. Addressing them is vital to the adoption and integration of agile methods and practices into your processes”. (Kettunen and Laanti, 2005) note that for embedded software development there is no one-size-fits all and developed a comparative process selection model which looks at the potential problems in a project and chooses the methodology best suited to handle those issues. They also conclude that for large projects a hybrid methodology composed of agile and planned components may be best. More recently (Srinivasan et al., 2009), following a state-of-the-art literature review, proposed six recommendations when considering agile adoption in embedded systems development: -
Determine the organisational and project ‘homegrounds’
43
-
-
TDD is a sound engineering practice but requires cultural change (they suggest that one of the major challenges is to find the right tools without comprising existing practices) Requirements development and management needs to be harmonised Agile methods need organisational infrastructure support Agile methods require both mechanisms and incentives to support knowledge transfer and sharing Transitioning to any new process requires the management of culture change
Importantly, (Sidky et al., 2007) developed an agile adoption framework aimed at assisting organisations in adopting agile practices. While this is a generic agile adoption framework, they followed this by developing a threestage process to help determine the applicability of agile practices to companies developing mission and life-critical systems (Sidky and Arthur, 2007). These stages are (Figure 2-9): 1. Making the Go/No-Go decision 2. Discarding inappropriate practices 3. Determining the right practices to adopt
Figure 2-10 The 3-Stage Agile adoption Process (Sidky & Arthur, 2007)
While this framework is a useful way of approaching agile adoption in safetycritical domains, it is a high level process description, and a closer examination
44
of domain specific agile practices, for example the medical device software industry, would be beneficial. From a lean SD perspective, there is little to be seen in terms of guidelines for adoption. The quest for something which is ‘leaner’ is evident but only very recently have we seen any real evidence of its existence. For example, Hou (1995) (Hou, 1995) reviewed existing development methodologies for complex electronic systems in order to find a suitable lean hardware/software system development methodology for the U.S. department of defence, but found none were adequate. Similarly, Wirth (1995) (Wirth, 1995) ‘pleas’ for “Lean Software” but more as a solution to the problem of what he terms “Fat Software”. Morgan (1998) (Morgan, 1998) describes how he applied lean manufacturing techniques to software development, while Raman (1998) (Raman, 1998) within the Boeing company, is still asking “Is it feasible?”. A useful contribution was made by (Sutton, 1996). Sutton summarises 10 years of a lean initiative within the Lockheed Martin Aeronautical Systems which included the embedded software components. He provides us with some of the lean guiding principles they used but also gives specific examples of their lean software development practices, such as formal methods and formal verification to support their enterprise principle of “Optimal First Delivered Unit Quantity”. It is not until the mid 2000’s that we start to see a more holistic view of lean SD with concrete and more comprehensive software practices being suggested (Poppendieck and Poppendieck, 2003), (Poppendieck and Poppendieck, 2006), (Charette, 2007), (Hibbs et al., 2009). However, academic and particularly empirically based publications continue to be difficult to find, and where they do exist tend to focus on small-scale experience reports (Middleton, 2001), (Middleton et al., 2005), (Robinson, 1997a). This gap in the literature, and industry interest in lean SD, therefore identifies an area which is in need of some clarification and particularly some guidance on implementation. 45
2.7.1 The Methods-in-Action Framework With the myriad of current software development methodologies and practices available, it becomes useful to have some mechanism by which to abstract, represent, and evaluate a particular methodology. One such framework is NIMSAD (Normative Information Model-based Systems Analysis and Design) (Jayaratna, 1986). Figure 2-10 shows the four elements of the framework – the ‘problem solver(s)’, the perceived ‘problem situation’, the problem solving process, and the evaluation of the three elements at three different time periods.
Figure 2-11 The NIMSAD framework (Jayaratna, 1986)
Particularly relevant to this study is the methodology context, since it is within the specific context of medical device software with its related regulatory controls, that we are exploring the SD process. The NIMSAD framework was modified by (Fitzgerald, 1995) to form a new framework representing a descriptive formulation of the systems development process. A subsequent refinement led to what is called the Methods-in-Action framework (Fitzgerald et al.,
2002).
Depicted
in
Figure
2-11,
the
methods-in-action
model
“…acknowledges the complexity and dynamic nature of the problem situation, that is, the methodology context” (Fitzgerald, 1995). It acknowledges the vital 46
contribution of people (Developers), since it is people not methodologies who create systems. It highlights the fact that despite the adoption of formalised or commercial methods, the actual enacted methods may turn out to be somewhat different, “Different developers will not interpret and apply the same methodology in the same way; nor will the same developer apply the same methodology in the same way in different development situations” (Fitzgerald, 1995).
Figure 2-12 The Methods-in-Action Framework (Fitzgerald et al., 2002)
The other important aspect to the methods-in-action framework is its recognition that methods may be put in place to fulfil overt rational objectives as well as covert political objectives, something which may also influence the enacted methods. The framework is therefore a very useful tool as a starting point for the conceptualisation of this research area given the importance of the context and
47
our exploration of formalised/planned versus lean/agile methods within this context.
2.8
Discussion
Representing the software development methodology spectrum as in Figure 2-12, adapted from (Boehm, 2002), we can view the different methodologies as being lightweight towards the left (lean or agile) or heavyweight towards the right (planned based). Within the safety-critical software domains, including medical devices, the methodology of choice is typically heavy weight with much focus on detailed planning and the generation of large amounts of documentation to satisfy the regulations.
Figure 2-13 The SDM Spectrum (Adapted from Boehm, 2002)
Many agile SD practices are focused at low-level practices and, as is the case in this thesis, can be viewed as supportive of a higher-level lean philosophy. Consequently, the lack of empirical research into lean software development suggests that the existing agile literature is a good place to begin our investigation. While many other industries are seeing large growth in agile adoption, it is interesting to explore why the same level of adoption is not occurring in the safety-critical ones, and to pose the question as to how these companies might 48
move their development approach to the left of the spectrum and avail of the benefits other industries are enjoying. Consequently the topic of regulation, and in particular MD regulations, will be an area of investigation relevant to this thesis. While medical device standards are quite rigorous and require detailed record keeping, they are not necessarily prescriptive. For example, IEC-62304 defines the various software development life-cycle processes and activities for medical device software development. It acknowledges that there are different software development methodologies but does not stipulate which ones to use. The objective is that the processes, activities and tasks, as identified by the regulations, are being implemented. One particular aspect of regulated industries such as Medical Devices is the need for an auditable trail of documentation that links requirements to specifications to verification and validation, and this is typically where a lot of suspicion exists around the more modern software development methodologies such as lean or agile (Lindvall et al., 2002). The literature, however, does help us clear-up this misconception and provides us with some examples of how companies have satisfied these traceability requirements and still remained agile (Lin and Fan, 2009), (Rasmussen et al., 2009), (Rottier and Rodrigues, 2008), (Gary et al., 2011). This indicates that specific regulatory standards will be of importance to this research.
2.8.1 Safety-Critical Software Development Other safety-critical industries such as Aviation, have equally stringent regulatory watchdogs to ensure that the risk of injury to humans is minimal. In the U.S., the Federal Aviation Authority recognises the RTCA’s (RTCA, 2011) DO178B guidelines (RTCA, January 1992) as a means to evaluate software against government regulations. Similar to ISO 13485:2003 (ISO, 2003), which governs the quality management for medical devices, the standard does not impose any particular software development methodology but rather specifies objectives, descriptions of activities, and descriptions of evidence that must be satisfied. It 49
is therefore also useful to understand the successes and failures of lean and agile in such closely related domains. Within the aerospace industry for example, (VanderLeest and Buter, 2009) found that nearly all agile practices can be mapped to the DO-178B regulatory guidelines and yet the aerospace industry has been slow in adopting them. Similar mappings were performed (Chisholm, 2007), (Wils et al., 2006a), finding that while most of the DO-178B requirements can be mapped to XP, Scrum, Crystal practices, some are outside the scope of these methodologies and some need to be reinterpreted. It is particularly interesting to see the Crystal family of SDMs included by (Chisholm, 2007) in his conclusions, since it is suggested that they fall short when applied in a life-critical context (Abrahamsson et al., 2002). (Wils et al., 2006a) found that, similar to other embedded software domains, the further on in the life-cycle you are, the less agility can be maintained. The final stage of certification is where they suggest that the least amount of agility is possible. Interestingly, the Open-DO Initiative (Open-DO, 2011) (referring to the DO-178B guidelines) is calling for a more open-source approach to aviation SD. They state that: “By leveraging on lean approaches and agility we aim… to shift the focus of safety-critical software development to more continuous and incremental certification approaches”. Within these domains the literature has reports of successful agile implementations but, as with the embedded-software domain in general, there are caveats which have led to ‘flavours’ of agile methods being implemented. Cochlear (manufacturers of hearing devices), introduced some agile practices in 2001 (Salo and Abrahamsson, 2008). An initial tentative and cautious approach was followed by a full Scrum implementation. (Lin and Fan, 2009) describe their experiences developing software for a Digital Subtraction Angiography 4 medical device. They note that the most important thing for FDA approval is the need to perform both a formal review and approval steps. They implemented a 4
A type of fluoroscopy technique used in interventional radiology to clearly visualize blood vessels in a bony or dense soft tissue environment.
50
hybrid planned-agile methodology in order to achieve the benefits of agility while maintaining discipline around certain areas such as documentation. (Spence, 2005) brings us through their journey in implementing agile (XP and Scrum) in Medtronic, a company developing class III medical device software. They found that the practices of pair-programming and test-driven development provided early feedback and better quality. (Rasmussen et al., 2009) discuss the successful implementation of agile practices within Abbott’s diagnostic division, and conclude that: “…an agile approach is the approach best suited to development of FDA-regulated medical devices”. (Cordeiro et al., 2007) made use of a combination of XP, Scrum and Organisational Patterns to overcome system constraints and regulatory issues related to safety. There is a lack of clarity in the literature about how best to begin trialling agile development within safety-critical contexts. From a generic embeddedsoftware perspective (Kettunen and Laanti, 2005) developed a comparative process selection model, while more recently, (Srinivasan et al., 2009) proposed 6 recommendations when considering agile adoption in embedded systems development. One of their recommendations is to ascertain the organisational and project “homegrounds” - as per (Boehm, 2002) - and tailor the Agile practices appropriately. Interestingly, (McCaffery et al., 2008) developed a process assessment method for Automotive, safety-critical small and medium enterprises (AHAA) which can propose agile or plan-like practices to improve the software development process. However, they state that “…in the case of safety-critical automotive software development…the AHAA will only recommend plan-driven practices”. Of interest for this research therefore is how the choice of methodology selection is made at a management level. What are the technical or human considerations which affect that decision? Additionally, if a lean or agile approach is taken, how and what are the practices that might be implemented.
51
2.8.2 Regulatory focus on validation and verification Within the different safety-critical regulations there is specific reference to the validation and verification steps required. Testing is an integral component of this, and much has been published around testing of embedded software development in general (Seo et al., 2007), (Kim, 2008), (Masticola and Gall, 2008). However, the effect of regulatory controls on the process does not get explored. (Grenning, 2002), discussing XP for embedded software, suggests that XP’s focus on automating testing can benefit critical systems, but due to the requisite care needed, suggests that it should be tried and evolved further to meet the specific needs. To facilitate the requirements of safety demonstrations throughout the lifecycle, (Yoo et al., 2005), describing software development within a nuclear power plant, developed a formal specification language for nuclear engineering applications. It can be used to automatically generate executable code (albeit for Programmable Logic Controllers), which they say helps validate correctness. David Vogel, who has been at the forefront of software’s evolving role in medical devices tells us that specifically in relation to MD testing activities: “... preliminary testing during the implementation of the software is probably the most productive activity for finding defects in the software” (Vogel, 2010). There is some evidence which supports agile testing techniques within a safetycritical context. (Rottier and Rodrigues, 2008) report on implementing test driven development (TDD), but encountered problems with their dependence on manual testing due to strict regulatory standards. In order to overcome this they successfully automated the testing of the physical devices, however, this entailed a large upfront effort in getting the process established. Similarly, (Huffman Hayes et al., 2009) suggest that by building upon a TDD process, it should be possible to generate the requisite traceability artefacts required by the regulations as a direct by-product of the process. (Tsai et al., 2005) discuss software testing of embedded systems including issues involved with regression testing for re-validation when using agile 52
practices such as extreme programming. They tell us that regression testing is limited because it re-uses test cases which may not be applicable for new features unless the test cases are modified. Similarly if a program’s structure is drastically changed, many of the existing test cases may have to be updated. They describe a Pattern-Oriented Scenario-Based testing approach which supports a lean/agile process, and which reduced the testing effort in Guidant by between 25% and 90%. The essence of their verification pattern approach is to classify system scenarios into patterns, and for each scenario pattern develop a test script template to test all scenarios for that pattern. The issues mentioned above may be encountered if using an agile SD approach where change is expected, therefore the use of verification patterns may provide an alternative which assists the verification effort while keeping the overhead to a minimum. Similar issues are highlighted with refactoring - a process of changing a software system in such a way that it does not alter the external behaviour of the code yet improves its internal structure (Fowler, 2000). This, however, has the potential to invalidate earlier certification (Chisholm, 2007), or to introduce timing issues (Ronkainen and Abrahamsson, 2003). A workable configuration management system combined with relentless testing can help. (Rasmussen et al., 2009) encountered auditing issues with their detailed test cases, but by making them less prescriptive, testing became focused on the intent of the test as opposed to its step by step execution. Within the context of this research, therefore, verification and validation will be central, and it will be important to take into consideration the situational context such as project characteristics and software expertise and how they affect the choice of practices.
2.8.3 Hardware/Software interdependencies Software development typically moves at a faster pace than hardware development does, and this can cause difficulties for agile testing techniques (Grenning, 2002). (Masticola and Gall, 2008) suggest that there are in fact some disadvantages to having hardware in the test loop, such as: cost, schedule 53
delays, and resource bottlenecks. (Van Schooenderwoert and Morsicato, 2004) describe various testing techniques they employed, for example bracketing-out hardware related calls and replacing hardware inputs with specific hardcoded values. This allowed them concentrate on the software issues that arose instead of trying to troubleshoot hardware issues at the same time. (Mueller and Borzuchowski, 2002) when reporting on their experiences with XP say that hardware simulators, although requiring an initial upfront investment, allow the software to be tested independently of the hardware and under conditions difficult to create in the hardware. When looking at ways to improve the research and development phase of a company involved in mechatronics devices, (Pang et al., 2012) found that a systems design approach enables an efficient product design evaluation among engineers from differing backgrounds, while compressing the R&D project life-cycle schedule. Such techniques and mitigations allow the development process to continue in an lean manner, and according to (Huffman Hayes et al., 2009), could be used to automatically produce the necessary artefacts to assist in continuous monitoring of the development process from a regulatory perspective. This approach, while supportive of a continuous risk management process, will not completely satisfy the regulators who will expect a fully integrated end-to-end test of the finished product, however, the final test should be less error-prone and more a final confirmatory step. This suggests again that project characteristics may play an important role in
our
thesis,
but
additionally
suggests
that
the
management
structure/management ethos or team configuration may also be important factors.
2.8.4 Regulatory focus on full traceability From a regulation point of view, it is imperative that there is full traceability right throughout the development lifecycle. This can seem problematic from an agile perspective considering its preference for working software over comprehensive documentation. However, (Ambler, 2006) suggests that the 54
agile practice of single sourcing information greatly simplifies requirements traceability within regulated development. (Hibbs et al., 2009) and (Ambler, 2006) point to source control management (SCM) as being a fundamental best practice which assists traceability, while (Huffman Hayes et al., 2009) propose building upon the practice of TDD to produce a requirements traceability matrix as a direct by-product of the TDD process. By introducing a mechanism whereby acceptance tests include a direct reference to the requirements, a partial requirements traceability matrix is obtainable. (Rottier and Rodrigues, 2008) when implementing Scrum, used the practice of user-stories from XP to maintain traceability for regulatory compliance. Within the healthcare company Abbott, (Rasmussen et al., 2009) implemented agile on a pilot project in 2004. Each iteration delivered key artefacts which they used to demonstrate their regulatory compliance, but they note the need for slightly longer development iterations than the one to three weeks agile typically suggests. Due to the overhead involved in generating the requisite documentation, six weeks did not generate enough velocity, and more than eight weeks resulted in a loss of focus. (Lin and Fan, 2009) decided that in their situation a hybrid approach was best suited in order to comply with the regulations. They combined agile practices which helped reduce project risk through iterative development cycles, with more formal methods to ensure adequate documentation and traceability for the FDA requirements. Upfront planning of essential documentation needs and using a bespoke traceability template, were successfully employed but introduced more overhead than a typical agile approach would recommend. Maintaining traceability is a clear requirement for regulatory compliance. By following some well established best practices, such as source control and configuration management, this traceability effort can be supported. In a straight forward approach (Vogel, 2010) suggests that different but related documents should resemble each other structurally, thus making them easier to review and relate. Some evidence does exist in support of the use of lean and 55
agile practices to help achieve this, although out-of-the-box agile practices seem to need some form of tailoring before satisfying the requirements. Within the MD domain, the generation of a traceability matrix is needed, and some techniques exist for building upon existing practices to partially fulfil this requirement automatically. As a result, this need for traceability will be critical to our theory development.
2.8.5 Organisational Challenges Any lean or agile strategy can only succeed if the people involved are organised and motivated appropriately. It was in the pursuit of Japanese efficiency that American carmakers “... were finally able to admit that Toyota’s real advantage was its ability to harness the intellect of “ordinary” employees” (Hamel, 2006). (Ambler and Kroll, 2007) wrote a three part article for IBM, where they identified a collection of practices for lean governance of software development projects. Ambler also contends that in his experience there are quality-oriented agile development practises which are much better suited to regulated environments than traditional practices. For example, the XP practice of pairprogramming and a collaborative modelling approach results in a continuous review process which can provide more effective feedback than the typical formal review process. This will not eliminate the need for some level of formal review, however, those formal reviews should become much less onerous (Ambler, 2006). Giving people a sense of full involvement and joint ownership of a project is at the core of a lean approach. Lean encourages employees to take responsibility for their own work area and work processes. (Van Schooenderwoert, 2008), when implementing agile development, ran into difficulty with stakeholders wanting to “own” developers. This was overcome by management receiving specific agile training. She also reports on how it is possible to unlock and engage the intelligence of inexperienced workers instead of assuming you need very experienced embedded software engineers (Van 56
Schooenderwoert, 2006). (Spence, 2005) has a large narrative given over to “Making Allies and Friends“, in order to build support for process change. This was achieved by inviting in expert speakers, holding discussion groups, attending conferences, and the company’s internal technical forum which allowed for an education and information campaign. This is a recurring element of any lean or agile implementation. It can be particularly important in safetycritical regulated areas where, for example, getting the QA department onboard may pose difficulties, since being focused on the regulatory requirements they may resist moving from a tried and trusted process. Moving to a lean/agile environment requires a cultural transformation. (Mueller and Borzuchowski, 2002) tell us that if the right attitudes and management supports are not in place, the effort may be doomed from the start. In their case, the lack of higher management buy-in from the start saw the process falter as the project deadline approached. They nevertheless believe that with an open attitude towards embracing change, XP can be a productive and valuable methodology in embedded software development. From this evidence, an organisational component and a people component will be important within the conceptual aspect of this research.
2.9
Conclusion
This chapter presents the findings from both a traditional and systematic review of the literature thereby capturing a detailed understanding of the existing domain knowledge but also identifying some gaps particularly in relation to lean software development. Medical Device software development is a closely regulated industry, a consequence of the concerns for patient and user safety. The software component is playing an increasing role in modern medical devices and has evolved to the point where it can be categorised as a medical device in its own right. The medical device industry is becoming increasingly cost focused resulting in all areas of the industry searching for more cost effective ways of operating. This includes examining the software development process, which 57
up to now, has been characterised by a traditional plan-based methodological approach. While other industries have seen a significant increase in adoption of agile software development methodologies, the MD industry has not. The Agile Manifesto exists for over 10 years this year, and the popularity of agile methods is evident from the continuing activity from both industry and academia. There have nevertheless been some issues highlighted with agile approaches, for example their inability to scale to larger teams or their suitability for mission- or safety-critical applications (Lindvall et al., 2002), (Wils et al., 2006a), (Sidky and Arthur, 2007), (Kajko-Mattsson, 2008). Despite their growing popularity, agile methodologies have had limited success in penetrating the safety-critical domains such as medical devices. As the practice of software development continues to mature, lean software development is seen by some as the next evolution of agile software development. With medical device industry practitioners showing an interest in pursuing these more lightweight methodologies and lean philosophies, and the absence of guidance or practical models, there exists a need to develop a leaner approach to medical device software development.
2.9.1 Research Questions In summary, the literature review identified the following main knowledge gaps: -
A definitive description of lean software development and its differentiation from agile software development (Poppendieck and Poppendieck, 2003), (Charette, 2007), (Oakleigh Consulting, 2007), (Fowler, 2008), (Wang et al., 2012)
-
An understanding of the issues within contemporary medical device software development caused by the regulations (ANSI/AAMI/IEC, 2006), (ISO, 2003), (ISO, 2007), (Jordan, 2005), (Gee, 2009), (Salo and Abrahamsson, 2008), (Lin and Fan, 2009)
-
Guidelines for adoption of either agile or lean software development within the medical device domain (Cockburn, 2000), (Graaf et al., 58
2003b), (Boehm and Turner, 2003b), (Boehm and Turner, 2005), (Kettunen and Laanti, 2005), (Sidky and Arthur, 2007). These lead to the over-arching research proposal: “To investigate the applicability of a lean software development methodology while complying with the medical device software regulations”. This was broken down into the following four sub-questions which are the focus of this thesis: RQ1: What are the primary characteristics of software development within the medical device software industry? RQ2: How are the medical device regulations affecting the software development process within medical device companies? RQ3: How can lean software development be applied to the medical device software development process without compromising regulatory compliance?
2.9.2 A Conceptual Framework for Lean and Regulated SD The data generated by the literature review and mapping study was synthesised both quantitatively and in a descriptive narrative. Table 2-5 shows a sample of the recurring themes which emerged from the review analysis. The full table can be seen in appendix M. These were synthesised into a Conceptual Framework which incorporates a ‘lean thinking’ component. The resulting model was then used as a framework through which a process model for lean medical device software development was generated.
59
Table 2-5 A sample of the high level categories from literature review analysis
Recurring Themes Agile practices Audit trail Certification Collaboration Communication Continuous integration Customer access Documentation Employee Expertise Employee Handoffs Employee Motivation Employee Responsibility Employee Team configuration Employees organised and motivated Face-to-face communication Feedback loops Hardware Dependency Leaders not managers Leadership Lean governance Lean or agile Lean practices Legacy code
Category(s) People Regulations Lean People People, Project Management Software Practice People Software Practice People Project Management People, Organisation, Management Organisation People, Management People, Organisation, Management People People Project Management, Software Practice Project Management, People Organisation Lean, Management, Organisation Lean, Software Practice Lean, Software Practice Software Practice
60
Sources Beck, 1999
Jeffries et al., 2001
Vogel, 2010
Ambler & Kroll, 2007
Chisholm, 2007
Vogel, 2010
Vogel, 2010
Ambler & Kroll, 2007
Pham et al, 2007
Tabaka & Martins, 2008
Cusumano, 2008
Poppendieck & Poppendieck, 2006
Vogel, 2010
Paasivaara et al, 2009
Huffman Hayes, 2009
Lin & Fan, 2009
Drobka et al., 2004
Liker, 2003
Poppendieck, 2003
Poppendieck & Poppendieck, 2006
Ambler & Kroll, 2007
Hamel, 2006
Poppendieck & Poppendieck, 2006
Ambler & Kroll, 2007
Ambler & Kroll, 2007
Hoegl, 2005
Ambler & Kroll, 2007
Poppendieck & Poppendieck, 2006
Vogel, 2010
Ambler & Kroll, 2007
Vogel, 2010
Ambler & Kroll, 2007
Vogel, 2010
Ronkainen & Abrahamsson, 2003
Ambler & Kroll, 2007
Poppendieck & Poppendieck, 2006
Ambler & Kroll, 2007
Kotter, 1995
Ambler & Kroll, 2007
Reinertsen, 2009
Ambler & Kroll, 2007
Konrad & Over, 2005
Robinson, 1997
Reinertsen, 2009
Vogel, 2010
Graaf et al, 2003
Bearing in mind the need to make the framework concise yet useful, I refined the categories down to capture the most significant aspects. Figure 2-13 shows the three main sources of direct influence on the SDLC, identified as: Organisation, Software Management and People, each applying an influence on the SDLC in different ways. Although the boxes ‘Project Management’ and ‘Software Practice’ could be treated as separate entities within the framework, I have grouped them together under the single heading of ‘Software Management’ to signify their interconnectedness but also for the sake of maintaining a concise model. Figure 2-13 is followed by an explanation of each of the framework components and their relationships.
Figure 2-14 The Initial Conceptual Framework for Lean and Regulated SD
61
2.9.2.1
Regulations
At the top of Figure 2-13 we have the first of the specific contextual elements of this thesis, the regulations box. Why do we have regulations, why do we need them and how important are they in terms of the development of medical device software? According to (Campbell, 2004), without regulations the free market system we know today would not exist. He argues that the state, through its use of regulations enforces the free market conditions through mechanisms such as the legal system, the police force, and uniformity of weights, measures and currencies. This, he tells us, is simply a form of social organisation, and therefore supports the definition of regulations used in this study as being: “rules, principles, or conditions that govern procedure or behaviour” (The-Free-Dictionary, 2011). Regulations have been applied in most aspects of human life such as national and international law governing human behaviour, capitalisation requirements for financial institutions, enforced driving practices, and payment of taxes. Consequently there are those who argue that there is too much regulation and that even the existence of autonomous self-regulation is in fact questionable (Ryan and Deci, 2006). In addition, there are those that argue that little research has been done in assessing the adequacy of regulations in achieving their intended aims (Campbell, 2004). For example, regulations governing financial institutions such as the Basel Accord (Supervision, 2004) and the Sarbanes-Oxley Act (SOX, 2002), which are aimed at ensuring banks are adequately capitalised and that financial reports are trustworthy, did not prevent the global banking crises of 2007. In fact although ‘light-touch’ regulation is blamed in many instances for this banking collapse (Evans, 2011), (The-Irish-Examiner, 2011), a World Bank report has found heavier regulation to be equally associated with “informality and corruption” (The-World-Bank, 2004). There are however, different types or levels of regulation ranging from low (self regulation), medium (Government regulation), to high (Litigation) (Bush, 2007). It is only when a lower form of regulation is seen to be ineffective is it
62
raised to a higher level. For example, the Therac-25 radiation overdoses in the 1980s which marked an important point in the history of MD regulation. Such high levels of regulation which aim to ensure human safety are difficult to argue against. For example, the standards and recommended practices issued by the Association for the Advancement of Medical Instrumentation (AAMI), aspire to a “continued increase in the safe and effective application of current technologies to patient care” (ANSI/AAMI/IEC, 2006), a laudable objective. While there is little debate about the merit of such motives, the AAMI does however have an additional objective, namely “the encouragement of new technologies”. It can be argued that it has been by incorporating and combining new and different technologies that the Medical Device industry has flourished. There is therefore a balance which needs to be struck between this drive for innovation and need for a high degree of device safety and the consequent higher level of regulation (Ziegler, 2006). Since regulations aim to govern the way people act, and since software development is largely comprised of human interactions between various stakeholders across one or more organisations, it can be seen how widespread the effects of software development regulation can be. Indeed as argued above, the regulations will specifically attempt to govern the activities of any party associated with the area. This is reflected in the Conceptual Framework by indicating a relationship between the regulations and all the influencing categories. The arrows emanating from the regulations box are shown leading to the three SDLC influences, indicating that compliance to the regulations must be addressed at multiple levels and within multiple contexts. It is therefore these specific influences, mandated by international standards and guidelines, which make the medical device software development process unique. The IEC 62304 standard was specifically created to govern the Medical Device software life-cycle processes, and IEC 80002 which provides guidelines on how to apply risk management specifically to medical device software in accordance with ISO 14971 (Application of risk management to medical devices). For example, within the MD regulations, the
63
international standard ISO 13485:2003 calls for a documented quality management system. In large companies, that will be satisfied at multiple levels. Firstly, the organisation’s management has a responsibility to establish a quality system, and may develop and publish corporate policies and procedures which describe such a system at a high level. This is then complemented by specific standard operating procedures (SOPs) at the business unit level, followed by software development process documents detailing the software practices which the developers will perform. Lower level processes which align with higher level ones, demonstrate a coherent and unified approach across the whole organisation to addressing the regulatory requirements. 2.9.2.2
The Lean Lens
The next component of the model is the ‘Lean Lens’, positioned between the regulations box and the three SDLC sources of influence. The significance of its position here is to identify lean as more of a philosophy or set of guiding principles which influence the way in which the regulatory requirements are interpreted and satisfied within the different contexts. ‘Lean thinking’ is based around a set of underlying principles, as described in Chapter 2, section 2.4.2. A common theme within the lean literature is that the people in the workforce are a major source of creativity and best positioned to improve the processes they are involved in. However, while the regulations may seem to want to restrict what certain people do, for example who must approve a software release or who must perform the validation activities, lean aims to empower the workforce through ensuring adequate training (Drobka et al., 2004), delegation of authority (Poppendieck and Poppendieck, 2006), and self-organising teams (Ambler and Kroll, 2007). The conceptual model identifies this conflict with the arrows from the regulation box passing through the lean lens box, entering red but exiting yellow, before proceeding to the SDLC influencers. From a software development practice perspective, there are a number of compliance considerations to be addressed during the actual writing of the MD
64
code. One of these is the need for formal software unit verification as mandated by (ANSI/AAMI/IEC, 2006). While testing is a key component of verification, it is considered an imperfect activity (Vogel, 2010), and consequently there can be a tendency to overdo testing in an attempt to ensure compliance. However, by applying a lean lens to this requirement, we can discern that what is important is the intent of the requirement, and providing evidence that the process meets that intent. How this is achieved is then a matter dependent on the context (complexity, type of device etc.) and therefore allows for tailoring of the testing process. For example, Abbott Diagnostics had regulatory compliant software processes in place but they had identified issues with their approach to testing (Rasmussen et al., 2009). They found that focusing on the intent of test cases as opposed to their step-by-step execution, resulted in more efficient test case development, an increase in first pass acceptance by the auditing function, increased code coverage due to alternate execution paths being exercised, and a level of immunity for the test cases to user interface changes. The significant point here is that by looking at a particular process through a lean lens the process can be improved while the outcome remains in compliance. 2.9.2.3
Organisation
The term ‘Organisation’ can be a nebulous term and so for clarification the following definition is used in this thesis: “[An] organisation is a complex social system and is the sum of many interrelated variables. The operations of the organisation are influenced by the external environment of which it is part” (Mullins, 2004). The relevance within the Conceptual Framework is that the organisation influences the software development process by defining development tasks and delegating roles (such as developers and testers), responsibilities (such as project managers and validation engineers), and authority (such as approvals). In addition, its relationship with the surrounding environment is an important aspect relevant to the model as identified by the line emanating from the regulations box. The regulatory context will influence the organisation’s ethos
65
in terms of ensuring safe and secure software, the workforce’s attitude to risk management, and their sense of responsibility. Although I am concentrating in this study on the influences on the software development process, the definition also indicates the relatedness between the organisation and the other two influencing categories. This is something which became enhanced as the research progressed and is described in more detail in the Conceptual Framework. From a lean perspective, (Ambler and Kroll, 2007) suggest that good lean governance is essential for a successful process. Lean governance can be summarised as providing all the necessary organisational support such as appropriate infrastructure (tools and methods), communication/collaboration approaches, mechanisms and incentives to support knowledge transfer and sharing, and the management of culture change, which is critical since it emphasizes the ‘soft factors’ governing organisational work (Srinivasan et al., 2009). Within a distributed team environment, these organisational supports can be seen as even more important. (Cusumano, 2008) suggests that commonly accepted development processes are essential, and include: practices which allow for parallel development of uncoupled modules, strong project management tool support for daily builds, continuous regression and integration testing, version control and configuration management. These guidelines and supports originate at an organisational level and often require significant investment. To protect that investment, an organisation will often seek to employ a formal software development methodology which is future proof (teachable), provides consistency, generates explicit deliverables, and provides an engineering-like development discipline (Roberts et al., 1998). The organisational influence plays a crucial role when it comes to the implementation of such a software development process or implementing changes to an existing process, for example, following the entrance into a regulated industry. Such process changes may affect the way people do their jobs or indeed the job descriptions themselves, and since changing work practices is not a trivial undertaking, it should be dealt with like any
66
organisational transformation (Kotter and Schlesinger, 1979), (Small and Downey, 2001). Leadership is an especially important attribute, and in a transformation process even more so. Especially within larger organisations, the need for a strong guiding hand is required (Kotter, 1995). This need manifests itself at a number of levels, for example, helping develop and drive a sense of urgency, creating and communicating the corporate vision, removing obstacles, and arbitrating on issues. Additionally, certain important leadership roles and responsibilities may even be mandated by the surrounding environmental context. SOX regulations, for example, hold the senior executives as individually responsible for the accuracy of the financial reports. Similarly, MD standard ISO 13485:2003 holds “Top Management” responsible for “evidence of its commitment to the development and implementation of the quality management system” and for the implementation of an ongoing risk management process. Consequently, the formation and implementation of software development processes will reflect a level of influence as exerted by these key roles within the organisation. It is important to realise the role the organisation plays in the evolution of the software policies and procedures. No matter how well designed and executed a process implementation is, it will be received with mixed opinions. It is incumbent on senior management to solicit this feedback and develop employees who embody and promote the necessary work practices in order to improve. One such process improvement approach might be to consider the implementation of lean or agile software development methodologies (Konrad and Over, 2005), (Petersen and Wohlin, 2010). Such an undertaking can have a significant effect of the software process and therefore requires some careful planning and execution (Boehm and Turner, 2005), (Smits and Rilliet, 2011). However, from a regulatory perspective things become a little clouded when the adoption of such SDMs are suggested. Particularly from a safety-critical perspective, a fear exists that an agile or lean approach will not demonstrate the level of rigor required by the regulations. Consequently, organisations are
67
tending to avoid these SDMs and instead adopt a far more heavy planned-based approach. What clearly emerges from the literature is that any lean or agile software development strategy can only be successful if the people doing it are organised and motivated appropriately. In the pursuit of Japanese efficiency “...it was only after American carmakers had exhausted every other explanation for Toyota’s success – an undervalued yen, a docile workforce, Japanese culture, superior automation– that they were finally able to admit that Toyota’s real advantage was its ability to harness the intellect of “ordinary” employees.” (Hamel, 2006). This harnessing of the workforce potential is the responsibility of the organisation’s leadership, and “with people, the things that matter are skill, pride, expertise, confidence, and cooperation” (Poppendieck and Poppendieck, 2006). All these attributes must get adequately addressed in some fashion in order for the process itself to function optimally. It is also important to continuously review the adequacy of such measures in satisfying individual needs which typically change over time5 ((Mullins, 2004) Page 481). Merged with this, the regulations will again influence these decisions. SOX regulations, for example, require that employees be provided with the appropriate information so they fully understand their responsibility in following compliant procedures. Specifically in relation to the MD industry, the regulations stipulate that the organisation’s leadership must ensure the workforce is fully informed about the regulatory context in which they operate, and any changes to the regulations must be disseminated. They must provide the appropriate systems and support processes for the workforce in order for them to be able to perform their functions in compliance with the regulations.
Maslow’s hierarchy of needs is typically depicted in the form of a pyramid showing 5 layers indicating different personal needs starting with the basic physiological at the bottom to self actualisation at the top. The pyramid shape indicates a thinning out of needs as a person progresses up from one level to the next. 5
68
2.9.2.4
Software Management
The software management box incorporates the overlapping activities of managing the tasks, resources and schedules, combined with the specific practices (many of which are of a technical nature) needed to perform the various activities within the SDLC. This category naturally influences the SDLC since it will be within these competencies, capabilities and situational contexts that the SDLC will need to be framed. For example, the technical nature of the product will automatically dictate the type of skills required and the type of development environment needed. The availability or lack or availability of these resources will shape the resulting SDLC (Kettunen and Laanti, 2005). In addition, the existing technical infrastructure will go some way towards assisting or hindering the adoption of a specific SDLC approach. For example a company which uses a language not conducive to an object oriented approach (Poppendieck and Poppendieck, 2003) (such as earlier versions of Visual basic, Fortran and Pascal), may have difficulty in following an SDLC which calls for such practices. Similarly, unless some investment is made in additional hardware and/or software, an SDLC which promotes test driven development (Beck, 2002) and continuous integration (Hibbs et al., 2009) is unlikely to be proposed where the environment is not adequately equipped. Different people and organisations approach project management of software development differently. Depending on the perceived importance of the software, for example is it seen as a strategic competitive advantage (Porter, 1998), (Prahalad and Hamel, 1990), or merely an asset to be managed (BenMenachem, 2008), (R, 2011), the SDLC will reflect this. A company which sees the software as being strategically important may also be more supportive of pursuing lean or agile approaches such as iterative development (Rasmussen et al., 2009) and/or allowing closer contact between developers and end users (Rottier and Rodrigues, 2008) in order to improve that key process area. Similarly the value placed on capturing and learning from the knowledge generated during a project will shape the process and influence the design of the
work
environment
and
specific
69
activities.
The
varying
interpretations/understandings of knowledge management can lead to many different forms of information capturing and sharing reflected in the SDLC (Reich, 2007). This positioning of the software effort relative to the rest of the organisation is therefore an important influencing factor. One other area of software management found to have a direct affect on the SDLC process is how progress and success are monitored and measured. There are very different approaches to these ranging from the heavy weight micromanagement of resources to ensure a strict schedule is maintained, to the selfdirecting and self-managing teams promoted within the agile methodologies. In the former, a detailed list of activities and resource assignment is maintained with close scrutiny of adherence to schedule. Any slippage to the pre-decided dates is a sign that the project is not staying on track and some form of intercession is required. The use of defined deliverables such as source code and supporting documentation are often used to signify the completion of a task or phase. The latter is where responsibility for scheduling resources and tasks are delegated to the development team, and customer participation and feedback is used to gauge how well things are progressing and when success can be proclaimed. However, within this context of project management, software development can not solely by described in terms of a finite list of sequential (or parallel) process steps, but includes intricate personal, interpersonal and situationally dependent activities which often lead to unpredictable events. Project estimates are often inaccurate, tasks are performed to varying degrees of quality, requirements and technologies change, and people leave (Herbsleb and Grinter, 1999), (Yabuuchi et al., 2006). The SDLC will therefore reflect to some degree the predominant management ethos within the organisation (Conway, 1968), however it must also allow for these unpredictable eventualities. When considering this component from a MD regulatory perspective, it can be observed that the effects are translated into the development of controls and guidelines for specific activities, some of which may be viewed as restrictive. For example, ISO 14971 describes how risk management has to be applied to
70
medical devices, and from a project management point of view it means incorporating “... an ongoing process for identifying hazards associated with a medical device” (ISO, 2007). This has natural consequences to the flow of a project, requiring additional activities be performed and recorded, and unavoidably increasing the cost of the project. For the software practices in particular, the regulations can seem to be prescriptive, in terms of requiring robust verification of code supported by documented test results and traceability between requirements, code and tests. For example, IEC 62304 calls for the following acceptance criteria to be specified and satisfied as part of the software unit verification if present in the design:
Proper event sequence Data and control flow Planned resource allocation Fault handling Initialisation of variables Self-Diagnostics Memory management and memory overflows Boundary conditions
While code verification is an accepted function of the developer, the execution and preparation of documentary evidence to this detail, required to satisfy the regulations, is something that may seem burdensome to some developers. As an example, the MD regulations hold risk management (RM) (safety risk as opposed to something like project risk) as a critical aspect of the product development: “The manufacturer shall establish, document and maintain throughout the life-cycle an ongoing process for identifying hazards associated with a medical device, estimating and evaluating the associated risks, controlling these risks, and monitoring the effectiveness of the controls” (ISO 13485:2003). The important words here are “throughout the life-cycle an ongoing process”. This means it is not something that starts and finishes during a particular phase of the life-cycle but is something that must be woven into the entire process. From a software point of view, RM is completely irrelevant without the context of the surrounding device or people and processes. A software failure alone
71
cannot cause harm. It is important therefore that the interaction between the software team and other teams, such as the hardware developers and quality engineers, is adjusted to establish an ethos of thinking about hazards in a cross functional, cooperative and holistic sense at all times. 2.9.2.5
People
An obvious influence on the software development process are the people who are involved with it on a daily basis. There are a myriad of areas which have been studied over the years surrounding the issues with software development due to the human condition. For example the fundamental problems associated with knowledge management in such a specialist environment continues right up to the present day (Robillard, 1999), (Damian and Zowghi, 2002), (Ye et al., 2008), (Levy and Hazzan, 2009), (Treude et al., 2011). How developers interpret, implement and communicate software requirements has a direct effect on the quality of the end product. Their ability to solve technical problems and their access to people and resources to aid in that resolution will determine how quickly and effectively an issue can be resolved. Another factor is the motivation of the software developers themselves (Burn et al., 1991),(Tessem and Maurer, 2007), (Beecham et al., 2008), (Sharp and Hall, 2009), (Franca and da Silva, 2010), (Treude et al., 2011), something considered to be the most impactful on productivity (Boehm, 1981). Indeed unmotivated developers can be seen as sources of negative productivity and a liability to a project’s success (McConnell, 1998). The distance between people is another aspect which effects how the software process will function. This distance is not only a consideration spatially, in terms of physical location (next room, next floor, other side of the country, other side of the world) but is also a consideration temporally (different time zones) and culturally (Carmel, 1999, Karolak, 1999, Herbsleb et al., 2001, Damian and Zowghi, 2002, Boland and Fitzgerald, 2004, Casey and Richardson, 2006, Richardson et al., 2008, Cawley and Richardson, 2010). Not
72
only is it the distance between team members that affects the SDLC but also the distance to (accessibility of) the customer or end-users. Since the need to decrease the length of the feedback loops with the customer is a key differentiator between traditional and agile SDMs, this distance can affect the success of implementing modern SDMs. Thus close and easy access to customers is advocated, which itself can affect how the development teams are configured and physically located. When close customer proximity is not practical, proxies-customers are used which again can affect the development process especially when the wrong people are chosen (Boehm and Turner, 2003b), (Spence, 2005). The effect of the need for regulatory compliance is very notable at a personnel level as it is precisely the human activities that are being governed. When moving from an unregulated into a regulated environment, unless the work processes are already fulfilling the regulations (experience suggests that this is unlikely), there is a need for peoples’ daily activities to change. For example, both SOX and MD regulations look for some level of independence in certain key areas. SOX looks for segregation of duties when it comes to code deployment or even access to a production system, while MD regulations expect independence between developers and validation engineers. When people are used to operating in an environment where issues can be fixed “on the spot”, these tighter controls can be very frustrating for both the technical employee as well as the person awaiting a resolution. An example of where companies are coming into contact with regulations for the first time is in the MD sector where the definition of a MD has recently been expanded (see section 2.5.2). This move into regulation typically means existing processes and work habits need to change, and as with any transformation process this in itself can be difficult. This transformation is not only a change in physical activities such as careful recording of test results or maintaining traceability, but equally requires a change in mindset across all stakeholders. The typical approaches to activities such as communication and knowledge transfer, where important ad hoc conversations go undocumented,
73
or an approval is given verbally, are no longer acceptable. In addition the required amount and detail of verification and validation may be much more than previously performed. 2.9.2.6
The Software Development Process
At the bottom of Figure 2-13, the software development process box depicts the defined process which produces the finished software product. This process, as we have seen, can take many forms but typically contains Primary activities (such as system and software requirements analysis, software coding and
testing,
software
installation),
Supporting
activities
(such
as
documentation, configuration management, verification and validation), and Organisational activities (such as management, infrastructure, process improvement, and training) (Standard, 1995). The business context will have a large influence on how this process is configured. Factors such as the intended business domain of the organisation (for example Medical Devices versus the Health and Fitness market) (Gee, 2009), whether the software will be embedded in hardware (Kettunen and Laanti, 2005) or whether it is considered a regulated device in its own right (McHugh et al., 2011), the project management approach (Barnett, 2009), (Casey and Richardson, 2006), the amount of legacy code (Boehm and Turner, 2005), the availability and competency of resources (Carmel, 1999), (Livermore, 2007), will all have a bearing on the design of the process and how it matures over time. One thing that appears common through the literature and particularly within the MD regulated software development, is that whatever the process is, it needs to be formally described and documented (RTCA, January 1992), (ANSI/AAMI/IEC, 2006). There is evidence, however, indicating that some companies are better than others in establishing and following a defined process (Feldmann et al., 2007). A second common characteristic is that the defined process should support establishing the safety and effectiveness of a product while ensuring the intended use is fulfilled without causing any
74
unacceptable risks (ANSI/AAMI/IEC, 2006). A third common characteristic is the generation and submission of objective evidence that good practices have been followed. This includes in particular the ability to demonstrate traceability between artefacts such as requirements, source code and tests (Wils et al., 2006b), (McCaffery et al., 2011).
2.9.3 The Initial SaLes-4-MD Model A second artefact to emerge from the literature review was the SaLeS-4-MD process model. This initial version of the model reflected the more practical concerns of software development under MD regulations, while at the same time incorporating the lean focus of this research. The Conceptual Framework indicates that adopting a lean lens has an effect on the other components of the model. Consequently any corresponding process model needed to reflect this. With this in mind each of the practices identified within the literature review was categorised/grouped within a ‘lean’ context. To facilitate this, the distilled list of seven lean principles, taken from the literature and shown in Table 2-6, was used. These seven principles were used to categorise the various practices, and only those practices which fit the definition of lean were included. For example, the lean principle of Identify and Eliminate Waste, a core lean concept, was examined using the definition of waste from Table 2-2. One form of waste within software development is the presence of defects (bugs) since they require additional effort to resolve them and so any practice which supports the identification, elimination, and/or prevention of software bugs would be placed into this category. The following practices were therefore included: -
Test Driven Development; which aims to catch bugs as quickly as possible in the development phase
-
Unit Testing by means of a testing framework; which advocates the use of a testing framework (many freely available) such as JUnit, CPPUnit, VBUnit, and which is used to develop test classes that can be collected together in a test suite.
75
-
Continuous Integration; which promotes the integration of software subcomponents into the full system on a regular if not continuous basis thereby identifying any integration problems as quickly as possible
Automated testing; which increases the frequency of test execution and when used in combination with automatic error notification to developer(s), speeds up the error detection and elimination process.
76
Table 2-6 The Distilled List of Lean Software Development Principles Lean Principle
Explanation
Identify and Eliminate waste
A core focus of lean, the aim is to reduce costs and make the products more cost effective (Poppendieck and Poppendieck, 2006). Waste can have multiple interpretations in a lean software development context (Table 22). This quest for waste elimination is on ongoing process which never ends and is therefore akin to continuous process improvement in order to seek perfection.
Build Quality In
From the Japanese word “Jidoka”, the aim is to have quality a focus of every step in the process thereby identifying problems early and preventing them propagating downstream through the system (Liker, 2003). Techniques such as the five S’s (Sort, Set-in-order, Shine, Standardise, Sustain) support this by helping to organise workspaces and removing clutter (Poppendieck and Poppendieck, 2006).
Create Knowledge
According to (Poppendieck and Poppendieck, 2006) “Those organizations that will dominate this century are those that shift to a focus on knowledge”. This knowledge comes in many forms, much of which is not written down. In addition documentation can be slow to generate and often not used. Lean software development suggests generating knowledge through techniques such as fast feedback through iterative development (Hibbs et al., 2009), face-to-face communication (Levy and Hazzan, 2009), (Poppendieck and Poppendieck, 2003) and work flow visualisation (Anderson, 2010).
Defer Commitment
Postponing firm decisions, discovering constraints, and then filling in the detail is a standard heuristic to solving problems (Thimbleby, 1988). In lean software development the key is not to commit to something (e.g. an architectural design) until you have enough information to make the right commitment. It is however imperative that the decision is made before it is too late (Poppendieck and Poppendieck, 2006).
Deliver Fast
Delivering working software quickly is a sign of a capable development process (Poppendieck and Poppendieck, 2006). Typical of most agile methods, short iterations assist fast delivery, and consequently identifies issues quicker, shortens the feedback loop, and helps manage and prioritise the customer requirements (Hibbs et al., 2009).
Respect People
Summarised as a culture of mutual respect and trust, open and honest communication, and synergistic and cooperating relationships of stakeholders (Oppenheim et al., 2011). This is achieved through training and developing employees, working closely with suppliers and partners, making decisions slowly by consensus (Liker, 2003). Self directing teams delegates authority to those who are best placed to perform it (Poppendieck and Poppendieck, 2006)
Maintain Flow
Creating flow means linking together operations that otherwise are disjointed (Liker, 2003). This “pull” approach controls how much work is in progress and therefore only allows as much work as can be handled. In lean software development, methods such as Kanban (Anderson, 2010) and Scrumban (Hiranabe, 2008) follow this approach. Techniques such as avoiding sub-optimisation (Poppendieck and Poppendieck, 2003) and managing queues and variability (Reinertsen, 2009) can be used.
77
Another type of software development waste is the development of Extra Features/Code, where either unrequested features are added to a product, or unnecessary code has been used to develop a particular feature. Therefore practices which assist in keeping this type of waste to a minimum are placed in this category. The following practices were therefore included: -
Prioritise requirements; which dictates what features should be worked on and in what order, ensuring that the highest value features are developed first. It also supports the possibility of low-value features being dropped from the requirements as the project draws to a close thus reducing the overall effort
-
Clearly state the customer value of each feature; only features with clear customer value should be worked on, which will also help eliminate unnecessary overhead associated with developing the associated test cases
-
Follow a YAGNI (You Ain’t Going To Need It) approach; which assumes a minimalist approach to writing code
-
Develop in short iterations; thereby increasing feedback from the customer and limiting the effect of any over production
-
Only develop for the current iteration; related to the previous practice this helps limit the possibility of developing more features than is required
-
Refactor; which is the practice of reviewing and simplifying existing code.
Another example of this categorisation process was the assignment of practices to the lean principle of Build Quality In, one of the pillars of the house of lean (Jidoka). This principle looks to make quality an integral part of each step in the software development process as opposed to a specific activity which occurs only at the end of the process. This is something particularly relevant within the MD domain. Quality within the SD process can be seen as: building confidence in the development process, the clarity with which the
78
software process is defined and communicated, and the robustness of the SD process in terms of areas such as testing and risk analysis. Some of the lean practices included in this category were: -
Component reuse; existing code which has already been validated, builds confidence in the code base
-
Integrated problem solving; many issues arise at the interfaces between groups and so this practice advocates involving the relevant groups in a collaborative effort to solve issues
-
Root cause analysis; whereby when a problem is identified, the underlying issue should be fully clarified and a resolution applied which will prevent that issue from re-occurring. An example technique would be the 5 Whys, whereby when an issue is identified, the question “Why?” is asked and continues to be asked of the subsequent answer (5 times is suggested) until the root of the problem is identified.
-
Automated testing; thus catching defects as quickly as possible and removing them before the software is released
-
Maintaining code clarity by: o o o o o
-
Refactoring Easy to understand naming conventions Common programming languages Simple notation Clear focused in-code comments
Continuous integration; identifies integration problems as quickly as possible
-
Embedding a mindset of compliance in the process through training; thus making it second nature in everyone’s daily activities and therefore improve the quality of the end product
-
Using established risk assessment techniques; such as Failure Mode and Effects Analysis (FMEA) or Fault Tree Analysis (FTA)
Figure 2-14 shows a partially expanded view of the final categorisation using the Mindmap tool (Sciplore, 2011).
79
Figure 2-15 Mindmap Diagram of Categorised Lean Principles and Practices
80
Returning to the Conceptual Framework we see how it indicates the effect regulations have on the other components and how they subsequently influence the development process. Having identified and categorised the lean practices, I then examined each one within the literature to see how they can be applied specifically within the context of a MD domain. Any issues that might need to be addressed in order for them to be applied, and what techniques (if any) could be used to overcome them were added to the data collection. Table 2-7 shows a section of the matrix produced. The complete Version 1, which contains 162 entries, can be seen in Appendix K. As an example, the first row, relating to the principle of eliminating waste, identifies the lean practice of finding defects early through the use of a Test Driven Development approach. However, within the context of MD software development, where there are both hardware and software components to the device, the schedule of the hardware group may not align with the delivery schedule of the software. In such a case a TDD approach will experience difficulties in testing those aspects of the software which interface with a hardware component which has not yet been developed. Nevertheless, according to (Cordeiro et al., 2007), (Van Schooenderwoert and Morsicato, 2004), and (Mueller and Borzuchowski, 2002) by utilising techniques such as mocks and stubs, commenting out hardware calls, or simulating the hardware in software, a TDD approach can still be used in this domain. Row 8 of the model again grouped under the lean principle of eliminating waste, refers to the practice of continuous integration. Continuous integration can have a number of benefits from a lean perspective, for example, catching integration issues sooner, avoiding a problematic ‘Big-Bang’ approach and providing fast and important feedback to the hardware team when the integration includes a hardware component. An issue identified within the literature with this was that the regulator will nonetheless look for evidence of final and successful full system integration. Although the model does not offer a mitigation for this, (Rottier and Rodrigues, 2008) suggest that by following a
81
continuous integration approach, the final system integration should be much faster and less troublesome. Row 10, also grouped under the lean principle of eliminate waste, refers to the practice of test-driven development and the need to develop a test for each defect as soon as the defect is found and fixed, and thereby ensuring it will not go undetected in the future. An issue identified here is that it is necessary to ensure that each defect and fix is recorded and traceable back to a requirement. The model offers the suggestion practiced by (Lin and Fan, 2009) where they incorporate formal documentation into their process to ensure the required level of traceability is maintained. Through this mapping process it was observed that some SD practices were mapped to multiple lean principles indicating that multiple benefits could be accrued through their adoption. For example, the practice of utilising short iterations intuitively supports the lean principle to “Deliver Fast” but in addition it supports the lean principles to “Create Knowledge” and “Eliminate Waste” by increasing the feedback cycles between the project team and the customer.
82
Table 2-7 Sample of the SaLeS-4-MD Model Version 1
83
Following this categorisation it was useful to understand where, within a typical software development life-cycle, these lean principles and practices would be applicable. This would then assist in the design of the subsequent SaLeS-4-MD process model. Using the software life cycle process model as described by the international standard ISO/IEC 12207 (Standard, 1995), a mapping was performed between these lean practices and the corresponding process area as depicted in Figure 2-15.
Figure 2-16 Mapping of Lean Principles to ISO 12207 Life Cycle Process Areas
As an example of what a link between a principle and a process means we take a look at the lean principle of ‘Identify & Eliminating Waste’. Looking in detail at the specific waste of software defects the following specific practices from the model were identified:
Find defects early through the use of test driven development
84
Practice continuous integration
Develop a test as soon as a defect is found
Practice automated testing
For user testing focus on the intent of the test.
The first four practices all have a direct affect on the development of the software (process 5.3). For example following a test driven approach means that the developer must write a test before writing any code, and automated testing finds problems very quickly which the developer must then fix immediately but also write a test for. These practices however, also support the verification (process 6.4) and validation (process 6.5) areas. By continuously integrating the software units and carrying out automated testing, a process of ongoing code verification is established. Also, since “... validation is dependent upon verification ...” (Vogel, 2010) page 83, these, in addition to focusing testing on the intended use of the software, assists in the validation effort. Looking at the waste of over processing, a typical example of which is writing documents that are not used by anyone, the following practices were found to correspond to the documentation process (6.1):
Achieve a balance to satisfy the different stakeholders. Although more of a guidance, this practice reflects the context specific nature of MD software development. The main tenet is that a broad based approach to producing detailed documentation at all process phases is unnecessary, and some documents require less detail.
Build on a TDD approach to auto-generate documentation and traceability. As suggested by (Huffman Hayes et al., 2009), by building on top of a process of TDD, the requirements of the regulations can be met by means of outputs automatically generated by the process itself.
This exercise was completed for all identified practices resulting in the mapping depicted in Figure 2-15. The shaded boxes highlight those process areas where lean principles have been found to be somewhat beneficial, while
85
the clear boxes show areas which could benefit from further research. It can also be seen from this, which lean principles are strongly represented, and which are less so. This provided a view of where in the life cycle these practices can be beneficial, but also provided data about which areas are not covered. It also proved useful when it came to narrowing the focus to a particular software development area when mapping to the Medi-SPICE reference model.
86
Chapter 3: Research Approach 3.1
Introduction
The objective of this chapter is to describe in some detail how this research was carried out. It begins with a brief look at modern research paradigms in section 3.2 Section 3.3 presents the research methodology adopted, and the considerations given in choosing the most appropriate empirical method for the objectives of this research. Specifically, a review of 5 empirical methods typically used is presented, followed by a description the model construction methods. Section 3.4 then presents the research process design. A detailed description of the constituent phases of the process and their execution is presented. Beginning with an overview of the process design, each core activity is described in terms of how it supported the iterative evolution of the Conceptual Framework and SaLeS-4-MD process model. Section 3.5 specifically looks at the consideration given to planning for and executing the data analysis steps within the research, while sections 3.6 and 3.7 deal with the concepts of research validity and quality as they relate to and were addressed during the research process.
87
3.2
A Philosophic Reflection
'Try all things, hold fast by that which is good' - Socrates The broad study of philosophy has changed significantly over the centuries resulting in a much more focused discipline: “… it began to shed one branch after another until now it is reduced almost exclusively to ontology, the science of being. If we really had to define it, we might say that philosophy questions the meaning of life” (De Crescenzo, 1990). There continue to be a myriad of philosophical standpoints. Guba and Lincoln (1994) (Guba and Lincoln, 1994), give an overview of the current main paradigms, namely: Positivism, Post-positivism, Critical Theory et al, and Constructivism. However, within the information technology domain, an additional perspective has been developing which is relevant for this work, namely ‘Design Science’ (March and Smith, 1995). From a paradigmatic perspective,
(Gregg
et
al.,
2001)
refer
to
this
as
a
socio-
technologist/Developmentalist paradigm, whereby reality is technologically created “Socially created multiple realities co-exist and are influenced by the need, acceptance, and/or comfort level of technology”. Vaishnavi and Kuechler (2007) (Vaishnavi and Kuechler, 2007) however, return to the term ‘Design’ and have added what they call ‘iterative circumscription’ to its epistemological perspective. Table 3-1 compares what (Vaishnavi and Kuechler, 2007) refer to as, the three “ways of knowing”.
88
Table 3-8 Philosophical Assumptions of Three Research Perspectives (Vaishnavi and Kuechler, 2007) Research Perspective Basic Belief Positivist Ontology A single reality Knowable, probabilistic Epistemology
Objective; dispassionate Detached observer of truth
Methodology
Observation; Quantitative, Statistical
Axiology: What is of value
Truth: Universal and beautiful; prediction
Interpretive Multiple realities, Socially constructed
Subjective (i.e. values and knowledge emerge from the researcherparticipant interaction) Participation; Qualitative. Hermeneutical, dialectical Understanding: Situated and descriptive
Design Multiple, contextually situated alternative world-states Socio-technologically enabled Knowing through making: objectively constrained construction within a context, Iterative circumspection reveals meaning Developmental Measure artifactual impacts on the composite system Control; creation; progress (i.e. improvement); understanding
According to (Vaishnavi and Kuechler, 2007), an essential part of a Design perspective is how reality (knowledge) is iteratively determined (or revealed) from the research effort, “Design science research, by definition, changes the state-of-the-world through the introduction of novel artefacts”. Through the process of generating artefacts, the researcher’s understanding is enhanced. However, the multiple ‘world-states’ of design science are not the same as the multiple realities of interpretative research. As with other design science researchers, I hold to a belief in a “single, stable underlying physical reality that constrains the multiplicity of world-states” (Vaishnavi and Kuechler, 2007).
3.3
Research Methodology
All research is about learning, and by making use of well established models of learning we can better examine its nature. For example, looking at the learning cycle developed by cognitive theorist David Kolb (Mullins, 2004)
89
(Figure 3-1), it can be seen how he emphasises the cyclical nature of learning and the importance of the synthesis between behaviour and evaluation of actions.
Concrete Experiences stage: Perception of the objective world
Active Experimentation Stage: Check out theories & hunches by testing in new situations
Observational & Reflective Stage: Beginning of internalisation
Abstract Conceptualisation Stage: Step back from reality and draw conclusions and generalisations
Figure 3-17 Kolb’s Learning Cycle (Mullins, 2004)
The Concrete Experiences Stage is where we learn from specific experiences and relations with other people. The Observational and Reflective Stage is where we look for meaning through observation from different perspectives and reflection on what we have learned. In the Abstract Conceptualisation Stage we logically analyse ideas and act based on an intellectual understanding of the situation. Finally, the Active Experimentation Stage is where we plan for and test our new understanding. What Kolb’s cycle demonstrates is that there is no end to learning, but merely another turn of the cycle. In addition it identifies the importance of reflection.
90
The cyclical nature of Kolb’s learning cycle lends itself well to an iterative approach to artefact development typical of design science research. From a validation perspective, this type of iterative development supports a verification process which aims to ‘check, confirm, make sure and be certain’ through ensuring they: “… are woven into every step of the inquiry to construct a solid product … by identifying and correcting errors before they are built in to the developing model and before they subvert the analysis” (Morse et al., 2002).
3.3.1 Design Science as a Methodology Information Technology Systems are “... complex organisations of hardware, software, procedures, data and people, developed to address tasks faced by individuals and groups, typically within some organisational setting”, and IT practice is concerned with the development, implementation, operation and maintenance of those systems (March and Smith, 1995). The potential for IT to dramatically affect organisational effectiveness has fuelled the interest of the research community. According to (March and Smith, 1995), this interest can be divided into two focuses, descriptive and prescriptive research. Descriptive research aids in our understanding of IT, and is knowledge-producing, where as prescriptive research aims to improve IT performance, and is knowledge-using. The authors suggest that the former corresponds to natural science, while the latter corresponds to design science. According to (Hevner et al., 2004), design science is “... fundamentally a problem solving paradigm”. They say that within the context of information systems “... it seeks to create innovations that define the ideas, practices, technical
91
capabilities, and products through which the analysis, design, implementation, management, and use of information systems can be effectively and efficiently accomplished”. These innovations, or ‘artefacts’, are the targeted outputs of design science, and can come in various forms. Four general output types of design science are described by (March and Smith, 1995). They are: (1) Constructs (2) Models (3) Methods, and (4) Instantiations. Constructs are the conceptualisation of the problem domain, defining the terms used when describing and thinking about tasks. Models represent a set of propositions or statements which express relationships among constructs. A model can be simply viewed as a representation of how things are. A method is a set of steps (an algorithm or guideline) used to perform a task. They are typically based on underlying constructs and a model of the solution space. Finally, an instantiation is the realisation of an artefact in its environment, sometimes even before the final constructs and models have been realised. An instantiation therefore demonstrates the feasibility and effectiveness of the models and methods they contain.
3.3.2 Artefact Construction Constructive/Design research should be based on rich empirical data (empirical foundation) and be evaluated and iteratively refined (empirical validity) (Karow et al., 2008). Bearing in mind the objective of this research, as
92
identified by RQ4, to find a way to apply lean software development within the MD domain, this research can be characterised as trying to “... create things to serve human purposes” (March and Smith, 1995). Consequently, it involved the development of a conceptual framework (a construct) and software development process model (model or method), focused on medical device software development. According to (Karow et al., 2008) the empirical validation of such models can be achieved by surveying domain experts or by applying the model in practise (an instantiation). In a similar fashion to (Reich, 2007), this study chose the former of these validation approaches.
3.3.3 Empirical Method Selection According to (Gray, 2004), in step four of the research process the researcher decides on their approach. However, choosing the appropriate research method is not a simple task because the strengths and weaknesses of the different methods available have not yet been well catalogued (Easterbrook et al., 2008). According to (Edmondson and McManus, 2007), the research approach needs to ‘Fit’ the research question, and they distinguish between 3 types of research depending on the state of prior theory and research, namely Nascent, Intermediate, and Mature (Table 3-2).
93
Table 3-9 Research ‘Fit’ (Edmonson and McManus, 2007)
State of Prior Theory and Research Research Questions
Nascent
Intermediate
Mature
Open-ended inquiry about a phenomenon of interest
Focused questions and/or hypotheses relating existing constructs
Type of Data Collected
Qualitative, initially open-ended data that need to be interpreted for meaning Interviews; observations; obtaining documents or other material from field sites relevant to the phenomena of interest
Proposed relationships between new and established constructs Hybrid (both qualitative and quantitative) Interviews; observations; surveys; obtaining material from field sites relevant to the phenomena of interest
Surveys; interviews or observations designed to be systematically coded and quantified; obtaining data from field sites that measure the extent or amount of salient constructs
Illustrative Methods for Collecting Data
Quantitative data; focused measures where extent or amount is meaningful
The practice of lean SD within the MD domain is clearly something which can be described as Nascent. This had been made evident through an initial literature review which: -
Identified the lack of clarity around the precise meaning of lean SD, and
-
Found very few publications on lean SD within the MD domain.
Accordingly, Table 3-2 indicates that research under these conditions is best conducted by exploring the area in an open-ended fashion using qualitative techniques: “Data collection may involve the full immersion of ethnography or, more simply, exploratory interviews with organizational informants” (Edmondson and McManus, 2007).
94
Because this study’s main research agenda was decomposed into three sub questions, it was possible to adopt different methods to answer each of them. As Table 3-2 indicates, the type of question will influence the research methods adopted. (Easterbrook et al., 2008) identify and compare five research methods, and similar to (Edmondson and McManus, 2007), suggest that it is important to clarify the research question when deciding on which to use. Each of the five methods was examined as a potential method for this research. Controlled Experiments A brief clarification of what constitutes controlled experiments is first required. According to (Basili, 1996) experimentation has many forms including controlled experiments and case studies. However, for this study a clear distinction is made between these two methods and the following definition is used: “A controlled experiment is an investigation of a testable hypothesis where one or more independent variables are manipulated to measure their effect on one or more dependent variables” (Easterbrook et al., 2008). Since this study is not attempting to test any hypothesis but rather gain an understanding of the SD process within a MD company, I eliminated this as an appropriate method. Survey Research Surveys are typically used to identify characteristics of a broad population (Punter et al., 2003), (Easterbrook et al., 2008). Since I have established that
95
this research area is considered nascent, it was unlikely that I would have been able to find a representative population to survey. A further problem with carrying out a survey is that there are recognised difficulties such as sampling bias, question design, inability of respondents to introspect on their own work practices (Easterbrook et al., 2008), and also the fact that due to the use of a structured questionnaire it: “... necessarily obtains a lesser depth and quality of information than a depth interview” (Hakim, 1987). Therefore in order to gain as much detailed information on my research area, I concluded that this approach would return too much superficial data and not give me the depth of understanding required. Ethnographies The study of culture within a community of people, or Ethnography, is centred around a belief that people interacting with each other over a period of time will evolve a culture (Quinn Patton, 2002). From a software engineering perspective: “... ethnography can help to understand how technical communities build a culture of practices and communication strategies that enables them to perform technical work collaboratively” (Easterbrook et al., 2008). An ethnographic approach could have been appropriate for this research, particularly since it is expected that the researcher brings with them some understanding or stand point while not imposing any pre-existing theories (Easterbrook et al., 2008):
96
“The ethnographer enters the field with an open mind, not with an empty head” (Given, 2008). However, typically in ethnographic research the researcher immerses themselves in the culture under study (Quinn Patton, 2002), typically living and working in the community for 6 months to a year or longer (Given, 2008). Such a long duration for this research was not feasible given the timeframe for completing the Ph.D. A second issue is that this type of research (Participant Observation) relies on the researcher finding a company willing to agree to such an on-site presence. For a number of reasons, such as, confidentiality, time commitments, work disruption, it was eliminated as a practical research method here. Action Research Action research takes a similar approach to ethnography in that the researcher again is immersed in the situation under study, but in addition, plans and introduces changes to the environment and evaluates the outcome of the changes. By engaging the people of the organisation in studying and solving their own problems (“Problem Owners”) (Easterbrook et al., 2008), it assists both the researcher in their knowledge acquisition and the practitioner through participation as co-researchers (Quinn Patton, 2002), (Given, 2008). However, the same issues that were identified with ethnography apply here. Additionally, it would be far more difficult to make changes to the SD process within a MD company due to the high level of risk involved. As was subsequently discovered by this researcher, any changes to process which
97
might possibly lead to a non-conformance in the eyes of the regulator is not easily tolerated by MD companies. This would have been a significant impediment had this research approach been followed. Case Studies (exploratory and confirmatory) Similar to ethnography and action research, case study research involves the researcher immersing themselves in depth in a particular phenomenon. However it distinguishes itself by ensuring the researcher remains un-involved and objective. Case studies can: “... offer in-depth understanding of how and why certain phenomena occur, and can reveal the mechanisms by which cause-effect relationships occur” (Easterbrook et al., 2008). Case studies do not however generate the same results as for example controlled experiments, but provide a deeper understanding of the studied phenomenon and are well suited for many kinds of research (Chen and Hirschheim, 2004), (Runeson and Host, 2009). This type of descriptive-process research (Easterbrook et al., 2008) gives an in-depth understanding of the phenomenon. Case studies can be exploratory initial investigations of a phenomenon, as in the second and third research questions, or they can be confirmatory – used to test existing theories – also a contributing approach to answering the third research question. (Easterbrook et al., 2008) tells us that case study research is most appropriate for situations where context is expected to play an important role. Certainly the context of the regulations which govern the SD of MDs and
98
the role they play would match that definition. Additionally (Yin, 2009) tells us that in general, case studies are the preferred strategy for answering “How” or “Why” questions, which is appropriate therefore in answering the second and third research questions. Case study research is considered an important and accepted means of research in its own right (Lincoln and Guba, 1988), (Siggelkow, 2007), (Eisenhardt and Graebner, 2007), (Yin, 2009) and appears to be growing in use (Chen and Hirschheim, 2004), (Höst and Runeson, 2007). Klein and Myers (Klein and Myers, 1999) tell us that: “One of the key contributions of the research methods stream in IS research has been the formulation of a set of methodological principles for case studies that were consistent with the conventions of positivism ... As a result, case study research is now accepted as a valid research strategy within the IS research community”. (Yin, 2009) makes the following two-fold technical definition of a case study: 1. A case study is an empirical enquiry that (a) Investigates a contemporary phenomenon in depth and within its real-life context, especially when: (b) The boundaries between phenomenon and context are not clearly evident 2. The case study enquiry (a) Copes with the technically distinctive situation in which there will be many more variables of interest than data points, and as one result (b) Relies on multiple sources of evidence, with data needing to converge in a triangulating fashion, and as another result (c) Benefits from the prior development of theoretical propositions to guide data collection and analysis
99
Although a major contributor to the case study as a research methodology, Yin would be considered positivist by nature, and so we need to be cognitive of that when taking an interpretative approach. The positivists put much value on building upon a priori theories and using the case study to prove/disprove those theories, interpretivists therefore must be weary of part (2c) of his definition. Interpretivists, although employing the same techniques during the case study, rely on the data to offer up explanations for what is occurring: “... the interpretive researcher attempts to derive his or her constructs from the field by in-depth examination of and exposure to the phenomenon of interest” (Orlikowski and Baroudi, 1991). A case study can be applied to an individual, group, or organisation (Yin, 2009), or in software engineering it could be a project, department, set of projects, an organisation etc. (Verner et al., 2009). For this research I am interested in the real-life process of software development under medical device regulations, and so this approach is appropriate. Single versus Multiple Cases Case studies, although intended to investigate the particulars of a single phenomenon, can include single or multiple cases. When examining any phenomenon, for example a software development project, it makes intuitive sense that by examining multiple occurrences of it, say a portfolio of software projects, the researcher can improve their ability to make causal inferences
100
from the data (Given, 2008). The evidence generated through this so called cross-case comparison technique is: “... often considered more compelling, and the overall study is therefore regarded as being more robust” (Yin, 2009).
However, the strength of the single case study is when the case in question represents a critical, extreme or unique, representative or typical, revelatory, or longitudinal case (Yin, 2009), or as (Stake, 1995) puts it: “A case study is expected to catch the complexity of a single case ... We look for the detail of interaction with its context. Case study is the study of the particularity and complexity of a single case, coming to understand its activity within important circumstances”. When we examine the context of the case chosen for this research, namely, lean software development within a Medical Device manufacturing plant, it is possible to regard such a combination as being unique (although some agile SD has been reported in this domain, no lean SD could be found), representative (from the point of view that every medical device manufacturer is governed by the same regulatory controls and so will be subject to very similar constraints), and revelatory (since very little is currently known or understood about it and so any research should illuminate the subject area for the community). For these reasons a single case study within a lean medical device manufacturing plant was considered sufficiently suitable.
101
3.3.4 Summary This research used a combination of different research techniques and methods. Because the overall research question was broken down into three sub-questions, it allowed for the adoption of different research methods. Table 3-3 shows an overview of the research questions and the approach taken to explore and answer them. Table 3-10 Research Approach for each Research Question Research Question What are the primary characteristics of software development within the medical device industry? How are the medical device regulations affecting the software development process within medical device companies? How can lean software development be applied to the software development process without compromising regulatory compliance?
3.4
Type of Research Exploratory
Design Science
Research Method -
Literature Review Mapping Study Industry Engagement
-
-
Iterative Model Development Literature Review Exploratory Case Study Exploratory Survey Iterative Model Development Literature Review Exploratory Case Study Expert Opinion
-
Conceptual Framework for Lean & Regulated MD SD
-
SaLeS-4-MD Process Model
Design Science
Outputs
-
-
State-of-the-art Knowledge Initial Conceptual Framework
The Research Process
The objective of this research was to probe the area of software development within the medical device industry. As the specific research questions began to be answered, a lean SD model and corresponding Conceptual Framework for lean MD SD were developed and evolved. These
102
models are structured around the need for maintaining regulatory compliance but allow for a lean approach to software development to occur.
3.4.1 Overview As shown in Figure 3-2 the research process can logically be separated out into four Phases. The core outputs of the research were a conceptual framework and a SD process model, and each of the phases successively contributed to the evolutionary development of those models. The initial Phase 0 involved the identification of the particular problem area. This was followed by a detailed literature review and the formulation of the specific research questions in Phase 1. Phase 2, building on the outputs from Phase 1, is where an iterative approach was taken to gradually build and refine the process model and conceptual framework, with each iteration building upon the output of the previous step but drawing on an additional key source of data. As the iterations progressed, the repeated process of reflection and sense making, helped keep the models grounded and in agreement with earlier work. A multitude of data sources and artefacts were used to progressively build up the models, for example,
company
documentation,
standards,
regulatory
documents,
interviews, work experience and expert opinion. This was augmented with an ongoing literature review. This construction process is presented in detail in Chapter 4.
103
Phase 3 concluded the research process, and involved the validation of the process model from Phase 2, and the subsequent generation of the final SaLeS4-MD model.
104
Figure 3-18 The Research Process
105
3.4.2 Research Scope Since it is important to make sure that the research is not too broad, and has a reasonable chance of completing, I explicitly looked at ways to bound it. To achieve this, I made use of the Conceptual Framework which emerged from the literature. As the research began to take a more defined shape, I refined the framework which helped better maintain the boundary around the subject matter. This is described in more detail in Chapter 7.
3.4.3 Research Validation “Validity depends on the relationship of your conclusions to the real world, and there are no methods that can assure you that you have adequately grasped those aspects of the world that you are studying” (Maxwell, 1996). However, in a commonsense way, validity is concerned with: “... the correctness or credibility of a description, conclusion, explanation, interpretation, or other sort of account” (Maxwell, 1996).
(Creswell and Miller, 2000) provide us with different approaches to validation within a qualitative research context. They suggest that different approaches are typically used when adopting different paradigmatic stances. Consistent with the paradigmatic stance of this research, the approach taken to validation was multi-pronged, and consisted of the following techniques: -
Multiple data sources, multiple methods (Triangulation)
-
Professional experience (Researcher reflexivity, Prolonged engagement in the field)
-
Participant feedback from the case study company (Member checking)
106
-
Mapping to an existing process model (quasi-Peer debriefing6)
-
Clear record of the research (The audit trail)
-
Full written narrative (Thick, rich description)
-
Feedback from experts in the field (Peer debriefing)
Participation of subject-matter experts is emphasised during reference model construction (Rosemann and Schütte, 1999) and demonstrated in the work of (O'Leary, 2010). Since part of the motivation of this research was to make it relevant and accessible to industry, this approach was employed on a number of fronts. Firstly, through discussions with the target audience, namely medical device companies, a high-level understanding of the issues encountered in their software development process was established. Secondly, during the model development, an exploratory survey of industry experts was used as a means to ‘check and see’ whether the emerging model was developing in the right direction. Finally, the SaLes-4-MD model was validated against a panel of domain industry experts through a process assessment and one-to-one interviews.
6
May also be referred to as “Enfolding Literature”, (Eisenhardt 1989)
107
3.4.4 PHASE 0 - “Defining The Problem Area” As presented in Chapter 1, my background in regulated software development suggested a number of areas which could benefit from focused research, something which helped focus the study: “Student’s proposals often seem to systematically ignore what their authors know from their own experience about the settings or issues they propose to study; this can seriously damage the proposal’s credibility” (Maxwell, 1996). In addition, by engaging with a world leader in medical devices, a more concrete focus was established. Some exploratory research was duly carried out and submitted for review to management. The subsequent report can be seen in Appendix J. Positive feedback led to an elaboration of the research problem and the formulation of possible research questions. 3.4.4.1
Phase 0 Outputs
During the process of establishing the research problem area, and discussions with ..., the macro research question was established, namely: How applicable is a Lean Software Development methodology in the medical device software industry?
3.4.5 PHASE 1 – “The Literature Review” A literature review and mapping study were performed as described in Chapter 2. This generated clarity around which lean principles and influences were most prominent, and resulted in the initial version of the conceptual framework which was focused around the most prevalent SDLC influences. 3.4.5.1
Phase 1 Outputs
108
In conclusion of phase 1, I had generated a Conceptual Framework showing the main influences on the software development process within a MD context and how the regulations and a lean mind-set affect these influences. In addition I had collated specific software development practices into an initial process model of lean software development for the MD domain. This was called Version 1 of the SaLeS-4-MD model.
3.4.6 PHASE 2 – “Iterative Model Development” Phase 2 is where the majority of the empirical research was carried out. This drew from several sources of data, requiring a mixture of data gathering methods. Each of these contributed to the model development process as depicted within Phase 2 of Figure 3-2. A brief description of each of these is now presented. 3.4.6.1
The ... Case Study Process
An important contribution to the model evolution was the industrial case study carried out within . Figure 3-3 illustrates the steps taken during the case study research, categorised as planning followed by executing phases with an equal level of importance attributed to both:
109
Figure 3-19 Case Study Process – Adapted from Verner et al., 2009
3.4.6.1.1
Case Study Protocol
An extremely important output from the planning phase is the case study protocol (Yin, 2009). It includes details of all aspects of the research, including: overview, background, research questions, design, case selection, data collection, and plan validity. In particular it is a living document which the researcher will revert back to/update throughout the case study and so ensure they remain focused (Verner et al., 2009). It is also a means to demonstrate the reliability of the research, since it is a record of exactly how the research was performed (Verner et al., 2009), (Yin, 2009). For this study, a case study protocol was created based on the protocol template as described by (Brereton et al., 2008). Most of their template elements were used, however, it was necessary to adjust the template to suit my particular case. For example a Conceptual Framework was included to maintain a lean focus, and a specific section titled “administrative concerns” was added in
110
order to clarify ethical, confidentiality, and legal concerns. The full protocol can be examined in Appendix E. 3.4.6.1.2
Engaging the Company
Following an initial on-site visit, I attended their two day internal software development process training course. This provided a firsthand detailed view of their internal processes, something which is particularly advantageous when carrying out a case study. I was also permitted to take away the course material, which included the full set of documentation detailing the development lifecycle of their equipment and automated systems. Such artefacts are the type of things which aid in developing a holistic view of a case study company and proved very valuable as a reference point during data analysis and theory building. Following management agreement to perform interviews with a cross section of the people involved in the software development process, a number of employees were selected who would be representative and also who would be knowledgeable/experienced enough to provide good input. Table 3-4 shows the people involved with some details of their work experiences: Table 3-11 ... Interviewees Job Title 1 2 3 4 5 6 7 8
Principal Engineer Software Developer Senior Software Quality Engineer Project Leader Project Leader Controls/Software Engineer Principal Process Development Engineer Controls/Software Engineer
111
Years Working 9 8 11 18 13 10 17 8
Years Working in a MD Context 9 8 8 7 7 6 6 4
3.4.6.1.3
Unit of analysis
Within case study research, deciding on the specific focus of the case, is “... a problem which has plagued many investigators at the outset of case studies” (Yin, 2009). (Miles and Huberman, 1994) describe the need for a central focus of a study but which has: “…a somewhat indeterminate boundary [defining] the edge of the case”. Within this case study, the unit of analysis was problematic on two counts: -
lean software development is a somewhat undefined phenomenon, and
-
lean software development can be applied to practically any aspect of the medical device life-cycle.
My research was focused on examining the structure of medical device software projects workflow, and the management and software development techniques used. The appropriate unit of analysis therefore, was the complete software development process with some necessary focus on particular software development practices. This allowed for the process focused nature of a lean philosophy to be incorporated, while more agile-like specific practices could be investigated at lower levels of detail. 3.4.6.1.4
Data Collection
Collecting case study data is a crucial activity, but according to (Yin, 2009), the following three principles should be followed: 1. Use multiple sources of evidence 2. Create a case study database 3. Maintain a chain of evidence
112
As the first principle suggests it is important to remember that there are many sources of data that can significantly contribute to the research. Yin lists many of the potential data sources, and in this case study, I made use of multiple sources of evidence which included both quantitative and qualitative data. These included:
Process documentation o Corporate o Site o Departmental
Project Documentation o Team sizes o Team configurations o Timelines o Effort estimations
Process improvement initiatives
Interview transcripts
On-site observations
Documentary evidence was available in many forms, and I examined all available material following a disciplined approach. The approach was to carefully read each document using the developing conceptual framework and ‘house of lean’ (Appendix E) as focusing tools, searching for key words or phrases which would add to my understanding of the phenomenon. For example, the initial conceptual framework generated from the literature review suggested that the regulations have an effect at multiple points within the
113
overall development process. Therefore while reviewing the various documents, I sought references which supported or refuted this claim. Additionally, one of the pillars of the house of lean, is the concept of Jidoka (building quality into the process). Consequently, I looked for any references within the documentation to areas such as process improvement, defect identification and defect prevention. It is important, however, to remember not to automatically take documents at face value. Documents are prepared for a variety of reasons and for a variety of audiences, and can also be biased by the originating author. For example, one interviewee had reported that due to the lean ethos within the company, any process improvement initiative had to be presented as if it was a lean initiative, even if it this was not the case. The consequence of this was that the project documentation had to be written up using the lean document templates, which could lead to a false interpretation of the project. 3.4.6.1.5
The Interview Process
Interview schedules were agreed and spanned three days. Three interviews were held on each of the first two days, with 2 more on the final day, intentionally spreading them out in order to make time for some data analysis between interviews and between visits. From a qualitative data analysis perspective this allowed knowledge gained from one interview to inform and evolve the questions and format for subsequent interviews. This technique, which forms the basis of a grounded theory approach, proved to be a very
114
useful exercise for more targeted questioning in subsequent interviews similar to what (Coleman and O'Connor, 2007) have reported. In line with the University’s ethical guidelines, each participant was provided with a participant’s information sheet outlining the study, details of the procedures that would be followed and various other considerations relevant to the participant. The document can be seen in Appendix F. Most case study interviews are of an open-ended nature (Yin, 2009). Therefore a semi-structured approach was taken to the interviews, which involved preparing a list of questions which were used as a guide for the interview. There were 10 pre-prepared questions, on a single A4 page in order to make the interview process as smooth and natural as possible. The initial questions were to situate the interviewee in terms of their role and work experience, and also as a way of getting the interviewees comfortable talking with me. For example the first two questions were: “What is your role?” and “How long have you been working? How long in a medical device context?” There was no particular order for the subsequent questions, but they were specifically formulated to probe the research areas of interest. For example, question 7 on the interview question sheet was: “How would your work practices be different if you were not working in a regulated environment?”
115
This was a very probative question aimed at soliciting information regarding their awareness/knowledge of the regulations, and also how they perceive the influence of regulations on their work activities. The final group of interview questions was: “With regard to the current processes: -
In your opinion what works well?
-
In your opinion what does not work well?
-
Is there anything you would change, bearing in mind the regulations?”
This had a two-fold objective, firstly as a means of giving the interviewee the chance to summarise a lot of what was discussed, but secondly as a mechanism to sense check later whether it supported or contradicted what was said during the earlier parts of the interview. The full list of questions can be seen in Appendix G. Performing the Interviews Each interview began with an introduction describing the researcher’s background and the research itself. The fact that I had an industrial software development background and also that I had attended their internal training course was positively received. As per the ethical guidelines, each interviewee was provided with (a) Participant’s Information Sheet (b) An Informed Consent Form (which they were asked to sign)
116
They were also asked for their permission to record the interviews to aid in the data analysis phase. Recording the conversations proved to be an extremely useful practice (Hove and Anda, 2005). All participants gave their consent to recording with the understanding that it was confidential. It is worth noting that the presence of a recorder on the table did cause some reaction: “Its like an interrogation here now” was one jovial comment made. While the practice of recording allows the interviewer concentrate more on what is being said, I also made use of memo sheets to jot down any thoughts which came to mind during the discussions. These memos proved a useful mechanism for making notes of topics or scenarios which should be followed up, and also fed into the preparation of subsequent interviews. Figure 3-4 shows a memo sheet from an interview with a principal engineer (the names have been obscured to ensure anonymity), with two of the notes indicating the need to follow up specific themes with other interviewees.
117
Figure 3-20 Sample 1 of an Interview Memo Sheet
Figure 3-5 shows another memo sheet from an interview with a software developer. The last note here recorded the thought I had about how the XP practice of pair-programming might assist in up-skilling the developers.
118
Figure 3-21 Sample 2 of an Interview Memo sheet
Each interview was scheduled for at least one hour, with 5 of the interviews lasting for 90 minutes or more. Following each interview, the recording was immediately backed up to a laptop. In order to maintain a chain of evidence, the recordings were given descriptive names, making them easily identifiable. The memo sheets from each interview were also given an identifying number (see
119
Figures 3-4 and 3-5) which corresponded with the recording name. This gave traceability between recordings and memos should any confusion occur. 3.4.6.1.6
Data Analysis
To analyse the interview data I carried out the following steps: 1. Verbatim transcription of each interview 2. Open coding of the transcription documents 3. Axial coding of the results Steps 2 and 3 were used as data reduction techniques to distil out the most important categories and concepts. Figure 3-6 shows a snapshot taken from the NVivo desktop application which shows highlighted sections of an interview transcript and the corresponding codes. The interview data has been obscured to ensure anonymity.
Figure 3-22 Snapshot of NVivo Application
120
Table 3-5 shows an extract from the coding matrix following the open and axial coding steps. The columns to the right indicate the number of different sources making up that coding combination and the number of individual references within those sources. This process served a very useful mechanism in identifying themes which potentially were highly relevant for theory development. Table 3-12 Sample of the Coding Matrix from the Data Analysis Process Axial Code Open Code Sources References Business Focus Cost 5 22 Business Focus Lean Focus Lean Focus
Time To Market Lean Forcing Lean
4 6 1
18 17 5
Organisation Organisation
Restrictive Policy Lack of visibility
2 1
2 2
People Factors
Dissatisfaction
8
93
People Factors
Job Description
8
51
Process
8
85
6
29
Process Improvement
Process Improvement Process Description Streamline
1
2
Process Improvement
Tailoring
1
2
Project Management
Project Management Requirements Regulatory Awareness Traceability
5
22
6 7
13 33
2
16
Tooling
5
26
Verification & Validation
5
25
Process
Project Management Regulations Regulations Software Development Practices Software Development Practices
3.4.6.2
Regulations and Standards
Core to any study of regulated industry are the regulatory controls which are in place at both national and international levels. Within the MD domain,
121
there are a number of such documents such as the U.S. FDA’s Quality System Regulation (FDA, April 2009) or the EU Medical Device Directive (EU, 2007) which are legally binding and must be adhered to by the affected companies. In addition to these regulations, there are a number of standards which if adhered to may satisfy the legal requirements of various countries for example: (ANSI/AAMI/IEC, 2006), (ISO, 2003), (ISO, 2007). Each of the relevant documents was read in detail searching for words and phrases which stipulated or suggested strongly a particular activity and how that may be performed. While there is a level of interpretation involved with these documents, this process suggested strong links between these regulations and how organisations might interpret them in order to be compliant. These links were then triangulated with the other empirical data to enhance the understanding and consequently to evolve the Conceptual Framework developed within Phase 1. 3.4.6.3
Enfolding Literature
As Version 2 of the SaLeS-4-MD model evolved, a general review was performed to assess utility of this artefact. This was achieved by presentations and discussions with other academics, researchers and an industry partner. In summary, the output from this process was that the model contained a lot of useful information but it was difficult to see how and where to apply the specific practices.
122
To address these deficiencies, I took advantage of existing peer-reviewed models to map the emerging model against. This process, described as Enfolding Literature “... enhances the internal validity, generalisabilty, and theoretical level of theory building from case study research” (Eisenhardt, 1989). Since the ultimate goal of this research was to improve the SD process, it made sense to look at existing process improvement frameworks. My involvement with the Medi-SPICE project provided the requisite direction for this. By adapting the SPICE framework (ISO/IEC, 2004) I was able to generate a highlevel process overview, while the Medi-SPICE reference model (McCaffery and Dorling, 2009) was used to frame the specific practices at a lower level. MediSPICE is based on the IEC 62304 standard (ANSI/AAMI/IEC, 2006) but also incorporates other sources such as Automotive-SPICE which itself has been internationally accepted. By incorporating these into Version 2 of the SaLeS-4MD model, a more intuitive and purposeful model was generated. This was once again reviewed by a number of third parties (academic colleagues), and subsequently deemed ready for final validation. 3.4.6.4
Expert Opinion
During this process I solicited feedback from a small number of independent software development experts. In a form of peer-debriefing (Creswell and Miller, 2000), I solicited their opinion of the emerging research data. An exploratory survey was developed which was comprised of 10 questions aimed at soliciting opinion on the core aspects and practices of the emerging model. Specific attention was given to try and avoid a simple Yes/No response, and
123
thereby evoke more thoughtful and rich replies. The survey was sent to three industry experts, two of which (MD1, MD2) had a MD software background, while the third (AV1) had an aviation software background and experienced in working in safety-critical regulated environments (Table 3-6). While AV1 was an experienced systems engineer, the input from his responses was not utilised due to his lack of MD exposure. A copy of the questionnaire can be seen in Appendix H. Table 3-13 Exploratory Survey Participant Details Survey ID MD1 MD2
Role Quality Engineer Global Quality Systems Leader
MD Experience 2 years 10 years
Other Experience 9 years in safetycritical (automotive) -
The survey results were mostly supportive of the model practices, but also provided some useful commentary which increased my understanding of the development process and the issues being encountered. This input and how it influenced the research is detailed in Chapter 4. 3.4.6.5
Phase 2 Outputs
In a similar fashion to Phase 1, this Phase generated two primary artefacts as outputs. The first was an updated Conceptual Framework for lean medical device software development. An evolution of the input from Phase 1, the initial Conceptual Framework evolved into a framework based around situated lean SD practices.
124
The second artefact from Phase 2 was an updated version of the SaLeS-4-MD model. With additions and modifications made as necessary, the model was grounded in the data generated from the empirical research and had undergone an initial validation process.
3.4.7 PHASE 3 – “Validation” Although validation is something which was an ongoing concern throughout the research process (section 3.4.3), I make specific mention here to the final validation process which concluded the study. Since it is not possible to achieve 100% accuracy within any model, the validation process was a “… process of ensuring that the model is sufficiently accurate for the purpose at hand” (Robinson, 1997b). 3.4.7.1
Expert Feedback
In deciding on how best to perform a validation of the final model, I examined the methods employed in similar research. In an ideal setting, carrying out action research (Lewin, 1946) by implementing the model and empirically evaluating it against the existing methodology would have been the preferred option. However, this was considered to be too costly to achieve, mainly in terms of time needed. An alternative validation process was therefore to seek an assessment of the model by experts in the domain. This approach has been shown to be a valid and effective form of validation (Emam et al., 1996), (Beecham et al., 2005), (Conboy, 2006), (McLoughlin, 2010), (O'Leary, 2010) and something which was
125
much more likely to be achieved within the time frame available. I adopted a two pronged approach to seeking this expert opinion. A Process Assessment A process assessment was conducted as part of the Medi-SPICE research project on a Medical Device software company. By incorporating specific questions into the process assessment questionnaires, I was able to solicit input about the various practices derived from my model. As a member of the MediSPICE research team, I was involved in developing the assessment questions and was able to formulate specific questions which were used to verify the SaLeS-4-MD practices. For example, the following question was added to the Software Construction assessment questionnaire: “Do you differentiate between R&D and commercial development phases?” To compliment this, a clarifying statement was included on the interview questionnaire as a note to the assessor, for example: “(R&D phase not governed fully by the regulations. Lighter process possible there)”. This would then clarify that from a lean perspective there is a possibility to do things with less effort during the R&D phase, something the assessor could then use to assess the company’s response and possibly probe for further clarification.
126
One assessment was carried out with Bio-Medical Research Ltd., a company with over 30 years experience developing medical grade products. There were three assessors present during the assessment, one asking the questions, while the other two researchers took notes. Following the assessment, which was conducted over two days, the research team convened a meeting where we reviewed the data. Both sets of researchers’ notes, in addition to the main assessor’s notes were circulated to the entire team and a consensual interpretation was agreed upon. Expert Opinion The second method of validating the model was through interviewing subject matter experts. A panel of experts was enlisted, with each expert being separately interviewed and asked specific questions about practices contained within the model. The interview data was again analysed using qualitative analysis techniques. A number of issues arise which need careful consideration if taking such an approach, such as choosing the appropriate people, and the size of the expert panel. The choice of feedback mechanism is also an important consideration. For this research I chose to perform one-to-one interviews in order to benefit from the increased level of detail that can be obtained with such an approach. In addition, this approach overcomes the issues that may be encountered with a questionnaire approach. The challenge with such written surveys becomes:
127
“… to ensure that the questions are designed in a way that yields useful and valid data. It can be hard to phrase the questions such that all participants understand them in the same way, especially if the target population is diverse” (Easterbrook et al., 2002). This can be difficult to get right and may lead to the unintentional biasing of the answers (Oppenheim, 2000). Since it is important that the people being interviewed are capable of giving an informed opinion on the subject, careful consideration was given to finding those people who can be considered experts in the field. The research for this thesis therefore required people with a number of years experience working within the Medical Device industry and who were involved in developing software which was subject to the MD regulations. Alternatively they needed to be established academics/researchers with research focus in this area. I interviewed 11 experts from 11 different companies and used the data to feedback into an update of the models. Table 3-7 presents an overview of the interviewees with any identifying properties such as name and company removed to ensure anonymity.
128
Table 3-14 Expert Panel Members ID
Role
Country
Experience in MD (years)
1
Director or regulatory and quality
Ireland
20
2
Associate Professor in software engineering
U.S.A
7
3
Senior Business Analyst/IT Project Manager
Ireland
8
4
Verification & Validation Manager
England
9
5
Global software design control manager
U.S.A
10
6
Senior software quality engineer
Scotland
6
7
Senior software developer
Ireland
4
8
Senior software development contractor
Ireland
7
9
Associate director of software development
U.S.A
1*
10
Software engineering manager
U.S.A
5
11
Research & Development Engineer
Ireland
7
* Holds a doctoral degree and over 15 years software development experience
All but one of these were industry practitioners with many years of experience in software development, the Medical Device domain and were knowledgeable about the regulatory landscape. One was an associate professor in software engineering and actively engaged in a MD industry project implementation. The feedback was gathered through face-to-face or phone interviews using a semi-structured format. Each participant was sent a PDF document describing
129
the proposed overall SD process together with a spreadsheet containing the model practices. Within the spreadsheet each specific practice had a validation statement around which the interview discussions were structured. Each interviewee was asked about a subset of the specific model practices. With 97 specific practices within the final model, it would have been infeasible to get each expert to discuss every practice, especially when seeking to identify minicases in the form of examples or stories. Therefore the goal of the validation was to validate 50 of the practices by 3 experts for each, thereby covering a majority of the model practices (similar to the approach taken by (McLoughlin, 2010)). To accomplish this, each practice would need to be discussed in depth with 3 experts, resulting in 150 expert opinions. Consequently it would require 15 domain experts validating 10 practices each to achieve this level of validation. Due to the open nature of the interviews, the validation goal was exceeded after the 11th interview. As an example of these discussions, the first high-level practice within the model describes the need to define and establish a software unit verification strategy. The first corresponding specific practice proposed by the model included the following validation statement: “A verification strategy does not need to include the R&D phase. However, it is prudent to generate a high level verification report”.
The second specific practice for this had the following validation statement:
130
“A verification strategy should specify different levels of verification depending on the level of concern of the software”. By inviting the respondents to think about the application of specific practices in their real-life experiences, helped them give a more believable and relevant opinion on the practices and the model itself. Therefore, specific focus was put on soliciting such evidentiary mini-cases as a means to support their opinion. This focused sampling method (Hakim, 1987) had the benefit of gaining a richer response and ensuring each respondent had the same understanding of the questions. In addition to these verbal responses, each participant was invited to give a summary opinion of the validation statements they discussed. This was in the form of choosing whether they agreed with the statement (Yes), did not agree with the statement (No), or partially agreed with the statement (Somewhat). This provided a very easy quantitative mechanism for reviewing the data. For example Figure 3-7 shows a summary of the 7 responses to the validation statement on row 26 of the model: “Allow customer to 'play with' prototype so they can refine requirements”. 6 6 5 4 3 2 1 0
1 0 Yes
ROW26 Somewhat
No
Figure 3-23 Example of Quantitative Summary of Expert Responses
131
Although the validation strategy was to solicit feedback from 3 experts on each of the model practices, some interviews proceeded faster and/or lasted longer than planned and so it was possible to review more than the allotted 10 rows with those experts. This had the positive effect of generating more than the targeted 3 expert responses for some of the model practices as can be seen from the 7 responses in Figure 3-7. 3.4.7.2
Phase 3 Outputs
The output from Phase 3 was the final SaLeS-4-MD model of lean software development for medical devices. Version 2 of the model from Phase 2 had undergone a detailed validation process with an expert panel and had been updated to reflect the findings. This output concluded the research process.
3.5
Data Analysis Considerations
Analysing the research data is an important activity since it is what informs the theory building aspect of the research. Using qualitative data analysis techniques as described by (Strauss and Corbin, 1990) and (Miles and Huberman, 1994) the various data sources were used to build an holistic view of the data and from which the models of the software development process were constructed. As highlighted by (Miles and Huberman, 1994), there are a number of issues which need to be kept in mind when carrying out qualitative data analysis: -
Data overload (limited human ability to process large amounts of data)
-
Researcher bias
-
Time demands of coding data
132
-
The adequacy of sampling with small numbers of case
-
The generalisability of the findings
The rest of this section describes how each of these was addressed during the research process, and this data analysis was used in Phases 2 & 3 of the research.
3.5.1 Data Overload By its very nature, qualitative research is typically done with words rather than numbers. This makes the data much more difficult to manipulate and interpret since words can have different meanings in different contexts (Miles and Huberman, 1994). Also, and especially when interview data is being analysed, they tell us that the quantity of data can be extensive: “A week at a field site often can result in something like 200-300 pages of typed-up field notes and ancillary materials”.
According to (Miles and Huberman, 1994), there are two techniques best suited for defending against data overload: conceptual frameworks and research questions, both of which I have employed here. Additionally, I used data reduction techniques such as open and axial coding and used physical and electronic folders to make the data more manageable. I also made use of software packages to assist in the storage, manipulation and presentation of the data.
133
3.5.2 Researcher Bias Most qualitative researchers are one person research machines: “... defining the problem, doing the sampling, designing the instruments, collecting the information, reducing the information, analysing it, interpreting it, writing it up. A vertical monopoly” (Miles and Huberman, 1994). It is important, therefore, that one reflects periodically on their research approach to ensure that no undue influence/bias is being exerted. (Miles and Huberman, 1994) tell us that there are three typical analytical biases:
The holistic fallacy (interpreting events as more patterned and congruent than they really are, lopping off the many loose ends of which social life is made) Elite bias (overweighting data from articulate, well-informed, usually high-status informants and under-representing data from less articulate, lower status ones) Going native (losing your perspective or your “bracketing” ability, being co-opted into the perceptions and explanations of local informants)
I needed to be particularly cognisant of researcher bias due to my own lengthy career in industry. Another possible issue with having a rich background in the subject matter is that it leads to some sort of ‘Researcher effect’. This may manifest itself in people assuming the researcher knows more than they do or perhaps being unwilling to fully engage with the researcher. Within the ... case study, ways in which this was countered were to spread out my site visits, include a wide variety of participants and try to keep words and thoughts conceptual. During the validation phase, the interviewees were provided with documents describing the SaLeS-4-MD model and clearly showing the source of the various practices. It therefore reduced the need for
134
them to rely only on what I was saying. In addition, many of the interviewees were sourced through medical device software discussion groups and resulted in a panel with a wide variety of backgrounds both professionally and geographically. This helped maintain a healthy cross section of opinion from the wider community. To counteract coding bias, objective verification by a third party of the coding technique and results was sought, namely the researcher’s supervisors.
3.5.3 Time demands of coding data One hour of a recording can take up to 8 hours to transcribe (Hove and Anda, 2005), and it can take an inexperienced researcher 10-15 minutes to code one page of a transcript (Miles and Huberman, 1994). For both the case study and validation interviews, this was a tiring exercise but was steadily completed over an extended period of time, taking breaks to perform other tasks so as not to develop a rushed attitude simply to get to the end. It was also made into a mentally active exercise by recalling the interview setting (when coding the interviews) or reflecting on past work experience when examining/coding artefacts.
3.5.4 The adequacy of sampling with small numbers of cases For this research, the ... case study offered a unique, representative and revelatory view of the phenomenon, and from that perspective, sampling size was not an issue.
135
In terms of getting access to as large as possible number of people to interview within the company, I specifically requested a cross section of employees including multiple developers. The participants were representative, highly experienced and the interviews were in depth, typically lasting more than an hour. This combined with the other sources of data helped reinforce the findings from the interviews. Finally the expert validation interviews consisted of 11 experts from 11 different organisations across multiple countries and geographical regions. This broad feedback provided an additional level of confidence to the findings.
3.5.5 The Generalisability of the findings A qualitative research perspective does not seek to demonstrate generalisability, for the research is: “… specific to the people, context, and the time the study was conducted” (Connelly and Yoder, 2000).
The findings may however be transferable to another context, but this then depends on the similarity between contexts. There is a distinction to be made between analytical generalisation and statistical generalisation. The former is relevant within a qualitative context, and is where the empirical material is raised to a general level, made possible by strategic choice of informants, not by statistical sampling (Stenbacka, 2001). In addition, case study research is not always meant to be generalised to a wider field (Siggelkow, 2007). While it is
136
accepted that multiple cases aid in developing a more robust theory (Eisenhardt and Graebner, 2007), this research concentrated on investigating a specific phenomenon, using one case study to generate knowledge about it, and solicited expert opinion to evaluate the transferability of that knowledge to other similar contexts. In addition, the literature from a wide range of sources, and empirical research within the Medi-SPICE team all contributed to the model development, making it more likely to be relevant to MD companies in a more general sense.
3.6
Research Validity
A critical aspect of research in general, is the concern with validity, and by implication, reliability. In other words how can the researcher convince the reader that the research findings are good and valuable? (Yin, 2009) provides us with four tests commonly used for testing the quality of case study research: Construct Validity, Internal Validity, External Validity, and Reliability. However, from an etymological perspective, some controversy exists about what exactly validity and reliability mean in a qualitative research context. It can be said that these terms have their origins in the positivist paradigm where “construct validity” refers to validating the construct, which in that sense is the a priori theory or hypothesis. Using the definition for validity as: “[that] the intended object of measurement is actually measured” (Stenbacka, 2001) says: “the validity issue has already proven itself useless ... simply because the purpose in qualitative research never is to measure anything”.
137
Some researchers argue, therefore, that these terms are inadequate, however agree there is a need for: “some sort of qualifying check or measure for their research” (Winter, 2000). Discussing the notion of validity in quantitative and qualitative research he says: “... it immediately becomes apparent that an aggregated definition, whilst representing the majority of opinion, can prove mutually exclusive to the definitions of certain researchers and methodologies. I refer of course to qualitative researchers, some of whom vehemently deny that replicability is either useful or possible in situations concerning highly complex and transient circumstances: namely those that involve the lives, thoughts and behaviour of actors”.
Therefore, the terms validity and reliability need to be redefined in the context of the research paradigm in which they are being applied (Healy and Perry, 2000), however, “Because of the multiplicity of paradigms and methodologies that are categorized in the qualitative field, it is overly simplistic, indeed inaccurate, to describe global qualitative criteria for validity” (Given, 2008).
Nonetheless, a researcher must make some attempt to present their research in such a way as to show the trustworthiness and credibility of the findings in order to be convincing. Section 3.7 deals with the research quality in more detail, and in addition I use the five, somewhat overlapping, attributes as
138
described by (Miles and Huberman, 1994) by which the study quality can be judged (Table 3-8). Table 3-15 Research Quality Attributes (Miles & Huberman, 1994) Quality Attribute Objectivity/Confirmability
Reliability/Dependability/Auditability Internal Validity/Credibility/Authenticity External Validity/Transferability/Fittingness
Utilisation/Application/Action Orientation
Guiding Enquiry - How free is the study from the influence and biases of the researcher? - Do I have all the data presented in detail and is the study data readily available? - Have things been done with reasonable care? - Do the findings of the study make sense? - Are the conclusions transferable to other contexts? - Are the original sample of peoples and variables described enough to permit comparisons? - Are the findings intellectually and physically accessible to potential users? - Does it work in practice?
Each of these 5 quality attributes is reviewed and answered in Chapter 8.
3.7
Research Quality
For this research, the approach has been to follow the advice of (Morse et al., 2002) and to demonstrate throughout the whole process, that each step has been prepared and executed with due diligence. This fits well with the design science approach taken to the research: “Since the phases of a study are performed in an iterative manner, so too are the various forms of [verification and validation]” (Robinson, 1997b).
139
The rest of this section aims to instil in the reader that sense of trustworthiness by highlighting the many considerations given to the careful and methodical manner in which each piece of the research was undertaken.
3.7.1 Quality Attributes and Principles Using the qualitative attributes, as identified by (Stenbacka, 2001) for inductive enquiry, this section describes the approaches adopted in the research to address them. Reflection: whereby the researcher reflects on and illuminates the preunderstanding they have and also the process of gaining access to the phenomenon under study: “Only a reflective researcher can make the gained level of understanding accessible for the relevant scientific body" (Stenbacka, 2001). Before conducting the mapping study, a protocol document was developed and evaluated by the researcher (see Appendix B). As suggested by (Kitchenham and Charters, 2007), the protocol was also evaluated and critiqued by three researchers including the researcher’s two supervisors.
Similarly, before
conducting the on-site case study work, a case study protocol was created (see Appendix E), again being reviewed by my two supervisors. This process forces the researcher to clearly articulate what it is they intend to investigate and how they will conduct the investigation. This type of evaluation and reflective preparation reduces the possibility of researcher bias during the actual research work.
140
First-hand pre-understanding: No researcher comes to a study without some level of understanding of the problem area. My first-hand understanding comes from direct experience of regulated software development. By utilising what (Maxwell, 1996) calls “memo writing”, a careful reflection of this experience was carried out. Second-hand pre-understanding: This is typically knowledge gained from a literature review and would be considered inferior to first-hand preunderstanding: “... first-hand pre-understanding is much more valuable to a study and ... many researchers unfortunately are victims of second-hand pre-understanding” (Stenbacka, 2001). Prior to starting the case studies research, I carried out a mapping study to gain an understanding of the state-of-the-art in this area. The findings from that review were published (Cawley et al., 2010). Access barriers: The intent of qualitative research is to get as close as possible to the phenomenon of interest. To get there however, many obstacles will need to be overcome. (Stenbacka, 2001) tells us it is important to be aware of these and to strive for “holistic access” within the study. In this research difficulties were anticipated in gaining access to the necessary companies and stakeholders in the relevant development processes. I was able to draw on my own experience to help alleviate some of these barriers by, for example, expressing an understanding of the difficulties developers are faced with and recounting similar situations I had personally experienced.
141
3.7.2 Quality Building Techniques Within qualitative research there is a need to clearly demonstrate to the reader that during the research process, a careful and considered approach was taken when gathering and analysing data. As (Miles and Huberman, 1994) tell us, such reports: “... are most often heavy on the “what” (the findings, the description) and rather thin on the “how” (how you got to the what)”. In order to build confidence in the research findings, the following techniques as suggested by (Miles and Huberman, 1994) were employed during the study: 3.7.2.1
Checking for Representativeness
(Miles and Huberman, 1994) identify three pitfalls when it comes to representativeness in qualitative studies: - Sampling non-representative informants - Generalising from non-representative events - Drawing inferences from non-representative processes. To help guard against these, the following approaches were used during the interviewing stages: Interviewees. As detailed in earlier sections, special care was given to the selection of interviewees during the case study, the focused survey, and expert validation stages. Stay on point. During interviews, many specific stories were recounted and anecdotes told, some of which were not pertinent to the research. Although interesting in their own right, I was conscious not to follow them too far but remain focused on the area of interest.
142
Back to source. Although ... informants worked quite closely together and were very familiar with each other’s role and responsibilities, they should only be used as experts in their own roles. The tendency during interviews can be for the interviewee to elaborate on areas that they observe from outside. In such cases, where the point was deemed worth following, a memo was taken and followed up with the relevant informant. Only then did I make a decision on how it should be recorded in the study. 3.7.2.2
Triangulation
According to (Given, 2008), triangulation is where a variety or combination of research methods is used to understand a phenomenon, and is important because it increases the precision of empirical research (Runeson and Host, 2009). According to (Stake, 1995), there are four types of triangulation: 4. Data (source) Triangulation 5. Observer Triangulation 6. Methodological Triangulation 7. Theory Triangulation In this research two of the above approaches were used to make the findings more robust: Data Source and Methodological triangulation. Neither Observer triangulation nor Theory triangulation were employed since there was only one researcher, and the research was not performing theory testing. 3.7.2.3
Checking the meaning of outliers (examine any strange data items)
As with any analysis, it is normal to observe data items which seem incongruent with the rest of the observations. There are two possible reasons
143
for this, either the data item is an anomaly and can be ignored, or there is an underlying cause for it which is pertinent to the entire data distribution and which therefore must be accounted for in any resulting theory. The important thing is to examine the data and determine which case it is, and in so doing protect against self-selecting bias (Miles and Huberman, 1994). During this research such outliers came in various forms, such as, stories recorded during interviews of untypical phenomenon, attitudes which were unusually disinterested, defensive or aggressive, observations of events which seemed incongruous with the rest of the environment, project metrics which were out of line. Each such occurrence was examined as closely as possible, the reason behind the phenomenon investigated and its significance in the overall study determined. 3.7.2.4
Following up Surprises
In a similar vein to outliers, surprises are those events or findings which are incongruent with other observations but are possibly more illuminating and should therefore be reflected and followed up on. For example, during the research I discovered a different SD process being followed in a different corporate division within .... Although a different company (which had recently been purchased) they were fundamentally doing the same activities and subject to the same regulatory controls. The difference was that the other division was delivering the projects with far less documentation. This, within the context on the research was a significant surprise since it indicated that perhaps the case
144
study plant was doing too much documentation and a more lean approach (less documentation) was feasible. 3.7.2.5
Getting feedback from Informants
Using the original informants to sense check research findings is, according to (Miles and Huberman, 1994): “... a venerated, but not always executed practice in qualitative research”. These stakeholders will have a much better understanding of the realities that the researcher is investigating, and for that reason are well placed to judge any findings. For this research the following two forms of informant feedback were used. Firstly, feedback does not have to wait until the end of the research but can be provided during data collection too. As the research progressed through the ... case study interviews and ideas/connections began to suggest themselves, subsequent interviewees were used as ‘sounding boards’ to explore their reactions to the emerging theory. This would then either support the finding or suggest further investigation was required. The second type of Informant Feedback was the final industry report. This happened both informally and formally. An informal meeting was held with the research contact, who was eager to get some sort of high level feedback. Despite not having the full data analysis complete there were some immediate findings which I was able to provide. The response was that this type of feedback was
145
exactly what the company needed to hear and that hopefully I could provide the full report in time for their upcoming process improvement program. Once I had completed the data analysis, an official report was generated, reviewed by the researcher’s supervisors and submitted back to the company. The report was circulated and “well received” within the company. In addition the company “... used it as a good input to help us decide our next steps to improve our equipment introduction process”. This very positive feedback helped build confidence in what the research had found and proposed.
3.8
Conclusion This chapter gives the reader a detailed understanding of how I approached
the choice of research methodology. Adopting a design science methodology, it describes the decision to use qualitative techniques to constructively and iteratively build up a lean software development process model and conceptual framework (Figure 3-2). Section 3.3 discusses the various alternative research approaches which were available, the choice of methods which best suited my research questions, and with the aim of constructing knowledge generating artefacts, the reasons for choosing a mixture of methods to iteratively construct a conceptual framework and process model for SD. Section 3.4 describes how each of the steps in the iterative design was performed and how they influenced the development of the two models. Section
146
3.5 demonstrates awareness of the issues associated with qualitative data analysis and the ways in which those issues were addressed. Sections 3.6 and 3.7 present in detail the importance attributed to the quality and validity of the research, highlighting the many ways in which I have planned for and addressed these within the research process.
147
148
Chapter 4: Development of the Models 4.1
Introduction
Phase 1 of the research generated an initial Conceptual Framework of the influences on the development of the SDLC within a MD company. In addition, the framework incorporated a lean thinking component which affects how the MD regulations are interpreted and consequently how the company defines its SD processes for compliance. The research found that some lean or agile practices are suitable within a MD context. It also revealed that some practices are not suitable or are only partly suitable. A collection of these practices were grouped into lean categories to form Version 1 of the SaLeS-4-MD process model. This consisted of 162 entries in the model. The SaLeS-4-MD process model provides a more user friendly view of the data since: “… what is the use of all our understanding if we cannot turn it into something practical and useful?” (Handy, 1985). What this model achieved was to identify appropriate practices, highlight any issues from a MD perspective and where possible suggest the ways in which those issues might be addressed in order for them to be applied within the development process. The rest of the chapter describes the evolution of both the Conceptual Framework and SaLes-4-MD process model from the initial collection of practices generated from the literature review, through to the final versions which reflect the findings from the empirical research.
4.2
PHASE 2 – “Iterative Model Development”
Phase 2 produced an evolution of the two models from Phase 1 through a combination of a number of real-life inputs, which included interviews, company documentation such as development processes, operating procedures and other project artefacts, industry standards, regulatory documentation, and expert opinion. Combined with an ongoing literature review, these made important contributions to the models as described in this section.
149
The Conceptual Framework developed in Phase 1, and used to focus the empirical data gathering, underwent some structural changes as a result of Phase 2. The key modifications which occurred include:
redefining the central focus as being the implementation of specific lean software development practices as opposed to the design of the entire software development process
restructuring the framework based on an existing published framework
renaming two of the influencing categories, and
reflecting the interconnectedness of the various components.
In addition, the SaLeS-4-MD model was enhanced by means of an overview process, and a guide to the types of activities which needed to be practiced within each iteration of the life-cycle. This was augmented with the addition of a significant number of new software practices and the identification of supporting evidence for many existing practices. Each of the empirical sources identified within the design of Phase 2 (Figure 4-1) is now examined and their contribution to the models’ development described. To highlight the contributions to the Conceptual Framework evolution, I have enclosed important points within square brackets [ ] and used a bold font type. For example a relationship identified between two framework components is indicated as follows: [Lean/Software Development], indicating a relationship from the Lean component to the Software Development component.
150
Figure 4-24 Phase 2 Empirical Sources
4.2.1 Regulations and Standards Since it is the requirements of the specific medical device regulations and respective standards which make the associated software development process unique, these two sources are dealt with here together given their close association. As mentioned in Chapter 2, IEC 62304 “Medical Device Software – Software Life Cycle Processes” (ANSI/AAMI/IEC, 2006) is becoming the accepted standard, and referred to within national and federal regulatory documents. A careful examination of the document, however, revealed two important points as described in its background information. Firstly, it is here that the concept of a software safety classification is first introduced. This demonstrates the link between the Regulations and Organisation, and Regulations and Software Management. It also however suggests the importance of the link between Organisation and Software Management, since it mandates the manufacturer to assign a software safety class to each software system. Secondly, the standard removed two processes from the ANSI/AAMI SW68 standard on which it is based, namely Documentation and Verification. While some of the requirements that were contained within these processes were moved to other processes, it is important to ensure that extra attention is given to these when designing a SD process.
151
Further, Section 1.2 highlights the fact that “This standard does not cover validation and final release of the MEDICAL DEVICE, even when the MEDICAL DEVICE consists entirely of software”. This is critically important and highlighted the need for a validation element within the SaLeS-4-MD process model. Indeed this standard treats the software somewhat independently of the encompassing system, something the Medi-SPICE team also had to cater for in the reference model development. The standard calls for the manufacturer to demonstrate the ability to provide medical devices that consistently meet customer and regulatory requirements. In section 4.1 Note 1 it states that: “Demonstration of this ability can be by the use of a quality management system that complies with: - ISO 13485 - A national quality management system standard - A quality management system required by national regulation”. Similarly from a risk management perspective, section 4.2 states:
“The
MANUFACTURER shall apply a RISK MANAGEMENT PROCESS complying with ISO 14971”. What this highlights is the need to be clear on what is contained within the relevant documents, and what other sources need to be reviewed when seeking to understand the meaning of compliance. For the SaLeS-4-MD model this meant that the specific categories of tasks which needed to be performed within each iteration, for example, validation and risk analysis, needed to draw from all these relevant documents and not solely rely on any one source. An additional example from this analysis is Annex B of the IEC 62304 document. Here it states that: “This standard does not require a particular SOFTWARE DEVELOPMENT LIFE CYCLE MODEL”. This is important for the research since it clearly states that an evolutionary methodology, as advocated within this study, is an acceptable one to use when developing MD software. From a conceptual point of view this demonstrates a link between an independent Lean component and Software Development component (adopting an
iterative/evolutionary
development
152
approach)
[Lean/Software
Development], and from the Software Development to the Regulations component [Software Development/Regulations] to indicate that it must satisfy the regulatory requirements., An analysis of the international standard ISO 14971 “Medical Devices – Application of risk management to medical devices” found that it clearly states: “Top management shall provide evidence of its commitment to the risk management process by: - Ensuring the provision of adequate resources, and - Ensuring the assignment of qualified personnel for risk management”. This supports the Conceptual Framework and the link between Regulations and Organisation [Regulations/Organisation], but also highlights the importance of
the
link
between
Organisation
and
Software
Management
[Organisation/Software Management], since the organisation will be responsible for ensuring the necessary supports are provided for the development of the software. Section 3.4 of the standard states that “Risk management activities shall be planned”. Consequently this was something that was included in the planning activities of the evolving SaLeS-4-MD model. Importantly, Section 4.1 Note 1 tells us that information from the risk analysis of similar devices can be used as starting points for new devices. This is a good example of a lean practice (reuse) which suits the philosophy of the SaLeS-4-MD model and was subsequently included. As a final example of this analysis, section D.4 “Risk evaluation and risk acceptability” of ISO 14971 makes the suggestion that, when conducting risk evaluation, to use a wide cross section of stakeholders. This is reflected within the SaLeS-4-MD model with the recommendation to “Use cross functional teams” when performing risk analysis and evaluation. Due to the regulatory requirement for traceability, a triangulation with the existing literature (Shoemaker and Van Schooenderwoert, 2010) highlighted the need to record any risk mitigations decided upon and the identities of the people making those decisions. Within the conceptual framework this suggests a link between the
153
Regulations
and
People
components
in
terms
of
their
behaviour
[Regulations/People].
4.2.2 Case Study Observations The case study was performed within . is a multinational Medical Devices company ... The plant manufactures medical devices and incorporates a research and development function. The initial case study data analysis generated a list of influences on the software development life-cycle in ... as depicted in Figure 4-2. This indicates the complexity of the situational factors within which the ... SDLC was designed and must function.
Process Improvement
People
Project Management
Organisation
Software Development
Lean Focus
Business Focus
SDLC
Standards
Figure 4-25 Influences on the ... SDLC as derived from the Case Study Analysis.
This also suggested that the focus of the Conceptual Framework from Phase 1, on the entire life-cycle process, might benefit from focusing at a lower level and reflect the myriad of different specific practices that are employed to satisfy the various requirements. Version 1 of the SaLeS-4-MD model was analysed in terms of the case study data and the results are summarised under the following headings:
154
1. The Software Development Process 2. Software Development Practices 3. A Lean Ethos 4. Approach and attitude to Risk 5. Regulatory Focus 4.2.2.1
The Software Development Process
The case study within ... provided a wealth of artefacts such as the overview of the software development life-cycle (Figure 4-3). Figure 4-26 The ... Software Development Life-Cycle
This shows the high level steps involved in the development life-cycle being: -
Planning
-
Specification
-
Design
-
Construction
-
System Installation & Testing
-
Maintenance & Retirement
Figure 4-3 depicts a phased approach to the software development and verification process typical of a waterfall-model methodology. However, a more interesting picture is contained in the detailed view presented in Figure 4-4, which clearly shows that the company does not advocate a purely waterfall-like approach but instead acknowledges that the specific phases of development: “... can be done in parallel and may overlap” (Internal SDLC documentation). Figure 4-27 The Detailed ... SDLC
This is significant, and taken with the company’s recommendation that developers should “... get feedback from the owners and stakeholders early and often” suggests that an iterative approach to developing the software could be appropriate. This is supported by the training handbook which states:
155
“The E&AS Life Cycle supports multiple development methodologies: Top-down designs Waterfall model Iterative or Rapid Prototyping”. What is particularly interesting however is that the case study revealed a waterfall-model of development was being followed despite this explicit process support for Iterative development. As one developer put it “...we don’t get the time to do iterations”. What this demonstrates is the way the regulations influence the organisation [Regulations/Organisation], for example by their choice of acceptable SD methodologies, and the way the regulations influence the behaviour of the employees [Regulations/People], for example, in the methodology they choose to adopt. During one of the interviews it was suggested that they could improve their manufacturing process if they spent more time researching and trying newer technologies and techniques: “Time to market is the big deal here ... It’s very rare that we get to develop prototype equipment, [and] test it” (a principal process development engineer). This is also supported by what one of their principal engineers said about how they would like the process to occur: “[The customer should] come and tell us what [they] want, write it down and we’ll meet every 2 weeks and we’ll give [them] something really quick and [they] can come back and tell us [what they think]”. This form of development could be described as evolutionary prototyping, whereby the prototype does not get discarded but is (potentially) released to the customer and gradually developed into the final product. This would seem to be very similar to an agile approach, however, evolutionary models are not agile by definition (Dagnino, 2002). Given the explicit reference within the SDLC documentation to support for “Iterative or Rapid Prototyping”, and because of the implicit involvement of the software development group in that process, it suggests that a general iterative approach to the software development would be more effective for them as opposed to the waterfall-type approach of their current process.
156
Consequently, the practice contained in Version 1 of the SaLeS-4-MD model, of using an iterative approach, is considered to be verified by the case study (Table 4-1). Table 4-16 The Iterative Practice Validated through the Case Study Lean Principle
Deliver Fast
Lean Practice
Description
Short iterations
Use short iterations to get feedback quickly
Potential Issue in MD Domain Typical agile iteration lengths may be too short
Mitigation Use iteration lengths appropriat e to your situation
Source
Rottier and Rodrigues 2008
Rassmu ssen et al 2009
Case study
An iterative process is at the core of a lean software process, and combined with the evidence from the detailed model here and standards review of the previous section, led to the high level process model having an iterative component at its centre as depicted in Figure 4-5.
Figure 4-28 An Iterative Nature added to the High Level Process Model
Research and Development Another important contribution to the SaLeS-4-MD process model from the case study was around the presence or absence of a Research and Development phase (R&D) and the associated activities. These activities are typically performed under the banner of new product development (NPD). However, they are quite different in nature to those activities which actually bring the defined new product to commercialisation. The principal characteristic of an
157
R&D process is uncertainty, with the “... initial activities of ideation, initial assessment, concept development, and business case analysis ... commonly referred to as the fuzzy front-end (FFE) of new product development” (Sperry and Jetter, 2009). Although the term ‘fuzzy frontend’ has been attributed to (Smith and Reinertsen, 1998), concepts of these “up-front activities” in the NPD process have been discussed for a much longer period (Reid and de Brentani, 2004). Although the FFE is typically part of NPD, (Sperry and Jetter, 2009) suggest that we need to view and perform the associated tasks differently. Figure 4-6 shows am NPD framework developed by the authors which shows the need for an iterative, dynamic approach to the FFE activities, while the actual development of the new product should happen in a much more linear, deterministic fashion.
Figure 4-29 An NPD Framework (Sperry and Jetter, 2009)
According to (Smith and Reinertsen, 1998) the FFE is also “... a place where one can find several low cost opportunities to achieve large improvements in time-tomarket”, however many firms acknowledge that this is also a place where their product innovation process has a serious weakness (Khurana and Rosenthal, 1997). The FFE phase of NPD is “intrinsically non-routine, dynamic, and uncertain” (Kim and Wilemon, 2002), and typified by an ill-defined process. Within the context of medical devices this is no different and so it is important to support a flexible and dynamic R&D process by ensuring that external constraints are minimised. One such potential source of constraints is the regulations. The regulations, however, are not intended to regulate how the R&D process operates, and do not expect that the same level of traceability is
158
maintained “Manufacturers are not expected to maintain records of all changes proposed during the very early stages of the design process” (FDA, 1996). In terms of the FFE this allows for a much faster, iterative process as suggested within Figure 4-6. The regulations are clearly focused on the phases where the final product is being designed with a view to bringing that design to clinical trial and commercialisation. As a result the auditors typically do not delve very deeply into the R&D activities and therefore the manufacturer is not required to follow their normal processes which are subject to a higher level of scrutiny. This highlights the importance of matching SD practices with the requirements of the regulations and suggests an important link between those two components within the conceptual framework [Software Development/Regulations]. What this meant for the case study company, for example, was they could bypass a lot of the cumbersome documentation and traceability activities and deliver prototype functionality and/or equipment months earlier than if they followed the normal process. As a principal engineer said of developing software for a prototype manufacturing line: “There’s no requirement for that line to be fully validated and to go through this whole system”, and if they did follow the normal process it would “... add 4 months onto the schedule of delivering the equipment to the customer”. Consequently the process model was enhanced to allow for the presence of an R&D phase which can be treated differently in terms of the specific practices (Figure 4-7). Here again we see a prototype development approach being utilised within .... Although such projects are a minority within the company (NPD is an infrequent process), it again suggests the possibility of incorporating a more iterative development methodology.
159
Figure 4-30 SaLeS-4-MD Process Model Overview with Possible R&D Phase
A strong factor coming through from the data analysis was the importance of the relationships between all the components of the Conceptual Framework. Not only did they affect the SDLC process, but the way in which they were interrelated with each other was showing to be an important consideration and something which warranted inclusion within a descriptive framework. In addition, the reality of operating in a lean manner, independent of being regulated, was revealed. Consequently the lean lens element of the Conceptual Framework was in need of being represented as a component in its own right. 4.2.2.2
Software Development Practices
Some of the practices within Version 1 of the SaLeS-4-MD model were also found to be employed within .... For example, the practice of using similar documentation structures for different documents, making use of coding standards and issuing development guidelines to the developers, were all practices that existed within the case study company, and therefore brought a level of confidence to those practices within the model. In some cases a modification to the model was necessary to reflect new information. For example, the practice of using an object oriented approach to coding was something already identified as supporting the lean principle to
160
‘Build Quality In’ by hiding complexity and limiting the effects of changes. The case study illuminated this practice as also being an enabler for more easily transferring code/applications from one manufacturing site to another, and also for supporting projects which were comprised of distributed team members. The practice of using a good source code management tool was supported by the case study as being a practice which supported knowledge creation by ensuring all developers had the latest changes to the code base. In addition, it informed that this practice also aids in demonstrating process integrity through the restricted access to the code base. It therefore supports the lean principle to ‘Build Quality In’. Other development related practices which were gathered from the case study, and which were subsequently added to the model included prototype building (allowing the customer to preview the product and thereby generate early feedback) and developer role interchanging (assigning developers to varying roles so as to develop a broader range of skills). The model was updated to reflect these findings. These various practices pointed towards the central importance of the specific SD practices within the process. Much emphasis is put on finding practices which supported the need of the organisation to maintain regulatory compliance, and indicated the circular relationship from the Regulations to Organisation
[Regulations/Organisation],
Development
[Organisation,
Software
Organisation
to
Software
Development],
and
Software
Development back to Regulations [Software Development/Regulations]. However, the specifics of the projects and the ethos of the organisation influence and shape what practices would be adopted, showing again an important relationship between these components of the Conceptual Framework
[Project
Management/Software
[Organisation/Software Development].
161
Development],
4.2.2.3
A Lean Ethos
... has a well established lean ethos across the entire organisation which is immediately evident upon entering the ... plant. The case study also found some evidence to support its application within the IT function of .... One particular improvement initiative I investigated was aimed at making the equipment validation process more lean (this included the software development process). Figure 4-8 demonstrates how they used the lean practice of value-stream mapping in order to identify non value-adding activities within the process. Consequently, the practice of value–stream mapping was added to the model. Figure 4-31 A Value-Stream Map of the Validation Process
Additionally, in order to reduce waste by identifying bottlenecks in the project management process and therefore assist in maintaining work flow, they employed the use of Heijunka. Heijunka is a Japanese term meaning production levelling or production smoothing, and one of the supporting practices of the foundation in the house of lean framework. By means of a large display board (similar to the Kanban board described in Chapter 2), the entire portfolio of projects was made visible and an instant picture of work in process was available to everyone (Figure 4-9). By colour coding the cards to depict different types of deliverables it facilitated the rapid identification of problem projects. In addition it aided in identifying how projects were allocated and what resources might be getting overloaded and therefore becoming bottlenecks in the process.
162
Figure 4-32 A Visual Display Board in ... “Heijunka”
Within the context of this research, such an approach supports the lean principles of knowledge creation and maintaining flow, and demonstrates a link between the conceptual framework components of a Lean Lens and SD practices [Lean/Software Development]. In addition, to reflect this in the process model, two new practices were added to the model as shown in Table 4-2. No issues were identified with either practice from a regulatory perspective.
Table 4-17 New “Heijunka” Practice added to the Model Potential Lean Issue in MD Lean Principle Practice Description Domain Don't overload team Perform load levelling Continuous Flow members (Heijunka)
Create Knowledge
Use a visual display
Mitigation
Open display of work in process informs everyone of status & issues
Quality Function Deployment (QFD) is very closely associated with lean manufacturing and both have a common focus, namely generating customer value. QFD has 3 main goals:
163
-
Prioritise spoken and unspoken customer wants and needs
-
Translate these needs into technical characteristics and specifications
-
Build and deliver a quality product or service by focusing everybody toward customer satisfaction.
In addition “a very important side benefit ... when appropriately applied, QFD has demonstrated the reduction of development time by one-half to one-third” (CIRI, 2007). As is demonstrated by Figure 4-10, ... advocates the use of QFD together with an FMEA (Failure Mode and Effects Analysis) approach to risk analysis. Since an FMEA approach is something which had already been identified within Version 1 of the model in support of the lean principle to build quality in, the case study was added as an additional source for that practice within the model. Figure 4-33 Extract from ... Training Manual
This data analysis again supported the need to have the lean lens of the Conceptual Framework replaced as a lean component independent of the regulatory component. The central focus, now being replaced by the situated practices, is then shaped by a lean mindset [Lean/Software Development] influenced and shaped by the context of the company [Organisation/Software Development] and shaped by the type of work being carried out [Project Characteristics/Software Development]. Lean initiatives can be instigated to tackle specific issues. Supported by specific lean activities such as Hansie (reflection), Kaizen (continuous improvement), Nemawashi (consensual decision making) and Kaikaku (revolutionary change) (Liker, 2003), (Womack and Jones, 1996), these activities are often treated as discrete projects, complete with assigned resources, start and end dates. Figure 4-11 shows the overview of a specific project within ..., generated from a lean thinking perspective in conjunction with an organisational need for cost reduction.
164
Figure 4-34 Sample Lean Project Overview within ...
The project was created to examine the development lifecycle, identify where the largest costs were being expended and look for ways to make it more efficient. This demonstrates a link between a lean mindset and specific projects or
project
characteristics
[Lean/Organisation],
[Organisation/Project
Characteristics], [Lean/Project Characteristics]. This is not to say that it is always a positive influence. In fact, some evidence was discovered showing a negative aspect to this. Within ..., this Lean Thinking, which had become an ethos within the company, in some cases had almost dogmatically been required to be utilised in all process improvement projects. Ironically this resulted in extra activities in order to make the improvement into a lean initiative: “As opposed to just doing [a project] we have to say, oh this is a lean project and it could be something very obvious, and then by making it a lean project you actually end up with all this documentation to show it’s lean”. This reinforces the existence and importance of a relationship between lean thinking and the way people behave [Lean/People].
165
4.2.2.4
Approach and Attitude to Risk
Due to the safety-critical nature of Medical Devices, the industry and regulators put much emphasis on identifying and minimising the risk of harm to humans. As we have seen this is the basis of the device classification system in both the U.S. and the E.U. The case study revealed that ... has a well defined risk management process. As depicted in Figure 4-12, risk is continuously assessed throughout the life-cycle in on ongoing fashion and not a once off task carried out at the start or end of the project.
Figure 4-35 The ... Life-Cycle showing Risk Analysis throughout
Similar to what was described in Chapter 2 on medical device classification, ... employ a risk analysis matrix to determine the level of concern (LOC) for a system. The matrix has two dimensions, patient risk and compliance risk as shown in Table 4-3. Based on these two factors the user can determine the LOC which in turn acts as a guide in determining what subsequent process steps should be followed and to what level of detail. Table 4-18 the ... Risk Analysis Matrix Compliance Risk
Patient Risk
Low
Medium
High
Low
3
3
2
Medium
2
2
1
High
1
1
1
166
The matrix indicates that the lower the LOC, the higher is the risk to the patient or to compliance or both. The LOC can be 1=Major, 2=Moderate or 3=Minor. Figure 4-13 is an overview of the company’s risk analysis process which uses the LOC as the guiding factor. Importantly it indicates that the higher the level of concern (numerically) the fewer risk analysis steps that are required. For example a minor LOC does not require any risk analysis, whereas for a moderate or major LOC the training manual states that in addition to performing risk analysis through the FMEA technique, that: “[if] the mitigation for the failure is a key function or control of the automated system, then the function is added to the System FMEA”.
Figure 4-36 Risk Analysis Process Overview within ...
Following an examination of all the risk related case data the practices shown in Table 4-4 were added to the model.
167
Table 4-19 New Risk Analysis Practices added to the Model
Lean Principle
Lean Practice
Description Regulations expect a designated authority responsible for quality management Every step should be considering sources of risk and how to mitigate Employees have to be made aware of the regulatory requirements
Build Quality In
Identify the responsible party
Build Quality In
Make risk analysis inherent in the process
Create Knowledge
Develop a training schedule
Eliminate Waste
Decide on level of concern
Build Quality In
Use predefined templates
Build Quality In
Consider Performing a S-FMEA
Less rigour required for “low” LOC Eases capture of risk data and assists in traceability Perform a System FMEA only for high or moderate LOCs
Build Quality In
Use Cross Functional Teams
For both risk analysis and reviews
Potential Issue in MD Domain
Mitigation
Different focuses at different levels within the organisation
Ensure somebody has responsibility for keeping up to date with the regulations
Mitigations need to be fully traceable
Add or change relevant documentation
Regulator may seek evidence
Record who has attended training
Choosing the wrong LOC
Have a clear process for deciding
Mitigations need to be fully traceable
Record who attends
The case study revealed another interesting aspect in relation to risk within .... As table 4-4 shows, the company is not only concerned with patient risk but also with ensuring that the company is in compliance with the regulations. As a senior engineer put it: “You could reduce this [the development process], make it 100% more efficient but if there’s a 10% risk that you won’t be [compliant], people will never take that. You have to be compliant first and you can’t introduce any risks or leave any gaps”.
168
This attitude to compliance is due to the expensive nature of MD development and the large investment the company has to make before launching a product commercially. This fear of failing regulatory audits and not being able to market a product, has led the company to overcompensate by employing a more heavyweight SD methodology than necessary. As one interviewee said: “This [the process] has nearly crippled the business. We need to Lean it and take it all back out again” (a principal engineer), and more strongly by another interviewee who said: “You can do all this [the process] and deliver [bad quality] to your customer ... which is why it’s seen as a failure in the overall organisation. Project managers and R&D complain about it, engineering managers complain about it, nobody understands it, it’s just a noose, it’s just a pain really” (a principal process development engineer). This strongly supports the link from Regulations to the other components within the Conceptual Framework but highlights the specific need for risk management and the influence it plays on the way specific practices are implemented. The relationships here, relevant to the conceptual framework are: [Regulations/People],
[Regulations/Organisation]
and
[Regulations/Software Development]. A final type of risk within the ... context is the risk of missing market launch dates. This time-to-market factor of the industry appears to be an increasing concern within the company and something which came out strongly in the interviews. A senior software quality engineer said: “Time is generally the biggest priority. Time out weighs cost a lot of the time here”. Following one particular process improvement initiative to automate a production line, but which led to a delay in time-to-market, one interviewee reported: “... the tolerance has gone to zero now for equipment upgrade or equipment strategy causing a delay in product time to market” (a project leader).
169
This has been one of the reasons why the development process has remained heavyweight in nature. While this reinforces the link influence the organisation has on the SD process and practices [Organisation/Software Development], it highlights again the need to represent the relationship between components, in this case organisational management and the people engaging in the SD practices [Organisation/People]. These examples add support to the view of the inadequacy of a heavy, planbased software development methodology within the MD industry. The regulations don’t require it, and modern business demands aren’t satisfied by it. An argument therefore can be made for the adoption of a more flexible and iterative approach, again something which supports the iterative aspect of this study’s high level process model as shown in Figure 4-5. 4.2.2.5
Regulatory Focus
Within the MD regulations, ISO 13485:2003 states that management has the responsibility of: “Communicating to the organisation the importance of meeting customer as well as statutory regulatory requirements”. Two framework relationships are indicated by this, the direct responsibility the regulations
place
on
the
organisation’s
management
[Regulations/Organisation], and the subsequent communication/direction from management to the rest of the workforce [Organisation/People]. ...’s approach to this was to develop a mandatory training schedule for all employees on the product development process, and which was coordinated by the human resources department. An internal application was developed which kept track of which employees had attended the training (either classroom based or online), which employees had not, and importantly which employees were overdue training. Due to updates to corporate policies and procedures as a
170
result of regulatory changes it is important that this type of process is put in place so that everyone is informed and kept up to date. In addition, MD standard ISO 13485:2003 holds “Top Management” responsible for “evidence of its commitment to the development and implementation of the quality management system” and also for the implementation of an ongoing risk management process. ... in ... achieves this by having a quality assurance department which takes guidance from a corporate compliance group responsible for ensuring internal process compliance. An interesting revelation within the ... plant is that at the local level, there is an assumption that someone in “corporate” is ensuring that the processes are in compliance with the necessary latest standards. As one senior software quality engineer said when asked about conformance to ISO 62304: “We’ve got a regulatory group that assess standards against corporate policy...So I assume that activity would have happened up there”. This reinforces the relationship between the organisation’s management and the rest of the workforce [Organisation/People]. To reflect this in the process model, a new practice was added which specifically calls for someone to be clearly designated the responsible person for keeping up to date with the regulations. Medical device regulations require that a company provide evidence of the steps that it undertook in developing any relevant software in order to assess its compliance. (ANSI/AAMI/IEC, 2006) states that: “Compliance is determined by inspection of all documentation required by this standard including the risk management file”. However, the documentation must also demonstrate traceability between important artefacts and activities such as user requirements, system requirements, detailed design, risk analysis and risk mitigation. This focus on documentation is one of the arguments for the unsuitability of modern agile SDMs which tend to place less value on generating documentation. ... focuses a lot of attention on generating documentation, and operates a typical phase gate approach within the development life-cycle to
171
ensure the necessary deliverables are generated. Figure 4-14 is a graphic representation of the ... life-cycle, which shows that each step of the development phase has clearly defined entrance and exit criteria. The Figure shows the criteria bubbles of the Planning and Specification steps expanded for illustration. Figure 4-37 A Graphic Representation of the ... Life-Cycle showing Entrance and Exit Criteria
The effort involved in generating all this documentation is something that many employees are unhappy with. As one principal engineer said: “Everyone using this [the process] is quite frustrated by how much time we spend having to document and test stuff .... And it costs a lot of money”. There are, however, some practices which ... performs in order to minimise this effort. Pre-defined templates are readily available and numerically identified for quick reference and retrieval. In addition a document identification system is described, and together with a clear technique of linking documents through the inclusion of the relevant document number, provides a relatively easy method for maintaining traceability. As shown earlier, the regulations differentiate between high risk and low risk devices and software, and seek more or less documentary evidence respectively. In order to capitalise on this, ... developed a techniques called ‘scaling’. Scaling is performed by using the level of concern (discussed earlier) and then using the scaling matrix (see Table 4-5 7 ) to determine what deliverables are mandatory/required (R) or optional (O). In addition the guidelines suggest that for less complex systems, several deliverables can be combined into a single document. Table 4-20 ... Documentation Scaling Matrix
7
OTS = Off The Shelf
172
These techniques are all aimed at minimising the effort expended on documentation creation and maintaining traceability, and therefore are supportive of a lean approach to reduce waste while ensuring regulatory compliance
[Lean/Software
Development],
[Software
Development/Regulations]. Finally, the use of the E-PDM software application (Enterprise Product Data Management) enables secure handling of project documents, automates workflow, enables multi-user design collaboration across all the ... sites while maintaining a full design and revision history. This is used to demonstrate a controlled process around documentation handling and as a central repository to retrieve the necessary documentation during audits. Within the evolving Conceptual Framework, the Regulatory component remained prominent, however, the constituent exemplars were in need of change to reflect those elements identified from the research data. Some of the existing exemplars were supported by the research, for example, the importance of organisational support in formulating compliant policies and procedures and ensuring people are adequately informed. One aspect which clearly needed to be made more prominent within the Regulatory component was the need for a constant risk analysis mindset. Although already identified that the regulations call for risk management (ISO, 2007), it wasn’t until I had reviewed the process in ..., that the true impact of pursuing proper risk management was revealed. From this perspective it demonstrates a reciprocal relationship between the Regulations and Software Development framework components. The regulations, with a requirement for constant risk analysis, shape the types of SD practices which are adopted [Regulations/Software Development]. At the same time, practices that are adopted due to Organisational influences [Organisation/Software Development] or Lean influences [Lean/Software Development], must do so in a way which satisfies the regulations [Software Development/Regulations].
173
In addition, a magnification was needed around the ‘Burden of Proof/Audits’ exemplar. These terms were replaced with ‘Verification and Validation’ and ‘Formal Audits’. The terms Verification and Validation are more accurate within the MD SD vernacular (Vogel, 2010) and formal audits are a significant presence in the lives of people within the industry. Not only are such audits a significant investment in time and money, but a successful outcome is imperative and significant preparation is involved. An influencing relationship between the regulations and organisation components [Regulations/Organisation], and regulations and people components [Regulations/People] is identified by this. Within ..., following an audit in 2005, a deficiency was found, where they were unable to show traceability back to a user requirement for a particular controlling software parameter. The serious nature of this instigated an expensive corporate initiative to revamp the SD processes. As a result of the observations and analysis concerning the regulatory focus and awareness within ..., the practices shown in Table 4-6 were added to the SaLeS-4MD model in addition to a number of existing practices being validated.
174
Table 4-21 New Regulatory Focused related Practices following the ... Case Study
Lean Principle
Lean Practice
Build Quality In
Develop a training schedule Define a document identification & linking process
Build Quality In
Use a documentation management tool
Create Knowledge
Build Quality In
Identify the responsible party
Eliminate Waste (Avoid OverProcessing)
Consider if in R&D phase
Eliminate Waste (Avoid OverProcessing)
Documentation grouping
Description Employees have to be made aware of the regulatory requirements
Assists in traceability Used to secure documents and automate workflow, including remote collaboration Regulations expect a designated authority responsible for quality management
R&D phase not governed by the regulations For less complex systems, deliverables can be grouped into a single document
175
Potential Issue in MD Domain
Mitigation
Regulator may seek evidence
Record who has attended training
Different focuses at different levels within the organisation Regulator may seek some high level indication of verification
Ensure somebody has responsibility for keeping up to date with the regulations
High level verification report is prudent
Eliminate Waste (Avoid OverProcessing)
Use predefined templates
Eases capture of verification results
4.2.3 Expert Opinion During Phase 2, I carried out a lightweight validation to ‘sense check’ the emerging process model. This was achieved by seeking opinion from three industry practitioners, a form of peer-debriefing (Creswell and Miller, 2000). The feedback from one practitioner, although an expert in aviation related projects, was not used due to his lack of experience in the MD domain. The input from this was positive, mostly supportive of the model practices, but also served to provide me with an increased understanding of MD SD and the potential issues. For example, in querying their opinion on using an iterative development approach, Practitioner 1 supported it and suggested that the focus should be on ensuring that information is accessible and flows along iterations so that it is not misinterpreted. Practitioner 2 answered this question in a more comparative fashion saying that although initial development costs might be less with an iterative approach, he thought that the maintenance phase would be less cumbersome if a waterfall approach is taken since more is done upfront. In response to the practice of ‘just enough’ up front planning, Practitioner 1 cautioned that: “the information needs to be detailed, short, and understood at all times with no coded information”. In addition, Practitioner 2 suggested that mistakes made with such an approach could lead to an entire iteration being scrapped. Clearly the advice here is that great care is needed with this type of practice to ensure that ‘enough’ is actually achieved. In a similar way both Practitioner 1 and Practitioner 2 cautioned against the use of user-stories as requirements, with concerns that important data may be missed resulting in an incorrect product being build: “... if the details are left out, and you just look at the entire forest, you may not actually develop what the user actually needs” (Practitioner 2).
176
The emphasis therefore must be put on expanding the user story once it has been prioritised for an iteration, ensuring the right level of detail is captured. A last example of this input gathering is question five from the mini survey (see Appendix H for the full survey): “From a validation point of view, the project plan would designate specific iterations (not all) to be fully validated, with a full system validation at the end. Would you see any problems with this?” Both Practitioner 1 and Practitioner 2 were in support of this practice as long as the rationale for it was specifically documented: “... as long as justification is documented on why there is no need to validate specific iterations, then it will meet regulations” (Practitioner 2), and “… will stand up in a court of law” (Practitioner 1). In fact Practitioner 2 suggests that the software will: “... also have a lower risk associated with any additional changes or iterations implemented thereafter”. Feedback from all 10 questions was analysed and served to support the way the overall process and specific model practices had evolved. This instilled confidence in the research approach as the study progressed.
4.2.4 Conceptual Framework Changes The initial Conceptual Framework from Phase 1 served well as a general overview of the core components revealed from the literature review, and also as a guide for the empirical data collection. However, it became apparent during Phase 2 that it lacked a certain level of depth and academic conformity that would benefit the reader. Although the intention was to generate a concise framework which did not ‘swamp’ the reader in too much detail, it needed to incorporate additional detail which was proving important enough to warrant inclusion. For example: -
the framework didn’t capture the difference between strategic and operational thinking evident within lean thinking (Hines et al., 2004)
-
the importance of the relationship between the software development process and the business vision and goals (Vähäniitty and Rautiainen, 2008) was only partially represented
177
-
the reality that whatever the defined process is, the actual practices which are finally used are formed by a collection of overt and covert influences (Fitzgerald, 1995).
The problem, however, was that to include all the above would lead to a framework which was much bigger and detailed than desired. I therefore settled on basing the framework mainly on a adaption of the Methods-In-Action model (Fitzgerald et al., 2002). An important aspect of the Methods-In-Action model is the central role given to what is referred to as ‘Developers’. This highlights the fact that it is people and not methods which develop systems. The term, however, is used in a broad sense and is intended to represent the multitude of stakeholders involved in the process. Additionally, the initial framework intentionally took a somewhat minimalist approach to the relationships between the various components, choosing a rather unidirectional depiction from the regulations through to the SDLC influences and onto the SDLC itself. The research revealed that such a simplistic view of the relationships was inadequate in terms of portraying all of the significant aspects of the problem area. The new framework structure therefore no longer depicts a quasi-hierarchical structure but appropriately presents the components in more of a network formation and includes the important relationships between them as identified in the previous sections. The data analysis was performed using qualitative techniques. The data was captured electronically with the aid of the NVivo application, where I performed both open and axial coding. A summary of the coding produced a report with 132 entries against 9 axial codes. Utilising the initial conceptual framework as a guide, a further refining of the data resulted in the five categories which became the main components of the conceptual framework, namely: Regulatory Requirements, Project Characteristics, Organisation, Lean Thinking, and Human Engagement. Table 4-7 shows a sample of the coding results.
178
Table 4-22 Sample table of the data analysis results Component
Axial Code
Open Code
Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements
Regulations
Regulatory Awareness
7
33
Regulations
Corporate Policy
3
24
Regulations
Traceability
3
17
Regulations
Risk Analysis
4
7
Regulations
Interpretation
3
7
Regulations
Audit
2
7
Regulations
Compliance
4
6
Business Focus
Not a software house
3
6
Regulations
Industry Sector
1
6
Regulations
Deviation
3
4
Regulations
Regulatory Effect
3
3
Regulations
Remediation
1
3
Regulations
Post Market Surveillance
1
1
Regulations
Regulatory Challenge
1
1
Regulations Software Development Practices Software Development Practices Project Management Software Development Practices Software Development Practices Software Development Practices Business Focus Process Improvement Software Development Practices
Standards
1
1
Tooling
5
26
Verification & Validation
10
41
Requirements
6
13
Complexity
2
6
Artefact Re-usability
4
10
Legacy Systems
2
5
Project Phases
4
10
Tailoring
1
2
Implementation
1
1
Project Characteristics Project Characteristics Project Characteristics Project Characteristics Project Characteristics Project Characteristics Project Characteristics Project Characteristics Project Characteristics
Sources
References
The rest of this section presents the evolution of the Conceptual Framework, describing how the empirical data analysis led to the modified components and
179
exemplar themes within them. The complete Conceptual Framework is described in Chapter 7. 4.2.4.1
A Methods-In-Action Model
The Methods-In-Action model places the actual development practices at the centre as opposed to the defined process: “In actual development practice, formalised ISD methods are rarely applied in their entirety, nor as originally intended by their creators” (Fitzgerald et al., 2002). This fit well with what the empirical research had discovered. This tailoring of the software process is a recurring theme in literature (Basili and Rombach, 1987), (Boehm, 1996), (Fitzgerald et al., 2003), (Beecham et al., 2011) and especially so when it comes to areas which are particularly context sensitive (Paige et al., 2008), (Taylor et al., 2008), (Srinivasan et al., 2009), (Sabetzadeh et al., 2011). Figure 4-15 shows the new focus of the conceptual framework now being the collection of practices which may result as part of the SDLC. Also included are the other main components and their inter-relationships indicated with the arrowed lines and associated comments.
180
Figure 4-38 The Conceptual Framework Components
4.2.4.2
Lean Lens
From the data analysis in both the case study and reflective analysis, the updated framework has the ‘Lean Lens’ component renamed to ‘Lean Thinking’ and has also decoupled it from the Regulations component. The name change is somewhat academic in nature since the idea is to reflect that the framework component is more a mindset or philosophy as opposed to anything more tangible. Both names achieve this, however the term ‘Lean Thinking’ appears to be a more common reference (Naylor et al., 1999), (Hines et al., 2004), (Oakleigh Consulting, 2007), (Oppenheim et al., 2011). The decoupling was performed so as to highlight the significance of Lean Thinking in its own right independent of whether the domain is regulated or
181
not. The importance of the regulations with respect to a lean mindset is then reflected in the choice of situated lean practices. Following this decoupling, the Lean Thinking component warranted the inclusion of some exemplars. Figure 416 shows the seven principles as identified in the SaLeS-4-MD model, added to the component.
Figure 4-39 Lean Thinking exemplars
4.2.4.3
Regulations
This component’s name was renamed to ‘Regulatory Requirements’ to portray the essence of its inclusion in the framework. The research clearly showed that the regulations are a critical piece of the software development process. Companies design their processes and choose their practices in order to satisfy the specific requirements of the regulations. The corresponding data analysis is summarised in Table 4-8.
Table 4-23 Data analysis for ‘Regulatory Requirements’ component Component
Axial Code
Open Code
Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements
Regulations
Regulatory Awareness
7
33
Regulations
Corporate Policy
3
24
Regulations
Traceability
3
17
Regulations
Risk Analysis
4
7
Regulations
Interpretation
3
7
Regulations
Audit
2
7
Regulations
Compliance
4
6
182
Sources
References
Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements Regulatory Requirements
Business Focus
Not a software house
3
6
Regulations
Industry Sector
1
6
Regulations
Deviation
3
4
Regulations
Regulatory Effect
3
3
Regulations
Remediation
1
3
Regulations
Post Market Surveillance
1
1
Regulations
Regulatory Challenge
1
1
Regulations
Standards
1
1
In line with the number of items in the Lean Thinking component, I included the top seven items from table 4-8 as exemplars for this component as shown in Figure 4-17.
Figure 4-40 Regulatory Requirements exemplars
4.2.4.4
Organisation
The Organisation component remains prominent within the framework and was not in need of renaming. However, the results of data analysis shown in Table 4-9, were used to update the exemplars displayed in the framework. Table 4-24 Data analysis for ‘Organisation’ component Component
Axial Code
Open Code
Sources
References
Organisation Organisation Organisation Organisation Organisation
Process Business Focus
Policies & Procedures
11
43
Cost
5
22
Organisation
Structure
8
21
Business Focus
Time To Market
4
18
Project Management
Project Prioritisation
6
15
183
Organisation Organisation Organisation Organisation
Business Focus
Industry Sector
3
12
Organisation
Visibility
4
10
Project Management
Resource Allocation
2
6
Business Focus
Risk Averse
1
2
Unsurprisingly, ‘policies and procedures’ are still a key aspect to this component, however what is more interesting is to see how prominent ‘cost’ and ‘time to market’ are. This would be reflective of the realities of the MD business sector and the increased focus on these aspects indicating the growing competitiveness of the market. Interestingly, the prioritisation of projects shows the influence of organisational management on the actual projects being worked on. Figure 4-18 shows the updated Organisation component.
Figure 4-41 Organisation exemplars
4.2.4.5
Software Management
In the case of Software Management, although it identifies and consolidates some important aspects of software development in general, the more focused nature of the research brought me to identify a significant effect caused by the nature of the individual projects. This effect was somewhat identified in the earlier framework through the reference to hardware dependency in both the Project Management and Software Practice sub-groups, but evidence demonstrated that it was not prominently enough represented. Since one of the functions of the framework is to represent the key factors or constructs of the study area, and the research was demonstrating the importance that the project
184
particulars played in the SDLC practices, I replaced Software Management with ‘Project Characteristics’ as a key influencer in the framework. The data analysis results for this component are shown in Table 4-10 with the first seven being inserted as component exemplars as shown in Figure 4-19. Table 4-25 Data analysis for ‘Project Characteristics’ component Component
Project Characteristics Project Characteristics Project Characteristics Project Characteristics Project Characteristics Project Characteristics Project Characteristics Project Characteristics Project Characteristics
Axial Code Software Development Practices Software Development Practices Project Management Software Development Practices Software Development Practices Software Development Practices Business Focus Process Improvement Software Development Practices
Open Code
Sources
References
Tooling
5
26
Verification & Validation
10
41
Requirements
6
13
Complexity
2
6
Artefact Re-usability
4
10
Legacy Systems
2
5
Project Phases
4
10
Tailoring
1
2
Implementation
1
1
Figure 4-42 Project Characteristics exemplars
4.2.4.6
People
The People component of the framework was renamed Human Engagement. This was done so as to reflect more vividly the nature of the involvement of
185
various stakeholders in the process. The title ‘People’ was too generic and the research had identified some human characteristics which needed a more appropriate description. Some of the characteristics identified within the original framework are still valid, such as competency and communication preferences, however, much more data emerged from the research around more psychological and emotional concerns. What the research showed was the importance of these factors in terms of influencing the way in which people engaged in the various processes and performed their individual functions. This is reflected by the title ‘Human Engagement’ and the relationship arrows joining it to the Project Characteristics component and the Situated Practices cloud in the centre. The results of the data analysis are shown in Table 4-11.
Table 4-26 Data analysis for ‘Human Engagement’ component Component
Axial Code
Open Code
Sources
References
Human Engagement Human Engagement Human Engagement Human Engagement Human Engagement Human Engagement Human Engagement Human Engagement Human Engagement Human Engagement
People Factors People Factors
Dissatisfaction
8
93
Job Differentiation
8
51
Project Management
Project Management
5
22
Process
Process Avoidance
6
21
People Factors
Knowledge and Skills
4
19
People Factors
Teamwork
6
16
People Factors
Expectations
4
13
People Factors
Staff Shortages
4
11
Process
Informal Processes
4
11
Business Focus
Quality Focus
5
10
The satisfaction people had with their job and how they viewed the positioning and security of their position came out as strong motivating factors for how they engaged in their work. ModusLink, for example, employed a large number of IT professional in positions such as business analysts, database
186
administrators, project managers and software developers. The nature of the business however meant that the technology did not need to change hugely for long periods of times and there was little appetite for introducing more modern techniques. This led to a form of technological stagnation for many of the IT professionals, and de-motivation when it came to carrying out ‘yet another customer integration project’. This was exacerbated when it came time to upgrade the enterprise resource planning system to SAP. Although viewed as an exciting opportunity by many, the choice of resourcing the project was to employ contract specialists, which would ultimately lead to make existing resources redundant. The effect was noticeable with a much lower level of willingness to carry out the normal development activities to the same degree of excellence or fully engage with the implementation project team. Within ..., a similar situation in terms of stagnations, was identified. This manifested itself through dissatisfaction by a number of interviewees with for example: -
the overall process: “You can do all this [the process] and deliver [bad quality] to your customer ... which is why it’s seen as a failure in the overall organisation. Project managers and R&D complain about it, engineering managers complain about it, nobody understands it, it’s just a noose, it’s just a pain really”
-
how work was allocated: “There’s an awful lot of problems with our resourcing and scheduling. I am supposed to work on a project next week that I hadn’t heard about and I’m already scheduled for”
-
and the lack of cross-training that occurs: “there’s an awful lot of cross skills that could improve things if everyone had them” (a software developer).
This has also manifested itself through individuals taking up further education in order to developer their marketability: “I’m doing a degree in mechatronics. ... I’ve been trying to get into this area [XXX’s area] for a while”. Motivation therefore, although already identified in the earlier version through the literature is now given greater significance through the inclusion of a number of more detailed characteristics within the Human Engagement framework
187
component. Figure 4-20 shows the list of exemplars added to this framework component.
Figure 4-43 Human Engagement exemplars
4.2.4.7
The Updated Framework
Figure 4-21 shows the updated conceptual framework structure which emerged following the empirical data analysis, and includes a list of some of the prominent exemplars which characterise the components and suggest the relationships between them. Chapter 7 provides a detailed description of the full framework.
188
Figure 4-44 The updated conceptual framework
4.2.5 Enfolding Literature The SaLes-4-MD model had grown to contain almost 200 practices, and was in need of some form of external review to assess its general usefulness. This was performed by engaging with other academics and researchers in the form of presenting the process model to them, discussions and soliciting feedback. In addition, I presented to an industry partner, a team from the telehealth division of the S3 Group, a leading company in telehealth consultancy. Feedback from these reviews identified two general weaknesses with this version of the model. Firstly, although the practices were grouped into lean categories, it was difficult to see and there was no guidance as to how they related to the wider organisational context and where in the life-cycle of MD software they could be applied. The second was that there were too many practices to be adequately
189
validated in order to bring some closure to the research within the time available. 4.2.5.1
Mapping to the SPICE Framework
In order to address the first issue, a process model overview was developed. The overview is necessarily at a high level and as such identifies the core underpinnings of the model. The process model was developed by utilising the SPICE software process model as described by part 1 of ISO/IEC 15504 (ISO/IEC, 2004). This is also in keeping with how the Medi-SPICE model has been structured (McCaffery and Dorling, 2009, McCaffery et al., 2010). The SPICE process model consists of 35 processes grouped into 5 process categories, and defines 6 maturity levels. The five process categories are: Customer-Supplier (including processes directly impacting the customer), Engineering (processes that directly specify, implement or maintain a system and software product), Management (processes which establish, co-ordinate and manage the project and related resources), Support (processes enabling and supporting the performance of other processes in the project), and Organisation (including processes which establish the business goals and develops the requisite assets to help achieve those goals) (Simon, 1996). Figure 4-22 shows a depiction of these categories and how they relate to each other. By using the SPICE categories and following a similar visual representation, the model overview was made as intuitive as possible. It is therefore especially useful for anyone already familiar with software process improvement and the SPICE framework.
190
Figure 4-45 Process Categories of the SPICE Software Process Model (Simon, 1996)
Four of the five SPICE categories were used, around which to structure the model overview, now named ‘Safe and Lean Software for Medical Devices’ or SaLeS-4-MD (Figure 4-23). The process category which was not incorporated into the model overview is customer-supplier, since the research did not look at any depth into that aspect of software delivery within the MD domain. However, since it is an important aspect, it is anticipated that later iterations of the model would look to address this.
Figure 4-46 The SaLeS-4-MD Model Overview
191
The software actually gets developed within the inner Engineering layer. The evolutionary nature of the SaLes-4-MD process model must therefore be shown within this layer, with a number of different activities occurring during each iteration. Many of the model’s proposed practices are relevant within this area as they relate to actual coding related activities. From a conceptual perspective this corresponds to the shift in focus towards the specific SD practices within the SDLC. A detailed description of them can be found in Chapter 5. Moving outwards, the Management layer represents activities such as project planning, quality and risk management, and resource scheduling, which aim to ensure the smooth operation of projects. This highlighted the link between software management (now named Project Characteristics) and the specific software practices within the evolving Conceptual Framework (see Figure 7-1). Although this serves well as a high-level process overview, it is somewhat over-simplistic since it hides some important process characteristics and relationships such as specific project requirements and personnel concerns. These are therefore expanded upon within the Conceptual Framework. From a MD perspective this layer ensures the regulatory requirements are met, for example, the implementation of a quality management system (ISO, 2003) and a risk management system (ISO, 2007). It is important to highlight the importance of this within the SaLeS-4-MD model and the influence it has on the activities within the iterations. For example the application of risk management which is paramount within MD SD must be applied throughout the entire lifecycle of the project. Therefore the arrows from the management layer are specifically depicted extending to all iterations of development, signifying that specific risk management activities have to be occurring within them. The outer Organisation layer shows that all related software development activities are governed and guided by the organisational goals. From a MD perspective, the organisational management team has responsibilities to provide the appropriate environment for the employees and in particular ensure they are adequately trained and informed about the relevant
192
regulations. Again, from a conceptual perspective, this better reflects the multiinfluential nature of the organisational component and the inter-relatedness of all the components. However, within the Conceptual Framework these needed to be more clearly defined. The final layer, Support, is shown at the bottom of the model indicating that its function is to support the activities happening in other areas. Similar to the Management and Process Improvement layer, there are arrows which point directly into the evolutionary cycles, again indicating that certain activities such as verification and validation must occur at these points. For example, due to the inconsistent and interchangeable use of the terms verification and validation, the FDA published a guidance document on the general principles of software validation (CDRH, 2002). In it they state that: “In practice, software validation activities may occur both during, as well as at the end of the software development life cycle to ensure that all requirements have been fulfilled”. This is reflected in the SaLeS-4-MD model by mandating that verification and validation, similar to risk management, are activities that need to be occurring during each iteration. However, in order to strike a balance between performing too much and not enough validation, the model proposes that specific iterations be designated to undergo more rigorous validation, and the final iteration should include a full system validation step. The choice of validation iterations is therefore project specific and should be decided and recorded as part of the initial project planning. Tying this back into the Conceptual Framework, this support function is seen by the links coming from the Organisation component to Human Engagement and Project Characteristics components. In addition this level of support will influence the specific practices adopted within the SD process, indicated within the framework by the link from Organisation to the central component of Situated Lean SD Practices. On the left hand side of the Engineering layer the model includes a shaded ‘Project Plan’ box. This is specifically identified within the model to reflect its importance from a regulatory perspective as mandated by IEC 62304 section 5.1 “Software Development Planning”. This box intentionally overlaps into the
193
Management & Process Improvement layer indicating that this high level plan needs input from multiple stakeholders in order to achieve its goal of addressing: the processes to be used, the deliverables, traceability requirements, configuration and change management, and problem resolution. Within the Conceptual Framework this is indicated by the link between Project Characteristics and the central Situated Practices box, and the relationships between Origanisation, Project Characteristics and Human Engagement. In particular such exemplars as Owner, Estimates, and Time to Market, represent this process requirement within the Conceptual Framework. Similarly, on the right hand side, the model specifically includes a ‘Deliver’ box. This signifies the need for a formal software release as mandated by IEC 62304 section 5.8 “Software Release”. Again this box intentionally overlaps the higher model layers to indicate that it is delivered to parties external to the development groups and so must be evaluated and documented appropriately. To conclude the model overview construction, Figure 4-24 shows a depiction of the SaLeS-4-MD model when a R&D phase is present in the process. As suggested in the model, the R&D phase does not need to be as tightly controlled and tracked as within the normal development. Therefore a number of initial iterations are designated within the R&D phase and consequently the influence of both the Management and Support layers are less. This is depicted by narrower lines pointing from the respective layers into the evolutionary circles.
194
Figure 4-47 The SaLeS-4-MD Process Model Overview with an R&D Phase
4.2.5.2
Mapping to the Medi-SPICE Model
The second issue identified during the validation review was that the collection of practices within the SaLeS-4-MD model had become quite large and some way of narrowing the focus for a validation phase was needed. The Medi-SPICE model was chosen as a means to address this issue. It provided a prime framework against which to map the specific practices and evolve the SaLeS-4-MD model to Version 2. In order to focus the study, just one of the Medi-SPICE process areas was chosen, namely ‘Software Construction’ (Figure 4-25). The Software Construction process was specifically chosen for a number of reasons. The Medi-SPICE project was still in progress at the time of this research, and not all process areas had been completely defined. The shaded bubbles in Figure 4-25 show the process areas available at the time, which were 1, 2, 4, 6, 7, and 8. This therefore limited the possible areas to choose for a validation phase.
195
Figure 4-48 The Medi-SPICE PRM showing completed Process Areas
In addition, a review of the practices within the SaLeS-4-MD model, together with the mapping carried out during Phase 1 (Figure 2-15) indicated a large influence on the development phase within the life cycle. Finally, a consultation with a senior researcher on the Medi-SPICE project highlighted the fact the ‘Software Construction’ process area was unlike the other process areas in that it covered more than one life-cycle component, namely it incorporated ‘unit testing’ as well. This therefore would result in a larger coverage of the SDLC than if a single component/area was chosen. The construction process area within the Medi-SPICE reference model is composed of 11 specific practices as shown in Figure 4-26. These are recommended practices for companies developing medical device software and although relatively high-level in nature, are accompanied by a detailed description within the Medi-SPICE documentation. This made mapping the specific practices from the SaLeS-4-MD model a relatively straight forward task.
196
Figure 4-49 The Specific Practices of the Medi-SPICE Software Construction Process Area
For example, the first specific practice within the Medi-SPICE Software Construction process area is: “SP1. Define and establish a unit verification strategy”. Within the corresponding assessment documentation for this specific practice it states: “Develop and establish a strategy for verification and reverifying the software units. The strategy should define how to achieve the desired quality with the available techniques”. A review of my model practices identified two lean practices relevant to the verification process: 1. Consider if an R&D phase exists and plan for a verification process with less overhead during that phase 2. Decide on the level of concern for the software and plan to verify accordingly, thereby possibly reducing the effort where the level of concern is low. The second specific practice is: “SP2. Define and establish software unit verification criteria”, which is described within the Medi-SPICE documentation as: “Develop and document criteria for verifying that each software unit satisfies its design, functional and non-functional requirements in the verification strategy”. A critical aspect of this is to document the criteria, and a review of the
197
model produced the following relevant practice which promotes a parsimonious attitude to documentation: -
Just enough documentation. Writing documents that are never read by anybody is a waste of time and effort. A blanket approach to writing detailed documents should be avoided by considering who the target audience is and what they are interested in seeing.
The third specific practice is: “SP3. Define and establish software unit acceptance criteria”, and is described as “Define and establish acceptance criteria for software units prior to integration into larger software items as appropriate, and ensure that software units meet acceptance criteria”. The practices from the model, shown in Table 4-12, are relevant to this. Table 4-27 Practices in Support of Medi-SPICE Model Specific Practice 3 Lean Principle
Eliminate Waste (Avoid OverProcessing)
Eliminate Waste (Avoid OverProduction)
Lean Practice
Description
Potential Issue in MD Domain
Mitigation
Source
Re-use existing artefacts
User stories become the requirements
May not have enough detail for regulator
Include clear conditions of satisfaction
Shoemaker & Van Shooenderwoert 2010
Each iteration gathers extra detail for its user stories
Shoemaker & Van Shooenderwoert 2010
Only for this iteration
Concentrate only on what is required for this iteration
Hibbs et al 2009 May not have enough detail for regulator
Each iteration gathers extra detail for its user stories
Shoemaker & Van Shooenderwoert 2010
This process was followed for the rest of the Software Construction specific practices, resulting in 97 lean practices mapped to the 11 Medi-SPICE practices. The SaLeS-4-MD model was updated with this new structure and Figure 4-27 shows a snapshot of the resulting model. This new structure of the model now gives a clearer and more relevant view of how the specific lean practices fit within a MD SDLC. Figure 4-27 shows the Medi-SPICE activity 5.5 as an example. From the Medi-SPICE mapping to the SPICE framework, we can see
198
that this belongs within the Engineering layer of the overall process. Version 2 of the model mirrors these existing SPICE and Medi-SPICE linkages. The full Version 2 can be seen in Appendix L.
199
Figure 4-50 Extract from the SaLeS-4-MD Model Layout Version 2
200
4.2.5.3
Other Important Literature Contributions
As in all research, it is important to keep up to date with the latest developments, including an ongoing review of publications both academic and popular. For this study, it is worth drawing attention to two contributions to the field, published while the research was underway, and which were very relevant to this work. Medical Device Software Verification, Validation and Compliance (Vogel, 2010). David Vogel has been at the forefront of software’s evolving role in medical devices (Vogel, 2011) and his book gives very detailed information on medical device software development with specific focus around the regulations. Much of the book’s focus is around verification and validation with respect to what the regulations look for, and he suggests best practices to developing software by having a clear understanding of what the regulations are trying to achieve. Through this, he advocates a sensible approach of doing what is necessary to meet those needs, and not adopting a heavy handed process. The information is taken from firsthand experience within the industry and as such adds a level of validity to the suggested practices. A careful review of the book identified practices already in the model but also some new practices which enhanced the model. For example, on page 253 he tells us that in relation to testing activities: “... preliminary testing during the implementation of the software is probably the most productive activity for finding defects in the software”. Consequently the model’s practice of employing test driven development which specifically aims to build testing into the entire development process, is considered supported. In relation to documentation, he suggests that different but related documents should resemble each other structurally, thus making them easier to review and
201
relate. Since this intuitively supports a lean approach, it resulted in a new practice being introduced to the model. A complete review led to the model changes as shown in Table 4-13.
202
Lean Principle
Lean Practice
Create Knowledge
In-code documentation
Eliminate Waste (Avoid OverProcessing) Build Quality In / Eliminate Waste (Avoid OverProcessing) Eliminate Waste (Avoid OverProcessing) Build Quality In
Documentation grouping
Table 4-28 SaLes-4-MD Model Changes incorporating Vogel’s Concepts Description Potential Mitigation Source Issue in MD Domain Put explanatory lowVogel level detail in the 2010 source code itself Try and organise code Vogel to resemble the SRS 2010 and SDS documents
Use similar documentation structure
Documents that resemble each other are easier to review and relate
Documentation grouping
Map sections instead of every single detail
Could be seen as cutting corners
Use Standards
Use coding standards and guidelines
Standards are not enforced
Build Quality In
Re-use existing artefacts
Build Quality In
Perform automated
Re-use existing code/components/OTS applications Supports GPSV's guidelines of
Tendency to lightly test or not test at all Hardware may be
Vogel 2010
Drobka et al 2004
Don't make sections too big
Vogel 2010
Case study
Incorporate check of standards into code review process Include as part of validation plan See SP4 (TDD approach)
Case study
Vogel 2010
Vogel 2010
Graaf et al 2003 Hibbs et al
203
Cordeiro et al
Case Study
Case study Shoemaker & Van
Vogel 2010
Poppendeick &
testing Build Quality In
Pair Programming
Build Quality In
Don't overdocument
"consistently fulfilling" requirements Use pair programming as a means of continuous review Keep maintenance documentation to a minimum
unavailable Familiarity breeds lax practices Products have long life support requirements
2007 Good oversight can make this work well Have non-team members evaluate the documentation's adequacy
204
2009
Vogel 2010 Drobka et al 2004
Vogel 2010
Shooenderwoert 2010
Poppendieck 2003
Guidance on the use of AGILE practices in the development of medical device software (AAMI, 2011). One of the motivations for this research was the lack of available literature and guidance on how lean software development can be applied with the MD domain. From the closely related agile perspective, a similar gap had also been identified by industry, resulting in the AAMI (Association for the Advancement of Medical Instrumentation) standards board drafting this technical information report (TIR). The TIR provides “... recommendations for complying with international standards and [U.S. FDA] guidance documents when using agile practices to develop medical device software”. Due to the inclusion of many agile practices within this study’s model, the draft TIR adds legitimacy to this research and can act as an indicator as to the validity of some of the model’s practices within a MD context. Caution is needed here however, since at this point in time the TIR is only a first draft and has been circulated to the committee members to solicit feedback. Nevertheless, some of the TIR committee members are familiar to me, with some of their work having been referenced in this thesis. A certain amount of confidence in the material is therefore achieved, however, the TIR is not used as a source of reference within the model itself. The iterative approach advocated by this study’s model is supported by the draft TIR by means of the mapping depicted in Figure 4-28. In it they depict how the requisite activities, as specified in IEC 62304, might be executed in an agile incremental/evolutionary life-cycle. An iterative life-cycle is a common component of both a lean and agile approach.
205
Figure 4-51 Mapping IEC 62304 Activities into an Agile Incremental Life-Cycle (AAMI, 2011)
Considering the lean approach of waste elimination within the SaLeS-4MD model, the guidance document presents the following suggestions which support some of the model’s practices such as, “Just enough documentation” and “Standardise documentation”: -
-
Produce documentation that has business value Produce documentation that satisfies the intended audience's use of it Produce documentation when it fits the flow of creating it and using it Define how documentation is written, controlled and approved as a sum of its parts Stories must be detailed enough to allow the developer to effectively elaborate and implement the story Well elaborated stories can be used as part of a final requirements documentation mechanism Executable requirements may be used as part of a final requirements documentation mechanism Implementation must be designed up-front with regulatory needs fully considered Produce documentation that is valuable to both the development team and regulatory stakeholders.
Another example of how the draft TIR supports the model can be seen by examination of the need to have a well established quality process. This process
206
needs to include things like continuous risk management, verification and validation and be evidenced through some form of traceability. The SaLeS-4-MD model’s practices such as, Automated Testing, Test-Driven Development, Daily Builds, Perform FMEA, Root Cause Analysis, Continuous Integration, Embed Compliance in Process, are all supported by the following suggestions in the TIR: -
Establish effective retrospective and reflection practices to ensure the software development process is controlled Story creation and backlog management integrate well with traditional risk analysis and risk mitigation activities Agile flow from stories to deliverables through backlog tasks supports traceability Stop the line is a good practice to prevent systemic issues from occurring Retrospectives and reflections stimulate ownership of continuous process improvement Risk control measures (such as risk mitigations, labelling) contribute to the creation of backlog items at the story, increment, and release layers Before stories, increments, and releases are completed, a re-evaluation of the safety risk must be done
Agile practices such as Test-Driven Development, pair programming, continuous integration, and continuous testing align well with the regulatory goal of delivering high quality software.
4.2.6 Summary Version 2 of the SaLeS-4MD model and an updated Conceptual Framework evolved from the data gathered during empirical research from a number of different sources. The Conceptual Framework evolved into a more accurate description based around an adaptation of the Methods-In-Action framework. With a central focus now the specific SD practices within this context, some of the components had changed and the relationships between them better defined. Within the SaLeS-4MD model, Phase 2 identified a number of new practices and provided supporting evidence for many of the existing practices. Table 4-14 summarises the changes to the SaLeS-4-MD model in this phase.
207
Table 4-29 Summary of SaLeS-4-MD Changes leading to Version 2
Number of entries in Version 1
162
New Practices Added
24
Existing Practices Supported
27
Practices without supporting evidence Number of entries in Version 2
135 186
The number of practices within this new version had become quite large and it was therefore necessary to consider narrowing the focus of the model in order to facilitate a validation process and achieve some level of closure to the research. Version 2 of the SaLeS-4-MD model was therefore restructured following the application of two academic models, the SPICE framework model for software process improvement and the Medi-SPICE reference model for medical device software development. Through a mapping of the model practices to the software construction process area within Medi-SPICE, a more focused subset of practices was achieved in order to facilitate a validation process. By using the SPICE framework to overlay the model, a process overview was generated indicating the core attributes of the model and how they overlap. In addition, by using the SPICE framework, the process is made more intuitive and conceptually clear for those already familiar with SPICE. The resulting model now contained 11 specific MD software development high level best practices as defined by a peer reviewed model, with 97 empirically based supporting lean practices. As a means of verification, each row also identified the source(s) of the practices, any potential issues and possible mitigations if available. This version of the SaLeS-4-MD model was then brought forward into a validation cycle. The concluding validation Phase 3 is presented in Chapter 5.
208
209
Chapter 5: Validation of the SaLeS-4-MD Model
5.1
Introduction
In Chapter 3 I presented a brief discussion on the role of validation within qualitative research. While some would argue against its relevance, “we as qualitative researchers need to ask ourselves this question: How can we have confidence that our account is an accurate representation?” (Pyett, 2003). This is the central point in the discussion. Since if we want to add to the body of knowledge and help improve existing understanding, then surely we must strive to demonstrate that what we propose has been ‘tested’ in some manner. Within this study, I have proposed a development process which suggested a different approach to SD within the MD domain than is typically practiced. It would be naïve and perhaps even dangerous to assume that the process could be implemented immediately within an organisation. It is therefore necessary that I seek to check the correctness of what I propose and for the purpose of this thesis I call that process validation.
5.2
PHASE 3 – “Validation”
This last Phase of the research included a detailed final validation of the SaLeS-4-MD model, performed via expert opinion. The choice of expert opinion as the means for final validation is discussed in Chapter 3, but the primary objective was to solicit opinion from industry practitioners about the validity, that the specific lean practice(s) were supportive of the respective Medi-SPICE practice and therefore also compliant with MD regulations. Gathering this feedback was achieved using the following two approaches.
5.2.1 The Medi-SPICE Assessment The extended definition of a medical device, which came into effect in Europe in 2010, found many existing companies having their products potentially falling within the new MD definition, and consequently needing to
210
have regulatory compliant software development processes. Due to the work being carried out by our Medi-SPICE research team, several companies were interested in having their software development processes assessed through the group’s model. Therefore, a light-weight assessment questionnaire was developed by the group. As described in Chapter 3, I used this opportunity to solicit data relevant to my model. As an illustration, the following is a sample of the questions and clarifying statements I added to the assessment questionnaire relating to Medi-SPICE specific practice 8 (Verify Software Units): Q: Do you always use the same “Level of concern”? (Different LOCs require different levels of detail) Q: Do you use any automated error prevention/detection techniques? (Practice of Poka-Yoke) Q: Do you enforce coding standards? (Should be built into the code review process) Q: What techniques do you use for root cause analysis? (The 5 whys method will assist in a full understanding of issues) Following this I assessed the responses in terms of the SaLeS-4-MD model and updated it accordingly. For example, a positive/confirmatory answer to a relevant question resulted in the model being updated with a reference to the assessment in the source column, indicating support for that practice (see Table 5-1). While this was very useful in validating practices already in the model, it should be pointed out that it did not lead to any practices being removed or additional practices being added to the model. This was due to the nature of the assessment being one of investigating the company’s existing processes, and not primarily to validate specific practices. Within the Software Construction process area, four lean practices supporting five specific Medi-SPICE practices were found to be supported by the assessment.
211
Table 5-30 SaLeS-4-MD with Sample Validation Points from the Medi-SPICE Assessment
5.2.2 Expert Interviews This section demonstrates the analysis process and the consequent changes made to the SaLeS-4-MD model following the expert interviews. The first 4 specific process areas covering 25 specific lean practices are described below in detail. As a form of referencing, the corresponding row in the model spreadsheet will be identified with the prefix ROW and when referring to a particular expert, I use the prefix E with the corresponding numeric identifier from Table 3-7. For example ROW32 refers to the 32 nd row in the excel spreadsheet which is the lean practice to “Maintain Code Clarity”, and E5 refers to expert number 5 from Table 3-7, the “Global software design control manager”. Of the 53 practices 38 were accepted as they were and 15 were amended following the feedback. 5.2.2.1
Fully Validated Practices
SP1. Define and establish a unit verification strategy Lean Principle: Eliminate Waste (Avoid Over-Processing). Lean Practice 2 (ROW6): Decide on level of concern. All Medical Devices must be assigned a classification, and accordingly any related software must be assigned a similar classification. However, due to the componential nature of software, sub-components may be assigned a lower
212
classification. 4 experts gave opinions on this practice, again with a majority in favour and no one disagreeing with it. This software classification, also known as a level of concern, is a relatively new distinction being allowed within MD software. Consequently, and as E5 points out, there can be a danger that two different agencies may adopt a different focus when it comes to reviewing the LOC a company has adopted. E5 suggests that the FDA in the United States and their European Union counterpart agencies may not take the same view, and so by assuming a blanket LOC for all jurisdictions could lead to compliance issues. He therefore suggests that if you are marketing your product in such a multi-regional context then you should do a mapping from your initial LOC to the other regions in order to determine an equitable level of risk. The other noteworthy point for this practice is from E6 who somewhat agrees. He points out that what will be important is that the architectural design should be carried out in a manner which supports the testability and therefore verifiability of the software, irrespective of the LOC. SP4. Develop software units Lean Principle: Eliminate Waste (Avoid Over-Production). Lean Practice (ROW22): Only develop for current iteration. A fundamental of the lean methodology is to develop in an iterative fashion thereby achieving the benefits of fast customer feedback and the ability to respond more easily to change. Five opinions on this were gathered, no disagreement with it and 3 fully agreeing with it. E3, while stating he somewhat agrees, suggests he is unsure how that would work in their situation since they outsource some software development to a supplier in India. He also is unsure how the regulator would view such an approach. Despite this he said: “I see perfectly well how such an iterative approach would be better. Some projects may take 6 months to deliver, and by that time the customer expectations/requirements have moved on”. Indeed, the global software development aspect is something which adds another dimension to the discussion. The issues which distance introduces
213
(temporal, geographical, cultural) can be significant but are outside the scope of this thesis. The issue of how the regulator would view this is also a concerning topic, and something that has been identified by the regulating agencies themselves. However, the regulators do not stipulate that such an iterative approach is unsuitable, and in fact specifically state in the IEC 62304 standard that they do not favour any particular development methodology. In addition, the Association for the Advancement of Medical Instrumentation have published a draft technical information report on the guidance of using agile practices in the development of medical device software (AAMI, 2011). This should help clarify for both regulatory agencies and MD companies the sympathetic view held by the regulator. E4 somewhat agreed but suggested that it is important that any changes to the code base should be followed by an impact analysis of the code written in earlier iterations. This valid concern is identified and addressed as part of the risk management practices further down in the model but also specifically for this in ROW23 “Perform V&V (verification and validation) during and after each iteration”. ROW23 aims to address the concern around ensuring any changes made do not adversely affect earlier code, something also identified by E4 above. As depicted in the model overview (see Figure 4-23) this is represented by the verification, validation, and risk management arrows pointing into the evolutionary engineering cycles. This practice received 4 opinions, all of which agreed with it. This is not surprising given that all experts are fully aware of how important risk management is to the process. Continuing with this lean practice but looking at the issue of trying to conduct the process in conjunction with non-iterative teams, ROW24 suggests a mechanism of defining minimum interfaces between separate components which allows both teams to work separately while allowing easier integration later.
214
Two experts were in full agreement with this, with E2 drawing attention to a journal article he co-authored with industry practitioners describing how they did this. Similarly E5 agreed but added that it is important to perform periodic validations and a lager one at the end when the teams come back together. E7 somewhat agrees, suggesting that in his experience it makes it very difficult to perform any automated testing. Lean Principle: Deliver fast. Lean Practice (ROW25): Use short iterations. The practice of iterations helps speed up the feedback loop by releasing software to the customer quickly. Making the iterations as short as possible optimises this feedback loop. The issue however is that iteration lengths should be decided on a company by company and even project by project basis. I received 5 opinions on this practice. E1, agreeing with the statement said that within his company the motto was “If you’re going to fail, fail early”. Although E3 is in agreement, he does suggest some issues which need to be addressed such as the slowdown in getting the relevant documented approvals if doing this in a global software development team configuration. E5 agreed and said from his experience the iteration length depends on the device and/or application under development. E2 reported that they use 2 month iterations, while E9 typically uses 4 week iterations but reduces it to 2 weeks towards the end of the project life. While E4 disagreed, he offered no explanation for his response. Lean Principle: Create Knowledge. Lean Practice (ROW26): Develop prototypes. Developing prototypes delivers an unfinished or preliminary version of the product to the customer for review. By setting the prototype up in a way in which the customer can ‘play with’ the system allows them to review and refine their actual requirements.
215
7 expert views gathered on this shows an overwhelming positive opinion of this practice. The amount of completed functionality may vary, with E11 saying they might get a prototype at 70% completion, while E2 suggests that they allow customer access to both their development and validation database instances at an early stage to let them ‘play’ with the system. E11 report that they may actually have a very crude prototype sitting on someone’s desk in the form of a breadboard type electronic solution in order to get something out for people to interact with. This was very positive feedback for the practice. Lean Principle: Continuous flow. Lean Practice (ROW27): Perform load levelling (Heijunka). This practice received somewhat mixed reviews, however, only one who disagreed with it. E3 fully supports this activity citing the need to be particularly vigilant around subject matter experts who can very easily turn into resource bottlenecks. Both E1 and E5, while not disagreeing, suggest that in their experience this is easier said than done. In particular, E5 pointed out that for smaller companies this may simply be an unavoidable consequence of a small workforce. E4 disagreed with the practice but again did not provide any explanation for his choice of response. Lean Principle: Build quality in. Lean Practice (ROW29): Re-use existing artefacts. ROW29 continues the investigation of this practice but suggests that by reusing code, a higher level of confidence in the software is established. This therefore supports the principle of building quality into the process. Mostly there was full agreement with this, with E6 stating that re-use for them is a fundamental practice. While an existing component or module may not be exactly perfect for your requirements, the fact that it has been tried, tested and documented is enough of an advantage over re-writing it. As E6 put
216
it “Better the Devil you know”. E4, somewhat agreed stating that the software still needs to be properly tested. Re-used software which does not pass the tests would naturally affect confidence levels. The tendency might be not to test as rigorously as is necessary. Lean Principle: Build quality in. Lean Practice (ROW30): Take an object oriented approach. Both E5 and E7 agreed with this with E7 saying they develop what they call ‘layers’ of software which aided in modularising the software in a similar fashion. This modularisation helps to partition the system into discrete units which naturally facilitate the software engineering practice of information hiding and support the maintenance of the system into the future. Both E1 and E4 somewhat agreed with E4 simply stating that it is not necessary to develop that way and that in some cases it may not be possible due to lack of language support. E1, also somewhat in agreement suggested that you may not be able to influence that decision. Given the increasing practice to outsource development to a software supplier, you may little if any control over the software architecture. Lean Principle: Build quality in. Lean Practice (ROW32): Maintain code clarity. Although this is more of a principle than a practice, there are a number of specific practices which support maintaining clarity of code. Code clarity is especially useful when working in environments where different people work on the same code. ROW32 suggests the use of an intuitive naming convention within the code. 5 responses were almost unanimous in their support for this. This is not at all surprising given its prevalence in software development in general. Common sense would suggest that software which gives code objects an easy to understand name helps reduce misunderstandings and can better assist in trouble-shooting. E1 only somewhat agreed but primarily due to the fact that he
217
did not write code himself but could clearly understand the benefit within his company’s context. Staying with this practice the next three rows in the model (ROW33, ROW34, and ROW35) received 4, 3, and 4 opinions respectively. These lean practices call for a common language and communication techniques, simple notation, and focused in-code comments to support code clarity and building in quality. All practices received positive feedback. There was a common opinion expressed that some of these can be quite subjective in nature. What one person perceives as simple and clear may not be seen that way by others. The case of utilising outsource sofware suppliers was a case that E8 raises as particalarly benefiting from these practices, especially ROW33. Lean Principle: Build quality in Lean Practice (ROW36): Use a good source code control tool. Having good control around the source code maintains the integrity of the code by ensuring it is secure but also accessible to all developers from a central location. Six expert opinions were all at least somewhat in agreement here. Although perhaps a practice which many would take for granted, E8 pointed out that in many cases there may not be a team of developers but merely a single developer. In such cases he has not had the need for a source code control application. In addition E1 suggests that when operating in an outsourced environment you may not have control on how the outsource partner controls the code.
218
Lean Principle: Create knowledge. Lean Practice (ROW38): Perform in-code documentation. By putting low level information in the code itself, as opposed to an external document, ensures that the information is available where it is needed and to the appropriate people. No disagreement with this practice was recorded. E8 suggests that this is critical due to the growing complexity of modern software. In fact he suggests that modern MD software should read more like a book that source code. Similarly, E10 agrees with the importance of this, citing a recent example where an old piece of software was not behaving the way the documentation suggested and they were relying on the in-code comments to reverse engineer the behaviour. E7 however does point out that the quality of in-code documentation can be a very subjective thing. Some procedural guidance could help to alleviate this. Lean Principle: Eliminate Waste (Avoid Over-Processing). Lean Practice (ROW39): Documentation Grouping. This technique suggests organising the code itself in such a way as to resemble the related documents such as the software requirements specification and the software design specification. This then makes the review process much easier since the connection between them can more easily and intuitively be seen. While E5 fully agreed with this approach he proposed an interesting variation of this which was to also try it the other way round. In other words, since have the specification documents resemble the code. This is in fact supported by E6 when he suggests that the code structure should in fact be influenced by the architectural design and the documentation should therefore follow the code structure. E1 again warns against the inability to control the work of an outsource company. E4 again disagreed with the practice but offered no explanation for his decision.
219
In deciding how these responses affect the validity of the practice I reconsidered what the practice was aiming to achieve. Since the intention is to eliminate waste by making the code and core documents resemble each other, and so reduce the effort involved in tracing between the two, it is not really important which comes first. Dealing with the same practice, ROW40 focuses on reducing the amount of documentation for less complex systems by compressing multiple documents into a single one. General agreement was obtained from 4 experts with E5 stating that he recently completed a project to edit their own SDLC process to allow for this type of flexibility. E7 while fully agreeing adds a cautionary note to watch out that you don’t run into having to version-control pieces of the document. E4, who somewhat agreed, suggested that this can also work for complex systems that have a lower risk assessment associated with them. Lean Principle: Eliminate Waste (Eliminate defects). Lean Practice (ROW41): Practice Test Driven Development (TDD) A TDD approach will help find defects earlier in the development process and consequently prevent a more expensive bug fixing exercise later in the project. E5, E7, and E9 were fully in agreement with this as a good approach. Interestingly, E7 said he would prefer to adopt a behaviour driven development approach. E5 does however warn that an important aspect required to reap the most benefit from a TDD approach is to automate the testing. E4 and E8 somewhat agreed but have some reservations about the potential of TDD within the context of embedded software due to the issues surrounding the availability of the hardware. E8 said that in his experience there are things which a TDD approach will not be able to test because the hardware will not be available, and simulating hardware does have some limitations. Although the concerns raised by E8 here are valid, the model specifically looks into some of these potential issues through techniques described in
220
ROW42 (mocks and stubs), ROW43 (simulators), and ROW44 (h/w abstraction layers). 5.2.2.2
Amended Validated Practices
SP1. Define and establish a unit verification strategy Lean Principle: Eliminate Waste (Avoid Over-Processing). Lean Practice 1 (ROW5): Consider if performing a Research and Development phase. Five expert opinions were recorded for this proposed lean practice with no one disagreeing with the validation statement. Overall the feedback was very positive for this. Not only was there majority agreement, many of the experts reported that is exactly what they do in practice. Interestingly both E4 and E5 are recorded as somewhat agreeing and both highlight the same issue, namely the need to clearly identify where your research and development phase starts and stops. This is also something E7 highlighted. As E4 put it: “... you can't have 12 months of R&D followed by 2 months of development”. The general experience reported is that the auditors do not spend time examining the R&D activities. E11 said that in her 7 years, she has never had an auditor examine their R&D work. It would however, seem prudent to heed the advice of E4, E5, and E7 and have this clearly identified as part of the initial project kick-off. Row 5 within the model was therefore amended to have this added as a condition for the suggested lean practice. SP2. Define and establish software unit verification criteria Lean Principle: Eliminate Waste (Avoid Over-Processing). Lean Practice (ROW9): Just enough documentation. Unsurprisingly this invoked a mix of opinions, with 3 different views being expressed. The need for documentary evidence for regulatory compliance is unquestionable, and is seen by many as a burdensome but necessary output from the process. With much focus being spent on ways to help automate this
221
and thereby reduce the effort, little has been proposed in terms of reducing the actual documentation. In terms of specifying verification criteria, E1 somewhat agreed with the practice but highlighted that they follow a process which only allows for a small amount of latitude in terms what the documentation should contain. E3 however rejected the practice, saying the practice is very loosely worded and would not be acceptable to them. In terms of verification and validation he said: “Our change management process is quite specific in terms of what the change process is ... The FDA scrutinises changes. [The company] doesn’t allow local groups to change the validation process. Strict adherence to corporate process is expected”. While E3’s response is understandable, it would be very typical of what the research has shown to be the case, that corporate policies and procedures govern the process from top down and full adherence to those procedures is expected. In addition, E3 did not elaborate on whether their process allowed for any flexibility similar to what E1 described within his company. In deciding how to treat the feedback here, I carefully reviewed the model practice and how it was described. Considering E2’s input, which highlighted their efforts in automating the documentary evidence, it would follow that if you can automate the process then this certainly is a leaner approach than manually creating it. In that case the amount of documentation is not a concerning factor at the point of generation. However, the practice also makes reference to the intended audience, and in terms of those stakeholders, documentation which is tailored to their information needs, makes their process leaner (easier and faster). Unless it is possible to automatically create different versions of the verification documentation then this would imply the need to in fact try and reduce the amount of documentation generated, but keeping in mind the different stakeholder requirements. Consequently the practice has been amended to include a reference suggesting to try automate the documentation process.
222
SP3. Define and establish software unit acceptance criteria Lean Principle: Eliminate Waste (Avoid Over-Processing). Lean Practice (ROW10): Re-use existing artefacts. Software re-use is a common practice within software development. However, this specific lean practice is extending the concept into other related areas such as the generation of unit acceptance criteria. Particularly within an agile process, there will already exist artefacts which can be re-used for other purposes. 3 of the 4 experts who expressed an opinion on this only somewhat agreed with the practice. All 3 admit that while they agree with the practice in theory, they themselves do not follow a strict agile approach. E3 said that although they intentionally have tried to stay away from what he calls “nimble” methodologies, increasingly their customers are driving them towards such an approach. They currently hold weekly customer meetings which repeatedly generate user requirement specifications. There was no disagreement on this practice and the only caveat was expressed by E5 who suggests that it may depend on the regulator and how they interpret the sufficiency of the user stories. However, as we will see with the next practice, this is already identified within the model as a possible issue, and a mitigation proposed. This practice was therefore amended to include a reference to consider ROW11 (add story details within the iteration) in conjunction with this. The model (ROW11) proposes that each iteration should spend time expanding on the user stories assigned to it and in so doing elaborate further on the acceptance criteria. The intention then is that this extra detail will be enough to satisfy the regulator. Out of 4 responses, no disagreement was expressed, however, 3 experts somewhat agreed. E5 completely agreed but nevertheless added that he thinks this extra detail should include some detailed design work also. E2 and E3, agreed with the practice in theory but it just wasn’t part of their corporate procedure, while E4
223
suggested that you need to be clear about how you define a ‘unit test’. A user story may be interpreted as a unit but a software engineer may implement that story with multiple software units. I have maintained this model practice, however, since the objective of this practice is to define the software unit acceptance criteria, it would seem prudent to stress the importance of its applicability at the software unit level. An amendment to the practice was made to reflect this. SP4. Develop software units Lean Principle: Build quality in. Lean Practice (ROW28): Re-use existing artefacts. Here again the principle of re-use is promoted through the re-use of code. This can be in many forms but particularly discrete components or modules or even full off-the-shelf (OTS) applications, will assist in building upon tried and tested software, thereby saving an amount of rework. In addition, the fact that the software is already being used in a production setting should add a level of confidence to the code. Therefore re-using existing software can speed up the development process, however it should be explicitly highlighted in the validation plan. 3 experts agreed fully with this, citing it as a major form of time saving. In addition E3 said that their approach is to use as much OTS software as possible but then limit the changes that they make to it. By limiting the changes it allows for an easier upgrade process in the future. E1 somewhat agreed saying that he believed it is important to ensure that the validation must include a proper risk analysis of the software in its new context. The practice remains within the model but E1’s concern here is noted. However, ROW97 and ROW98 already identify this as a possible issue with reuse. In addition ROW81 identifies the tendency to not do enough testing on reused software. Following this analysis, the practice was amended with a reference to these other rows being added.
224
Lean Principle: Create knowledge. Lean Practice (ROW37): Daily check out and in of source code. The opinion on this particular practice was divided. While E6 agreed and said it was necessary to have this discipline so that a full and complete system build could be achieved, E7 suggests that it is unnecessary until getting closer to release time. The issue however is that if you are following a continuous integration approach, waiting until closer to release time would not be suitable. E4 on the other hand disagreed with the practice saying that: “Only fully tested, workable code should be checked in”. Given this feedback it would seem that the wording of the practice may be a little simplistic given the different contexts that developers work in. While it does indeed allow all developers access to the latest source code, it is reasonable to consider if that code is actually tested and integration ready. As a result, the practice has been amended within the model to reflect this as a consideration. Lean Principle: Eliminate Waste (Avoid Over-Production). Lean Practice (ROW46): Perform code refactoring. Refactoring is the periodic review and modification of existing code in order to improve the internal structure. Not surprisingly there is some uncertainty around this which is reflective of the number of issues already identified within the model (ROW47 – ROW53). In general my review of the expert opinion leads me to realise that there are varying interpretations of what exactly refactoring is. While none of the experts disagreed with it, some of them discussed upgrade projects being equivalent to a refactoring. E1 for example is not a developer and so may not be best positioned to understand the nuances of code changes, although he clearly understood the potential knock on effect that a code change could have and its importance within the MD context. E8 argues that it really depended on the system, the design and the type of changes, and E5 suggests that if you are able to clearly isolate the code you are changing, then refactoring in later iterations may be possible.
225
In conclusion of the validation for this practice I was not entirely satisfied that I achieved adequate input to fully validate this specific practice. I have decided, however, given that no-one disagreed with it, that it should remain in the model but should be evaluated in conjunction with the other practices which aim to mitigate some of the potential issues with refactoring, namely ROW47 – ROW53. 5.2.2.3
Validation Summary
The above validation process was performed on 53 of the 97 model practices and a breakdown of the results is summarised below in Table 5-2. The table groups the SaLeS-4-MD model rows by Medi-SPICE specific practice, identifying the number of lean practices which belong to that grouping. This is followed by a summary of the outcome of the validation process for that group. The summary shows how many of the lean practices achieved the intended validation (at least 3 expert opinions having been achieved), and how many did not (the right most column). Of the practices which did achieve adequate validation, the table shows the outcome. Either the practice was accepted without modification (Accept As Is), accepted with modification (Accept Modified) or rejected (Reject).
226
Table 5-31 Summary of the Validation Process Results Validated Medi-SPICE Practices
ENG.5.5.SP1
ENG.5.5.SP2
ENG.5.5.SP3 ENG.5.5.SP4
ENG.5.5.SP5
ENG.5.5.SP6
ENG.5.5.SP7 ENG.5.5.SP8
ENG.5.5.SP9 ENG.5.5.SP10
ENG.5.5.SP11
Define and establish a unit verification strategy Define and establish software unit verification criteria Define and establish software unit acceptance criteria Develop software units Ensure consistency and bilateral traceability to system and software requirements Ensure consistency and bilateral traceability to detailed design Ensure consistency and bilateral traceability to test specification of software units Verify software units Document software unit verification results Conduct risk analysis Conduct risk evaluation and risk control activities
Number of Specific Practices
Accept As Is
Accept Modified
Reject
Needs Further Feedback
4
1
1
0
2
1
0
1
0
0
3
0
2
0
1
41
26
6
0
9
2
1
1
0
0
2
1
1
0
0
4
1
1
0
2
22
3
1
0
18
3
0
0
0
3
13
5
1
0
7
2
0
0
0
2
97
38
15
0
44
55%
227
For example, row 3 of Table 5-2 (ENG 5.5.SP1) indicates that the SaLeS-4MD model proposes 4 lean practices which support that specific Medi-SPICE practice. Two of those lean practices met the validation requirements with one practice being accepted as it is, while the other was accepted but required some modification to reflect the expert feedback. The other two lean practices did not achieve the required validation criteria and so are deemed not yet validated. An overall summary can be seen at the bottom of Table 5-2. Of the 97 rows within the SaLeS-4-MD model, 53 achieved the requisite validation, with 38 being accepted as they are, and 15 accepted but with some modification. The remaining 44 rows are considered un-validated even though many of them were reviewed by one or two experts. The subsequent final version of the SaLeS-4-MD model detail can be seen in Appendix I.
5.3
Conclusion
The SaLeS-4-MD model is a collection of best practices for software development for the Medical Device industry, and this chapter presented the validation approach taken to determine the legitimacy of the proposed model’s practices. A two-fold approach was taken to achieve this validation. Firstly by means of a process assessment of a MD company and secondly by interviewing a number of select experts in the field. The rich qualitative data gathered during this process was used to determine the suitability of the proposed practices to the regulated MD domain. Ensuring that at least 3 independent experts gave an opinion on a practice helped form a good foundation for making an informed decision about its validity. Each practice was either accepted as it was, accepted but with a modification within its description, or rejected out of the model. A majority (53) of the model practices went through the full validation process. Of these 53, none were rejected, 38 were accepted as is, and 15 were accepted with modifications. Table 5-2 presents a summary of the validation results.
228
229
Chapter 6: Safe and Lean Software for Medical Devices (SaLeS-4-MD Described)
6.1
Introduction
The model presented here draws together the principles and practices of a lean software development perspective and marries them with the requirements for safety and risk reduction of the medical device regulations. The resulting model includes a high level process roadmap for iterative software development, with a collection of specific lean practices and principles to help guide development towards a compliant, but least burdensome outcome. The model is called Safe and Lean Software for Medical Devices, or ‘SaLeS-4-MD’. During the research process, the SaLeS-4-MD model evolved through a number of versions, and the version presented in this chapter is the final post validation version which has been updated to reflect the validation outcomes.
6.2
Process Overview
The high-level structure of the SaLeS-4-MD model is presented in Figure 6-1. The format will be familiar to those readers with knowledge of the SPICE framework for software process improvement (Simon, 1996), (ISO/IEC, 2004). Working from the outside in of Figure 6-1, the model is enclosed within an organisational context which indicates that all related software development activities are governed and guided by the organisational goals. Without the strategic direction being set by the overall organisational needs, and the requisite supports being provided, the lower-level processes may not result in the right product being built or the desired organisational culture being followed. It also represents the importance of the responsibility that senior management has in terms of ensuring that the whole organisation is fully aware of the regulatory context within which they operate.
230
Within the organisational layer is the Management and Process Improvement layer, which maintains the management function of the various lower level activities such as project management, and risk management. The influence from this layer is exerted within the engineering iterations indicating the direct influence this layer has on specific lower level activities. This is depicted by the arrows pointing into the iterative cycles. For example the application of risk management which is paramount within MD SD as specified by IEC 62304 (ANSI/AAMI/IEC, 2006) section 7 and ISO 14971 (ISO, 2007), must be applied throughout the entire lifecycle of the project. Therefore the arrows from the management layer are specifically depicted entering each iteration of development, signifying that specific risk management activities have to be occurring at those points. Continuing inwards we see the inner Engineering layer, and this is where the actual software development and associated activities take place. The model is at all times cognisant of the regulatory environment in which it is designed to operate and so it includes, where necessary, a reference to the governing regulations. Starting from the left hand side of the Engineering layer there is a ‘Project Plan’ box. This high level software development plan is mandated by IEC 62304 section 5.1 “Software Development Planning”. This box intentionally overlaps into the Management & Process Improvement layer indicating that this high level plan needs input from multiple stakeholders in order to achieve its goal of addressing:
the processes to be used,
the deliverables,
traceability requirements,
configuration and change management, and
problem resolution.
231
Figure 6-52 The SaLeS-4-MD Process Model Overview
232
This project plan feeds into the first development iteration within the process. As can be seen, there are an undefined number of cycles, each one feeding into the next to signify the iterative and evolutionary nature of the development process. This will help to achieve the benefits of iterative development, such as fast delivery of working software and early feedback, as identified in the model detail against ENG.5.5.SP4. The activities within each iteration are detailed in section 6.3. Following a number of these evolutionary cycles, the product is completed and a formal software release is performed as depicted by the box on the right hand side. Again this box intentionally overlaps the higher model layers to indicate that it is delivered to parties external to the development groups and so must be evaluated and documented appropriately. This, as indicated, is mandated by IEC 62304 section 5.8 “Software Release”. The final layer, Support, is shown at the bottom of the model. Similar to the Management and Process Improvement layer, there are arrows which point directly into the evolutionary cycles, again indicating that certain activities must occur at these points. For example, because of the inconsistent and interchangeable use of the terms verification and validation, the FDA through their Centre for Devices and Radiological Health (CDRH), published a guidance document on the general principles of software validation (CDRH, 2002). In it they state that: “In practice, software validation activities may occur both during, as well as at the end of the software development life cycle to ensure that all requirements have been fulfilled”. This is reflected in the model by mandating that verification and validation, similar to risk management, are activities that need to be occurring during each iteration. However, in order to strike a balance between performing too much and not enough validation, the model proposes that specific iterations be designated to undergo more rigorous validation (as depicted by the pink circles), and the final iteration should include a full system validation step. The
233
choice of validation iterations is therefore project specific and should be decided as part of the initial project planning.
6.2.1 Research and Development An important aspect within many product development companies is the need to carry out research and development (R&D) type activities. This need is given a specific mention in the model and is depicted in Figure 6-2 by the dotted rectangle around the first few iterations. The significance of bracketing a number of initial iterations is to identify that they are not considered part of the normal product development cycle and so not subject to the full rigors of the regulations. What this allows for is a far more parsimonious approach to recording the various risk management and verification and validation activities. It is important to state that the model does not advocate not performing these activities but rather that a lighter approach can be taken. This is depicted in Figure 6-2 by the thinner lines entering these particular iterations and emanating from the Management and Support model layers. The model does in fact suggest that a high level verification report is a prudent output from any such research and development phase.
234
Figure 6-53 The SaLeS-4-MD Process Model Overview with a R&D Phase
6.3
The Iteration
The software development happens within each of the evolutionary cycles of the engineering layer, and as we have seen is influenced by the governance of the outer layers. Figure 6-3 depicts the specific activities that the model suggests need to happen within each iteration.
235
Figure 6-54 The SaLeS-4-MD Iteration
As Figure 6-3 shows, apart from the actual coding, there are a number of different activities that must happen in order to demonstrate compliance with the regulations. All of these tasks are mandated by the regulations. For example, an important artefact for the MD regulators is documentation relating to various activities. While it is often considered a time consuming exercise with questionable value, it is nonetheless specifically called for in the regulatory requirements. As suggested by (Medi-SPICE, 2011), the model calls for specific documentation tasks be performed for specific activities. Consequently, the SaLes-4-MD model suggests that this documentation be created but with a view to only generating the necessary amount. Therefore within the evolutionary cycles, these forms of documentation should happen, however, the focus should remain localised to the specific iteration and guided by the specific lean practices of the model detail (see Appendix I).
236
6.4
The SaLeS-4-MD Practices
The model currently focuses on the software construction phase as defined in (Medi-SPICE, 2011), and is composed of 11 high-level practices. Structured around these, the model presents a selection of lean software development practices and principles and where necessary, specifically tailored to satisfy the medical device regulations. Each of the 11 high-level practices is presented below in detail as follows: -
The Medical Device recommended high-level practice and a brief description
-
The lean principle(s) being applied
-
The lean practice(s) and description(s)
-
Possible issue(s) within a medical device context
-
Possible mitigation technique(s) if available
This detail can be viewed in table format in Appendix I.
6.4.1 SP1. Define and establish a unit verification strategy Description: Develop and establish a strategy for verification and re-verifying the software units. The strategy should define how to achieve the desired quality with the available techniques Lean Principle: Eliminate Waste (Avoid Over-Processing). Lean Practice 1: Consider if performing a Research and Development phase. Since the regulations do not govern research and development, the verification strategy should take that into consideration. Specific mention should be made to the R&D phase including defining when it starts and when it ends. Within this period, a much less rigorous process can be implemented which does not require the full formality of documenting activities such as risk analysis and verification and validation. Potential Issue: During an audit, the regulator may request a high–level verification report covering the R&D phase. Potential Mitigation: Generate a high-level verification report covering the R&D phase.
237
Lean Practice 2: Decide on level of concern. Depending on their intended use, different medical devices require different levels of verification. This also applies to the software. Like the medical device it is associated with (unless stand alone software), the level of concern associated with the software should guide the level and detail of the verification strategy. Lower levels of concern should be reflected within the strategy by lower levels of verification. Potential Issue: The risk with this approach is that it can be a somewhat subjective decision and therefore an inappropriate level of concern may be chosen resulting in too little verification being carried out. Potential Mitigation: Have a clear mechanism for determining the level of concern. This mechanism should be documented so that everyone is comfortable and confident when making a determination.
6.4.2 SP2. Define and establish software unit verification criteria Description: Develop and document criteria for verifying that each software unit satisfies its design, functional and non-functional requirements in the verification strategy. Lean Principle: Eliminate Waste (Avoid Over-Processing). Lean Practice: Just enough documentation. This is a generic lean approach to documentation which is applicable in many situations. The objective is to avoid writing documents that nobody ever reads, or producing documents which has more information than the intended audience needs. In terms of documenting unit verification criteria, a strategy which minimises the detail up front, avoiding trying to capture every detail in advance, and elaborating during the relevant iteration is suggested. Potential Issue: Although different stakeholders can be interested in the same information, some may require more detailed information than others. There is a danger therefore in trying to use a single document to satisfy both types of stakeholders.
238
Potential Mitigation: By careful consideration it may be possible to strike a balance between including too much and too little while satisfying both types of stakeholders.
6.4.3 SP3. Define and establish software unit acceptance criteria Description: Define and establish acceptance criteria for software units prior to integration into larger software items as appropriate, and ensure that software units meet acceptance criteria. Lean Principle: Eliminate Waste (Avoid Over-Processing). Lean Practice: Re-use existing artefacts. This approach promotes the practice of re-using user stories as the requirements for acceptance criteria. Potential Issue: The regulator may deem user stories as lacking the required level of detail. Potential Mitigation 1: By including a clear ‘condition of satisfaction’ segment within each user story, the necessary level of detail can be achieved. Potential Mitigation 2: Each iteration gathers extra detail for its assigned user stories. This extra detail would include more detailed acceptance criteria if required.
6.4.4 SP4. Develop software units Description: Develop and document the executable representations of each software unit. Lean Principle: Eliminate Waste (Avoid Over-Production). Lean Practice: Only develop for current iteration. The code written in any iteration should only be for software units selected for that iteration. This avoids code being written which hasn’t been requested or prioritised and therefore potentially wasted effort. Potential Issue 1: Regulator will be focusing on reviewing evidence of verification and validation activities.
239
Potential Mitigation: Ensure verification and validation activities are a core element of each iteration and not left to the last iteration. Potential Issue 2: Problems can arise with iterative software development when the team must collaborate with non-iterative teams. Potential Mitigation: By defining common skeleton interfaces which can be broken into discrete pieces can assist both teams to continue their respective approaches and still collaborate. Lean Principle: Deliver fast. Lean Practice: Use short iterations. The practice of iterations helps speed up the feedback loop by releasing software to the customer quickly. By making the iterations as short as possible makes this process quicker. Potential Issue: The typically recommended iteration lengths of 2-5 weeks may be too short for medical device software projects. Potential Mitigation: Use iteration lengths appropriate to the project and environment. Lean Principle: Create Knowledge. Lean Practice: Develop prototypes. Developing prototypes delivers an unfinished or preliminary version of the product to the customer for review. By setting the prototype up in a way in which the customer can ‘play with’ the system allows them to review and refine their actual requirements. Potential Issue: None identified. Potential Mitigation: Not applicable. Lean Principle: Continuous flow. Lean Practice: Perform load levelling (Heijunka). Heijunka, a Japanese term meaning production smoothing, promotes a constant and predictable processing rate. In this context, this load levelling refers to the
240
developers who need to analyse, write, test and document the code. Development resources who are overburdened actually take longer to perform tasks. The aim is to ensure this does not happen by good scheduling and project management. Potential Issue: None identified. Potential Mitigation: Not applicable. Lean Principle: Build quality in. Lean Practice: Re-use existing artefacts. Here again the principle of re-use is promoted through the re-use of code. This can be in many forms but particularly discrete components or modules or even full off the shelf applications, will assist in building upon tried and tested software, thereby saving an amount of rework. In addition, the fact that the software is already being used in a production setting, adds a level of confidence to the code. Potential Issue: There can be a tendency to assume that the re-used code does not need to be tested or only gets lightly tested. Potential Mitigation: Any reused code should be explicitly included and marked for testing within the validation plan. Lean Principle: Build quality in. Lean Practice: Take an object oriented approach. Practices such as componentisation, modularisation, and parameterisation can be used to hide away complexity and limit the scope of future changes. In addition, this approach supports a distributed software development configuration by making the exchange of code a simpler and more reliable process Potential Issue: None identified. Potential Mitigation: Not applicable.
241
Lean Principle: Build quality in. Lean Practice: Maintain code clarity. Although this is more of a principle than a practice, there are a number of specific practices which support maintaining clarity of code. Code clarity is especially useful when working in environments where different people work on the same code, where the code will be undergoing peer reviews or audits, and for cases where the code will need to be maintained in the future. Code clarity reduces the effort needed to read and interpret source code and enhances the appearance of the code. The following are some of the practices which support this clarity: o Use of an intuitive naming convention for variables, types, functions etc. o Use of a common language o Use of simple notation o Use of clear and focused in-code comments Potential Issue: None identified. Potential Mitigation: Not applicable. Lean Principle: Build quality in. Lean Practice: Use a good source code control tool. Having good control around the source code maintains the integrity of the code by ensuring it is secure but also accessible to all developers from a central location. Potential Issue: None identified. Potential Mitigation: Not applicable. Lean Principle: Create knowledge. Lean Practice: Daily check out and in of source code. Having a good source control tool should be augmented with a process of daily checking in and out of source code by all developers. This then ensures that the
242
master version is always up to date and consequently any developer pulling out the source code can be confident they have the latest version. Potential Issue: None identified. Potential Mitigation: Not applicable. Lean Principle: Create knowledge. Lean Practice: Perform in-code documentation. By putting low level information in the code itself, as opposed to an external document, ensures that the information is available where it is needed and to the appropriate people. Potential Issue: None identified. Potential Mitigation: Not applicable. Lean Principle: Eliminate Waste (Avoid Over-Processing). Lean Practice: Documentation Grouping. This technique suggests organising the code itself in such a way as to resemble the related documents such as the software requirements specification and the software design specification. This then makes the review process much easier since the connection between them can more easily and intuitively be seen. An extension of this practice is that for less complex systems the relevant documents can be grouped together into a single document in an effort to make documentation and review process even more efficient. Potential Issue: None identified. Potential Mitigation: Not applicable. Lean Principle: Eliminate Waste (Eliminate defects). Lean Practice 1: Practice Test Driven Development (TDD) A TDD approach will help find defects earlier in the development process and consequently prevent a more expensive bug fixing exercise later in the project.
243
Potential Issue 1: When developing code in conjunction with hardware, the hardware may not be available for the frequent test runs as advocated by a TDD approach. Potential Mitigation 1: One technique to overcome this it is to make use mocks or stubs within the code which generate a static hardware response. In the case where hardware is not a required input, the hardware call can be commented out of the code. Potential Mitigation 2: Another technique is to make use of a hardware simulator within the software. This simulator can be substituted for the hardware and provide hardware-like responses back to the software and allow the testing process to continue. Potential Mitigation 3: Similarly, a hardware abstraction layer, again implemented in the software, can be used to shield the software from the actual hardware implementation. This allows for greater flexibility in swopping in and out different hardware components. Lean Practice 2: Develop a test as soon as a defect is found. In support of a TDD approach this is a mechanism to ensure that defects which are identified are fixed, but importantly that they are not allowed to re-occur and therefore reducing future defect fixing effort. Potential Issue: Within the medical device regulations, special attention is given to recording defects which have been found and importantly what was done to fix them. This must include the specific software requirement the defect relates to. Potential Mitigation: Ensure a formal mechanism is put in place to capture and record these activities. Lean Principle: Eliminate Waste (Avoid Over-Production). Lean Practice: Perform code refactoring. Refactoring is the periodic review and modification of existing code in order to improve the internal structure.
244
Potential Issue 1: Refactoring code late in the project life-cycle may cause concern for the regulator. Potential Mitigation 1: Refactoring should be limited to early iterations of the software. Potential Mitigation 2: Constant testing should help to identify any defects being introduced. Potential Mitigation 3: The practice of pair programming can help identify possible issues which might result from a code change. Potential Issue 2: Potentially hazardous to hardware. This can result, for example, by code changes which change an execution path making it longer or shorter and introducing a change in timings which can be detrimental to a hardware component. Potential Mitigation 1: By making use of a good configuration management tool which incorporates the associated hardware version defects can more easily be traced back and recovered from. Potential Mitigation 2: Perform a system level impact analysis of any changes. This will include the hardware components and so catch possible issues which have been introduced. Potential Mitigation 3: Relentless testing can help identify new issues introduced. Potential Issue 3: Changes made during refactoring may break existing code. Potential Mitigation 1: Use source code branching to separate any changes from the existing working version, and only integrate the changes when all tests have passed. Potential Issue 4: Refactoring legacy code can be problematic due to the lack of an existing build and test scripts.
245
Potential Mitigation 1: Legacy code should not be refactored. However as modifications are requested to legacy code, build and test scripts should be created and used going forward. Lean Principle: Eliminate Waste (Eliminate Defects). Lean Practice: Employ Behaviour Driven Development (BDD). The basic tenets of behaviour driven development is to extend the TDD mentality to obtain a clear understanding of the desired software by discussions with all stakeholders, and the use of natural language to write test cases which therefore minimises the need for translation between the technical and business levels. Potential Issue: BDD tool support may not yet be mature enough. Potential Mitigation: Revert back to using a TDD approach Lean Principle: Eliminate Waste (Eliminate Defects). Lean Practice: Continuous Software Integration. By continually integrating the software units into the larger system, integration issues are caught sooner. In addition, it avoids the ‘big bang’ approach to system integration. Potential Issue 1: Continuous integration will generally require the use of scripts. These scripts can be seen as important pieces of code used in the development of the software and may therefore require validation. Potential Mitigation: The integration scripts should be maintained within a source control system similar to the rest of the code. Potential Issue 2: The integration process may require new hardware on which to build the software. Potential Mitigation: Make the appropriate investment if required.
246
Lean Principle: Eliminate Waste (Eliminate Defects). Lean Practice: Frequently integrate the software with the hardware (where applicable). Except where the software is stand alone, by frequently integrating the software with the hardware, integration issues are caught sooner, which provides valuable feedback early to the hardware team. Potential Issue 1: The regulator will still insist on final full system integration. Potential Mitigation: Perform a formal final full system integration which should be much faster and less troublesome. Potential Issue 2: Legacy code may be problematic due to lack of integration and test scripts. Potential Mitigation: None identified. Lean Principle: Create Knowledge. Lean Practice: Use a visual display. By openly displaying all work in process and their status, keeps all stakeholders informed and helps identify bottlenecks which may be developing. Various types of display can be used such as burn-down charts, cumulative flow diagrams, or Kanban boards. Potential Issue: None identified. Potential Mitigation: Not applicable. Lean Principle: Create Knowledge. Lean Practice: Interchange developer roles. Assigning developers to different roles increases the team’s knowledge of the complete system and broadens the expertise base thus reducing the possible development of bottleneck resources. Potential Issue: It is important to assign experienced developers to critical areas of the system.
247
Potential Mitigation: Assign experienced developers to critical areas of the system.
6.4.5 SP5. Ensure consistency and bilateral traceability to system and software requirements Description: Ensure consistency of system and software requirements including verification criteria to the software units. Consistency is supported by establishing and maintaining bilateral traceability between the system and software requirements including verification criteria and software units verification criteria Lean Principle: Build Quality In & Eliminate Waste (Avoid Over-Processing). Lean Practice: Use similar documentation structures. Documents that resemble each other are easier to review and relate and therefore make the traceability analysis more intuitive for the regulator. Potential Issue: None identified. Potential Mitigation: Not applicable. Lean Principle: Eliminate Waste (Avoid Over-Processing). Lean Practice: Perform documentation grouping. This technique suggests that when mapping between different documents to show traceability, to map sections of documents instead of every little detail. This is also easier to do if the documents resemble each other as above. Potential Issue: This could be perceived by the regulator as cutting corners. Potential Mitigation: Don’t make the sections that are mapped too big.
6.4.6 SP6. Ensure consistency and bilateral traceability to detailed design Description: Ensure consistency of detailed design including verification criteria to the software units. Consistency is supported by establishing and maintaining bilateral traceability between the detailed design including verification criteria and software units verification criteria
248
Lean Principle: Build Quality In & Eliminate Waste (Avoid Over-Processing) Lean Practice: Use similar documentation structures. Documents that resemble each other are easier to review and relate and therefore make the traceability analysis more intuitive for the regulator. Potential Issue: None identified. Potential Mitigation: Not applicable. Lean Principle: Eliminate Waste (Avoid Over-Processing). Lean Practice: Perform documentation grouping. This technique suggests that when mapping between different documents to show traceability, to map sections of documents instead of every little detail. This is also easier to do if the documents resemble each other as above. Potential Issue: This could be perceived by the regulator as cutting corners. Potential Mitigation: Don’t make the sections that are mapped too big.
6.4.7 SP7. Ensure consistency and bilateral traceability to test specification of software units Description: Ensure consistency of software units including verification criteria to the software units test specification including test cases. Consistency is supported by establishing and maintaining bilateral traceability between the software units including verification criteria and the software units test specification including test cases Lean Principle: Build Quality In & Eliminate Waste (Avoid Over-Processing) Lean Practice: Use similar documentation structures. Documents that resemble each other are easier to review and relate and therefore make the traceability analysis more intuitive for the regulator. Potential Issue: None identified. Potential Mitigation: Not applicable. Lean Principle: Eliminate Waste (Avoid Over-Processing). Lean Practice 1: Perform documentation grouping.
249
This technique suggests that when mapping between different documents to show traceability, to map sections of documents instead of every little detail. This is also easier to do if the documents resemble each other as above. Potential Issue: This could be perceived by the regulator as cutting corners. Potential Mitigation: Don’t make the sections that are mapped too big. Lean Practice 2: Build on top of a TDD approach. Test driven development advocates a continuous focus on testing through the entire software development by means of automated tests. By adding references to these test scripts, which identify the requirements they are testing, a partial requirements traceability matrix (RTM) can be obtained as a direct by-product of the TDD process. Potential Issue: Refactoring may create new links or break existing ones. Potential Mitigation: Failing test should trigger an update of the RTM.
6.4.8 SP8. Verify software units Description: Verify software units against the detailed design according to the defined verification strategy. Lean Principle: Eliminate Waste (Avoid Over-Processing). Lean Practice 1: Consider if performing a Research and Development phase. Since the regulations do not govern research and development, the verification strategy should take that into consideration. Within the R&D phase, a much less rigorous verification process can be performed. Potential Issue: During an audit, the regulator may request a high–level verification report covering the R&D phase. Potential Mitigation: Generate a high-level verification report covering the R&D phase. Lean Practice 2: Decide on level of concern. Depending on their intended use, different medical devices require different levels of verification. This also applies to the software. Like the medical device it
250
is associated with (unless stand alone software), the level of concern associated with the software should guide the level and detail of the verification strategy. Lower levels of concern should be reflected within the strategy by lower levels of verification. Potential Issue: The risk with this approach is that it can be a somewhat subjective decision and therefore an inappropriate level of concern may be chosen resulting in too little verification being carried. Potential Mitigation: Have a clear mechanism for determining the level of concern. This mechanism should be documented so that everyone is comfortable and confident when making a determination. Lean Principle: Eliminate Waste (Eliminate Defects). Lean Practice: Employ Poka-Yoke techniques. The Japanese term Poka-Yoke refers to the practice of defect prevention and defect detection. These are generally tools or mechanisms for automatically preventing a defect from occurring or identifying defects as soon as they appear. Potential Issue: None identified. Potential Mitigation: Not applicable. Lean Principle: Build Quality In Lean Practice 1: Use standards. Make use of coding standards and guidelines, improving code quality and making it easier to review and maintain. Potential Issue: There is the possibility that despite standards being laid down and guidelines being issued, that developers simply do not adhere to them. Potential Mitigation: When performing code reviews, a check for adherence to standards and guidelines should be included.
251
Lean Practice 2: Perform root cause analysis. Whenever issues are encountered with the software, a thorough root cause analysis should be performed to ensure the problem is identified, understood and an appropriate remediation can be designed. The ‘5 Whys’ technique is an example of an easy to use root cause analysis technique. Potential Issue: Root cause analysis must be rigorous. Although the ‘5 Whys’ method is useful it may be seen not to be rigorous enough. Potential Mitigation: Employ best practice risk analysis techniques such as those listed in SP10 and SP11. Lean Practice 3: Re-use existing artefacts. Here again the principle of re-use is promoted through the re-use of code. By building upon tried and tested software a level of confidence is introduced to the code. It may also be possible to re-use test cases and build scripts already developed. Potential Issue: There can be a tendency to assume that the re-used code does not need to be tested or only gets lightly tested. Potential Mitigation: Any reused code should be explicitly included and marked for testing within the validation plan which would then be addressed during verification also. Lean Practice 4: Use experienced developers on critical areas. Critical areas of the code will require more concentrated verification effort. By demonstrating that experienced developers have been used, a level of confidence can be instilled in the code. Potential Issue: If too many critical areas are identified, there is the danger that these experienced developers can become bottlenecks. Potential Mitigation: Build up developers’ experience.
252
Lean Practice 5: Perform automated testing. By continually running automated tests and being able to show repeated positive test results, can be seen as addressing the GPSV’s guidelines on “consistently fulfilling requirements”. Potential Issue: For medical devices which include a hardware component, the hardware may not always be available. Potential Mitigation 1: One technique to overcome this it is to make use of mocks or stubs within the code which generate a static hardware response. In the case where hardware is not a required input, the hardware call can be commented out of the code. Potential Mitigation 2: Another technique is to make use of a hardware simulator within the software. This simulator can be substituted for the hardware and provide hardware-like responses back to the software and allow the testing process to continue. Potential Mitigation 3: Similarly, a hardware abstraction layer, again implemented in the software, can be used to shield the software from the actual hardware implementation. This allows for greater flexibility in swopping in and out different hardware components. Lean Practice 6: Use a testing framework tool. Specially designed testing frameworks exist for individual languages such as XUnit, JUnit and VBUnit. These frameworks will make the automated process easier to manage. Potential Issue: User interface testing can be difficult to automate. Potential Mitigation: By separating the frontend user interface from the backend business logic, the tools can continue to be used on the backend portion. Lean Practice 7: Maintain clarity. Although this is more of a principle than a practice, there are a number of specific practices which support maintaining clarity of code. Code clarity is
253
especially useful when working in environments where different people work on the same code, where the code will be undergoing peer reviews or audits, and for cases where the code will need to be maintained in the future. Code clarity reduces the effort needed to read and interpret source code and enhances the appearance of the code, thereby adding to the verification of the code. The following are some of the practices which support this clarity: o Use of an intuitive naming convention for variables, types, functions etc. o Use of a common language o Use of simple notation o Use of clear and focused in-code comments Potential Issue: None identified. Potential Mitigation: Not applicable. Lean Practice 8: Avoid over detailed maintenance documentation. When creating supporting documentation for future maintenance, keep detail to a minimum. Potential Issue: Medical devices can have long life-cycles so the software will most likely be maintained by different developers than the ones who created it. The maintenance documentation should contain enough information to allow for an adequate understanding of the software. Potential Mitigation: Have non-team members evaluate the adequacy of the documentation. Lean Principle: Build Quality In & Create Knowledge. Lean Practice 1: Perform pair programming. By having pairs of developers working alongside each other, a form of continuous code review occurs. This practice also supports the sharing of knowledge between developers.
254
Potential Issue 1: Can be difficult to pick appropriate pairs. A balance is needed between developers’ knowledge/skill levels and being able to peer review each others’ work. Potential Mitigation: Not identified. Potential Issue 2: Familiarity between developers can lead to lax practices developing. Potential Mitigation: With good management oversight this can be made to work well by anticipating weaknesses. Lean Practice 2: Perform integrated problem solving. By using cross functional teams to analysis and resolve problems, a more robust verification process is established. This also acts as a mechanism to spread knowledge about the software. Potential Issue: None identified. Potential Mitigation: Not applicable.
6.4.9 SP9. Document software unit verification results Description: The results of software unit verification are documented and recorded. Lean Principle: Create Knowledge. Lean Practice: Avoid over-documenting. By standardising the way data is presented makes it easy for that information to be communicated with different stakeholders. Potential Issue: None identified. Potential Mitigation: Not applicable. Lean Principle: Eliminate Waste (Avoid Over-Processing). Lean Practice: Use pre-defined templates. Making use of standard pre-designed templates for capturing verification results, makes the process faster and easier to manage. Potential Issue: None identified. Potential Mitigation: Not applicable.
255
6.4.10
SP10. Conduct risk analysis
Description: The planned risk analysis activities are performed. The implementation and the results of the risk analysis shall be recorded in the risk management file. Lean Principle: Eliminate Waste (Avoid Over-Production). Lean Practice 1: Re-use existing artefacts. Minimise rework by re-using the risk analysis carried out for similar devices as a starting point. Potential Issue: There is a danger that the devices differ in some fashion and therefore the old risk analysis does not fully cover all the potential risks of the new device. Potential Mitigation: Perform a systematic evaluation of any differences between the two devices. Lean Practice 2: Decide on the level of concern. The level of risk analysis performed should be commensurate with the level of concern of the software. Lower LOCs require lower levels of risk analysis. Potential Issue: The risk with this approach is that it can be a somewhat subjective decision and therefore an inappropriate level of concern may be chosen resulting in too little risk analysis being carried out. Potential Mitigation: Have a clear mechanism for determining the level of concern. This mechanism should be documented so that everyone is comfortable and confident when making a determination. Lean Principle: Build Quality In. Lean Practice 1: Use established risk analysis techniques. There are many existing risk analysis techniques that can be used to assist the process. Potential Issue: There is a danger that the chosen techniques will not be considered rigorous enough. Potential Mitigation: Use industry best practices such as:
256
-
Failure Mode and Effects Analysis (FMEA)
-
System-FMEA
-
Failure Mode, Effects and Criticality Analysis (FMECA)
-
Fault Tree Analysis (FTA)
Lean Practice 2: Perform risk review. It is necessary to review, and revise if necessary, the risk and hazard analysis that has been performed earlier in the development process. This can help identify any new unanticipated risks and also bring clarity to earlier risk assumptions. Potential Issue: Any mitigations introduced to resolve potential hazards must be fully traceable. Potential Mitigation: ‘Negative’ user stories can be introduced into the requirements
and
the
necessary
references
added
to
the
relevant
documentation. Lean Practice 3: Use cross-functional teams. By bringing together stakeholders from different functional areas and performing risk analysis as a group from different points of view, a more comprehensive view of risk can be achieved. Potential Issue: None identified. Potential Mitigation: Not applicable. Lean Practice 4: Use pre-defined templates. Making use of standard pre-designed templates for capturing risk analysis, makes the process faster and easier to manage. Potential Issue: None identified. Potential Mitigation: Not applicable.
257
Lean Practice 5: Use a document management tool. There are electronic document management tools available which ease the management of documents. Such tools can be used to secure documents by controlling access rights, but also can automate workflow for approvals and supports remote collaboration between stakeholders. Potential Issue: None identified. Potential Mitigation: Not applicable.
6.4.11
SP11. Conduct risk evaluation and risk control activities
Description: The risk evaluation and risk control activities are conducted and the results shall be recorded in the risk management file. Lean Principle: Build Quality In. Lean Practice: Use cross functional teams. The international standard (ISO 14971 pg 40) tells us it is well established that the perception of risk often differs from empirically determined risk. Therefore by bringing together stakeholders from different functional areas a more comprehensive evaluation of risk can be achieved. Potential Issue: In line with the regulatory requirements, any risk controls which are put in place to mitigate potential risk must be traceable by recording them in the risk management file. Potential Mitigation: Ensure the relevant documents are updated including the risk management file. The people who attended the risk evaluation meetings should also be recorded.
6.5
Conclusion
This chapter provided a top down description of the SaLeS-4-MD model of software development within a medical device context, a primary output of this research. A high level overview shows how an iterative and evolutionary approach to software development is linked with organisational, managerial and supportive influences. Within the individual development cycles, a number of specific activities need to occur in order to satisfy the MD regulations. The
258
model’s specific lean practices support these activities but always from a lean perspective, aiming at reducing or eliminating wasteful processes and making important processes more efficient and therefore cost effective.
259
Chapter 7: The Conceptual Framework Described 7.1
Introduction
Conceptual frameworks are a mechanism for explaining the primary objects of a research project (Miles and Huberman, 1994). These objects include key factors, constructs or variables and the presumed relationship between them. Without such an organising framework, it is difficult to generalise, communicate and apply research findings (Glass and Vessey, 1995). However, although very useful in this regard such conceptual frameworks are only ever approximations of the theory which they aim to describe (Weick, 1995) and so it is useful to accompany them with a descriptive narrative. The Conceptual Framework presented here, and depicted in Figure 7-1, evolved from the initial Conceptual Framework as described in Chapter 4. The rest of this chapter describes the final framework structure and each of its components. In order to highlight the important connections, which can be viewed as specifically generated from this research, relationships with the Lean Thinking component are coloured red and relationships with the Regulatory Requirements component are coloured blue.
260
Figure 7-55 A Conceptual Framework for Situated Lean Software Development
261
7.2
The Framework Structure
The structure used for the framework is an adaptation of the Methods-In-Action model (Fitzgerald et al., 2002). Research has shown that software developers rarely follow any formal methodology fully, but in reality apply and interpret a methodology differently. This interpretation and attitude is influenced by a number of overt and covert components resulting in a varying collection of development practices being employed. Within the context of lean software development for medical devices, this research has shown that the important influencing components are: -
Regulatory Requirements
-
Lean Thinking
-
Organisation
-
Project Characteristics, and
-
Human Engagement.
Each of these components is introduced in the following sections and their relationships with each other and with the central focus of the framework is described.
7.3
Situated Lean Software Development Practices
The central cloud represents the collection of specific software development practices which get implemented within the particular development context. Since these are what are used to design, develop and deliver the software they are given the central focus within the framework. As with the Methods-InAction framework (section 2.7.1), this central component is meant to reflect the activities which contribute to the full software life-cycle, and therefore includes the broader range of activities and stakeholders involved. The cloud shape used to depict this component reflects the somewhat undefined nature of its contents. As in the SaLeS-4-MD process model, a collection of software practices exist which may or may not be used by the developers depending on a number of factors. These factors are depicted within
262
the framework as the outer components which encircle the central cloud, indicating
their
varying
influence
on
the
actual
software
process
implementation.
7.4
Regulatory Requirements
Given the increase in medical device regulations, this component reflects the necessity of compliance when performing the software development activities. The framework highlights the relationships between Regulatory Requirements and the other framework components by means of the blue lines. The growing amount of regulation, for example the expanding definition of a medical device, is causing an increasing number of organisations to ensure they have compliant development processes. This is reflected by the ‘Influences’ line connecting it with the Organisation component. The regulatory documents mandate certain things of an organisation and it is therefore necessary for these organisations to ensure they are satisfied. Medical device regulations insist that all staff be fully informed about what the regulations are, why they exist, and what part they must play in ensuring they are satisfied. This is reflected in the framework by the ‘Influences Behaviour of’ line linking it with the Human Engagement component. The behaviour of employees is influenced in the sense that they are required to maintain a constant vigilance for anything that might introduce risk into the MD, something developers within non-safety-critical domains do not need to maintain. In addition, almost all development activities are required to have traceability. This requires developers to ensure that the necessary steps are taken as part of their process to record what they have done and any consequences derived from those actions. The third relationship for this component is with the central cloud. The relationship is bidirectional and is described as ‘Shapes/Satisfies’. The ‘Shapes’ describes the influence the regulations have on the practices chosen within the SD process. Practices which do not (or are perceived not to) support compliance will not be employed. For example, an iterative approach to development may
263
be utilised, however, due to the increased overhead for regulatory traceability and verification, the iteration length will be longer than typically advocated by the agile community. In some cases a more plan-based methodology may be adopted for fear that an agile approach will not satisfy the regulations. The ‘Satisfies’ description refers to the returning connecting line, likewise indicating that the specific practices will be chosen so that they support fulfilling the requirements of the regulations. Within the SaLeS-4-MD model this is seen by the selection of proposed practices where the model identifies potential issues with certain practices and how they might be tailored (‘Shaped’) to satisfy the regulations. For example utilising user stories as requirements is a typical agile development practice, however, from a regulatory perspective they may not be seen to contain adequate detail. Therefore the model suggests that each user story should have clear conditions of satisfaction as part of its description.
7.5
Lean Thinking
The relationship between Lean Thinking and the other components is highlighted by the red lines in the framework. Given the aim of the framework to describe a lean SD context, this component has been made prominent within the framework. The first relationship link is with the Organisation component and is described as ‘Influences’. From an organisational perspective, unlike the regulations, lean does not mandate anything but rather suggests a mindset that should be adopted in all aspects of the business processes. In this sense the influence is seen by the way in which the organisation guides and supports the developers. For example, do they centralise decision making or do they allow for self-managing and autonomous development groups. To what extent do they promote process improvement? Is it a sporadic activity or something which they try and weave into peoples’ daily lives? How is company knowledge managed? Do they facilitate knowledge transfer and promote knowledge creation through software prototyping, good software coding practices,
264
developer role interchanging and visual display boards? Are employees treated as valued resources by fully engaging them in process improvement and allowing them a sense of responsibility and ownership? The next relationship is with the Human Engagement component, described as ‘Influences Behaviour of’. The seven principles of the Lean Thinking component aim to instil a lean mindset in how people carry out their daily activities. Developers, when performing their duties are encouraged to reflect on their process and identify any steps which could be eliminated or improved. For example recurring issues should have a root cause analysis carried out and if possible a mechanism developed to ensure it cannot happen again. If the integration phase is a troublesome activity, perhaps employing a continuous integration strategy would detect and eliminate issues faster and earlier in the process. Documentation that does not add value to the process is perhaps unnecessary or could be reduced. In a similar fashion, focus is directed towards creating customer value and soliciting customer feedback in order to achieve that end. A means of achieving this is by close collaboration between developer and customer, something that may or may not be feasible given the influence of the Organisation component. The third relationship is with the Project Characteristics component. This relationship is represented by the dotted line merging with a relationship line from Organisation and proceeding to the Project Characteristics component. This relationship emerged from the research and is important since it highlights again a relationship independent of the regulatory context, and in particular that it may affect (hence the dotted line) the project characteristics themselves. The type of relationship is described as ‘Adds To’ by which is meant two things. Firstly, a project may have a particular requirement which is borne out of a lean mindset, and secondly, a specific project may in total be the result of a lean initiative. As identified in the first of the lean principles, waste identification and elimination should be an on-going process. The final relationship is again with the central cloud. The relationship is described as ‘Shapes’ to signify that as particular software practices are chosen,
265
the application of a lean mindset may influence the way in which they are tailored and implemented. For example the SaLeS-4-MD model identifies the need to ‘Define and establish a unit verification strategy’. Typically this would involve drawing up a detailed process stipulating the necessary steps which, within this domain, will include the requisite documentation and traceability requirements. However, a lean perspective would suggest that by identifying the R&D phase and separating it from the rest of the development life-cycle, the resulting strategy will call for a less rigorous process to be implemented during that phase.
7.6
Organisation
(ANSI/AAMI/IEC, 2006) tells us that “Compliance with this standard is defined as implementing all of the PROCESSES, ACTIVITIES, AND TASKS identified in this standard”. It is therefore down at this level of detail that companies are busy maintaining compliance. From an organisational perspective this is achieved through the provision of guiding and supporting practices which enable the development teams accomplish them. This component is related to the Human Engagement component through a relationship described as ‘Influences Behaviour of’. Employee motivation, as identified in the initial Conceptual Framework, had proven to be too simple a term to capture the important research findings. Terms such as Governance Approach, Delegation of Authority and Resourcing Strategies were added to the component to signify the importance that the organisation plays in the development of its employees and consequently positive motivation. An organisation which actively promotes a safe working environment, both in terms of the products they are producing, and psychologically in terms of employees feeling comfortable with identifying issues, will instil a strong work ethos and a mindset of continuous risk awareness. An organisation which has a strong process improvement focus will build a supportive attitude in the workforce towards continuous improvement. Additionally, an organisation which values the development group and sees it as a core competency will
266
stimulate employee pride in their work and generate a sense of job security and job satisfaction. Equally important however, is that the lack of such organisational qualities will adversely affect employee morale, motivation, engagement and ultimately quality of work. The second relationship is shown with the link to the Project Characteristics component. Described as ‘Determines’, this relationship signifies how the organisational context directly affects the specifics of the various development projects. Naturally the particular industry the organisation is operating in, and the specific products it is developing will define the technical requirements of the projects. Internal and external market forces will influence the time line for projects, something which often causes stress within the organisation. The market segment the company is targeting may influence the importance or priority of projects within the portfolio. Finally, the pace of innovation and external competitive forces may influence the durability of project requirements, in other words the changeability of user requirements. The final relationship for the Organisation component is again with the central cloud. This is given two descriptions ‘Influences’ and ‘Shapes’. ‘Influences’ represents how the organisational ethos towards, for example, risk management, regulatory compliance, process improvement, or employee governance, reflects in the choice of practices eventually employed within projects. These influences can be overt in the form of written procedures, or may indeed be covert, for example, where a blind eye is shown to practices which help speed a process along but which by-pass some of the formal process steps. The ‘Shapes’ description is used to indicate that the organisational supports, which may or may not be provided, directly affect the choice of practices. Practices which require specific software and even dedicated hardware, for example to support continuous integration, cannot be performed if such an environment is not provided.
267
7.7
Project Characteristics
Figure 7-1 shows this component with some exemplars of what it encapsulates. It is shown as shaping the situated development practices and it does so, for example, by requiring specific technical abilities of the stakeholders. An obvious technical need will be that the team members are adequately skilled in implementing the technical requirements in the hardware and software, but less obvious is that they can communicate in a way that is easily understandable. The technical need may also manifest itself by dictating the type of environment required and consequently some specific practices may not be feasible or necessary. Specifically in relation to embedded software, approaches such as TDD are more problematic when hardware components are involved (Van Schooenderwoert and Morsicato, 2004), (Kane et al., 2006), (Cordeiro et al., 2007), (Masticola and Gall, 2008), (Hibbs et al., 2009). Other project characteristics included in this component are ones which are determined by the organisational context, as indicated by the relationship line between the two framework components. Market conditions and organisational goals will determine some aspects of a project typically defined within the process itself. For example, within the MD industry, getting a new product through development as quickly as possible in order to move into the clinical trials and then commercialisation phases is becoming ever more important. Once a launch date has been set, there is very little appetite for delays. This can have the effect of bypassing the natural flow of the project backlog by mandating a priority and implicitly subsuming development resources. This has a consequence for the choice of software development practices in that there is little tolerance for changes to process that might introduce risk to compliance and therefore affect time to market. This concentration on time to market can also create organisational inertia to change within a new product introduction process. This may be typical within a MD environment generally, with (Weyrauch, 2006) stating: “In such a world, any significant change to culture or process can be difficult. Sometimes it is difficult because of the inertia inherent in a large organization”.
268
7.8
Human Engagement
At the heart of any MD software development process are the people. While this framework focuses on the specific SD practices that are employed, it maintains awareness of the importance of the human element through the inclusion of the Human Engagement component. Apart from the relationships already described, this component has two other relationships, one with the Project Characteristics component and the other with the central cloud of practices. Various people are involved in analysing the requirements and specifics of the MD projects. These will be people with varying skills and experience. Some will be better at judging the technical difficulty or estimating the time it will take to complete a task. Others will be more aware of the significance of one project over another and therefore the importance of it to the organisation. For projects with both software and hardware components, there will be a need for teams to work together to perform the analysis, some people are better than others at this. It is also quite common for a resource bottleneck to develop, someone who has built up specialist knowledge in a particular area and is sometimes the only person able to do a particular job. What all these convey is the well established fact of the complexity of working with knowledge workers such as SD personnel, and how their differing skills, attitudes, availability, knowledge, work ethos can affect project analysis, the development practices employed and the degree of excellence with which they perform these tasks.
7.9
Conclusion
The second important output from this research, the Conceptual Framework, is presented in detail in this chapter. The framework evolved from the initial Conceptual Framework as presented in Chapter 4 (Figure 4-1) which was derived from the literature review. Phase 2 of the research developed the framework’s core components which are described here together with their inter-relationships.
269
From a MD regulatory perspective, the framework shows the key factors which play a role when the SD process is put in place. From a lean perspective it includes an additional component which is seen as independent of the regulations, but when performed within this context, can result in a different approach to the selection of SD practices. Each of the other components has some form of inter-relationship highlighted within the framework in order to signify the importance those relationships play in the process.
270
271
Chapter 8: Summary and Conclusions 8.1
Introduction
This final chapter summarises the main components of the thesis. It begins with a review of the study objectives and the overall approach taken to the research. This is followed by a narrative which answers each of the specific research questions presented in section 2.9.1. The primary outcomes of the research are described which include the conceptual framework and the SaLeS4-MD process model for software development under Medical Device regulation. The chapter is concluded with a discussion of the research limitations, suggested future direction for related research, and some final remarks from the author.
8.2
Research Objective
This study began in 2009 when the consequences of the global financial crisis of 2007 had begun to take hold. This was quite a remarkable global event given that the banking industry would have been considered to be a heavily regulated one. Whether successful or not, such regulations are a modern reality within many industries today. In a world which is continuing to become more and more technology dependant, concern exists around the dependability of computer systems and particularly so for those systems which are considered life-critical. One such area is the medical device industry where computer software is playing an increasingly critical role. Consequently, recent years have witnessed an increase in the amount and type of regulation governing medical device software development, so much so that a piece of software can now be classified as a medical device in its own right. The medical device industry has historically been somewhat immune to cost pressures, however, the global financial crisis has led to a call for lower cost devices with increased functionality. This additional functionality is increasingly being implemented in software, and so the industry is interested in
272
looking at ways of making that process more efficient and cost effective. Typically, medical device companies employ traditional software development processes which they believe offer a more predictable outcome in terms of satisfying the regulatory requirements. In many other industries, including other safety-critical ones such as aviation and automotive, more modern approaches to software development, such as lean or agile, have been adopted and which have been shown to reduce the cost and time taken to develop the software. However, the medical device industry has been slow in adopting them with fear being expressed over the perceived lack of process control and backup documentation, necessary to satisfy the regulations. What this thesis explores is the applicability of a lean software development methodology within the medical device industry, and proposes a software development process model which allows such a lean approach to be adopted.
8.3
Summary of Research Approach
Taking a design science approach this study iteratively develops a conceptual framework and practical process model of software development for MD software. The methods employed included a focused literature review (a mapping study), a case study, model construction, literature enfolding, and expert validation. An overview of the research process can be seen in Chapter 3 (Figure 3-2) which consisted of four high level phases: Phase 0 – “Defining the Problem Area”, Phase 1 – “The Literature Review”, Phase 2 – “The Iterative Model Development”, and Phase 3 – “Validation”.
8.4
The Research Questions Answered
The macro research proposal for this thesis was: “To investigate the applicability of a lean software development methodology within the medical device software industry”. This was then decomposed into four sub-questions which have been answered as follows:
273
RQ1: What are the primary characteristics of software development within the medical device industry? In order to investigate a phenomenon it is important that we have a good understanding of the fundamentals of the context in which it occurs.
An
objective of this question, therefore, was to explore the realities of MD SD, for example, the types of devices developed, the resource requirements (technical and human), the governing regulatory landscape, and the SD methodologies being adopted. Through a combination of the literature review and industry engagement, this question was answered. The MD industry covers a very broad range of companies and products. These products range from non-technical devices, for example a syringe, to high-tech devices such as an implantable pacemaker. The development lifecycles of these products contain both technical and process elements which involve a collection of technologies and personnel which much be controlled and coordinated. While these in themselves are difficult to achieve, the regulatory aspect of the industry adds another dimension which manifests itself in changes to work practices including additional tasks which, in other industries, may not have to be performed. Although the definition of medical devices is quite broad, software has been playing an ever increasing and important role in their development, not only by being embedded within many devices but equally, by the role it plays in supporting the manufacturing processes. From a software development perspective, the embedding of software in hardware brings with it many issues which standalone software development does not encounter, for example, the need to integrate and test the software with the hardware components. When different teams are used for the hardware and the software, this introduces scheduling problems and interpersonal issues which need to be managed carefully. Given the intended use of a MD, the industry maintains a high level of risk awareness in terms of the potential risk to human safety. Consequently the SD processes are designed to ensure that software requirements are clear, code is
274
adequately tested, potential issues are evaluated, potential risks are addressed and all these are documented in a traceable manner. This risk management mindset and the ability to demonstrate traceability right throughout the SD lifecycle is evaluated through regulatory audits by the relevant national authorities. Failure to comply with the regulations can have a devastating effect on a company’s business. As the industry has become more cost focused, an increase in the use of external partners has been observed. While this has been common practice within other industries for many years, it is an increasing trend within the MD domain also. This outsourcing can be offshore or onshore, co-located or distributed and may include the software development itself, and/or the final validation process. All of these bring with them a number of issues which have already been well documented within the GSD literature (Carmel, 1999), (Beecham et al., 2011). An important consideration is the SD methodology that MD companies adopt. This research found that MD companies unnecessarily employ heavyweight methodologies in order to “be sure” that regulatory compliance is achieved. Fig. 8-1 is an adaptation of the software planning spectrum (Boehm, 2002) with the positioning of typical MD software development processes superimposed.
Figure 8-56 The Planning Spectrum (Adapted from Boehm, 2002)
275
The question relevant to this research is how we can move this positioning to the left of the spectrum. One of the issues with the subject matter of this thesis was that it brought together two topics which had not been done before, namely lean software development and medical device software. Therefore, while this first research question may be seen as somewhat generic in terms of a research project, it was specifically designed to bring some clarity to both lean software development itself and the related area of agile software development within the MD domain. Figure 8-2, generated from the literature review aids in that understanding.
Figure 8-57 Foundations of Lean Software Development
Specifically, there was no mention of lean SD within the MD domain, however, this can be explained by the relatively low level of awareness and lack of clarity around what lean SD constitutes and how it relates to agile SD. This research presents agile SD practices as being supportive of a lean SD methodology, and bearing this positioning in mind, shows how it has begun to be applied in the MD domain. What is clear, however, is that many such agile practices have been tailored to accommodate the need for the enhanced verification, validation and traceability required in this domain. Figure 8-3, generated from the literature review, presents a conceptual view of the domain and how these primary components relate. It therefore gives us a broader understanding of use of lean within MD SD.
276
Figure 8-58 The Conceptual Framework for Lean and Regulated SD
The MD industry continues to be very cautious of any changes to process which might negatively affect regulatory compliance. Consequently it has been slow to change and takes a sceptical view of such “light weight” methodologies such as lean or agile. RQ2: How are the medical device regulations affecting the software development process within medical device companies? From a MD regulatory perspective, the focus is to ensure the safety of the devices to reduce the risk of injury. National and international regulations have been created to help protect the consumers and users of these devices. This is achieved through stipulating the organisational processes and systems that have to be in place. Consequently these regulations influence more than just the software development itself but affect multiple levels and process areas with the organisation. Within the SD area, they are augmented by a recommended list of activities and tasks that software developers should be performing on a continuous basis, for example risk analysis, verification and validation. While many of these practices are not mandatory, subsequent regulatory audits will be guided by these recommendations and are therefore viewed by industry as mandatory in practice.
277
The increasingly critical role software is playing within the industry has introduced new sources of safety risk. Consequently the definition of a MD continues to be expanded, with some software now being defined as a MD in itself. This has caused additional companies, who heretofore were unregulated, to now fall within the bounds of MD regulation. This poses the problem for these companies of changing their SD processes in order to satisfy the regulations, something which is proving to be problematic due to the lack of direction. The MD regulations, and in particular the subsequent audits, expect a detailed organisational software development process which developers follow. They call for objective evidence that the necessary activities of this process are being carried out, and that they fulfil the regulatory requirements. This has led to companies designing processes which do more than necessary in order to satisfy the regulations, while not necessarily adding to the safety of the devices. This added cost and in particular the amount of documented evidence that is generated and maintained, has increased and is frustrating the people involved. In an industry which is becoming much more cost focused, this added cost is a very stressful consequence, and something which has been shown to have ramifications throughout MD organisations. Figure 8-4 shows an updated Conceptual Framework generated following the empirical research which helps us gain a far better interpretation of these complexities of software development within the MD domain. Chapter 7 provides a full walk through of the various components and their relationships.
278
Figure 8-59 A Conceptual Framework for Situated Lean Software Development
RQ3: How can lean software development be applied to the software development process within medical device companies without compromising regulatory compliance? This research has found that the SDLC within MD companies can in fact be based on a lean SD methodology which has at its core an iterative approach and which is guided by lean principles. The seven lean principles advocated by this thesis are: 1. Identify and Eliminate Waste 2. Build Quality In 3. Create Knowledge 4. Defer Commitment 5. Deliver Fast 6. Respect People 7. Maintain Flow
279
The regulations specifically do not favour any particular methodology over another. An iterative approach, which seeks to deliver value to the customer quickly and receive feedback early in the life-cycle and as fast as possible, assists in verifying the software on an ongoing basis and therefore supportive of those objectives of regulations. It is important, however, to ensure that the choice of methodology, the rational for that choice and how it supports the regulations are clearly documented in advance. Lean is a philosophy which advocates minimising non-value-add tasks, reuse to avoid duplication, and utilising the full potential of the workforce. While it is important to define what value is in this domain, for example, compliance with regulations should be included, these lean concepts support the regulations through concentrating on important product characteristics such as safety, re-using proven components, and tapping into the workforce who are best placed to identify process improvement possibilities. Eliminating waste is difficult to argue against, however, waste is an emotive subject, and similar to value, must be given careful consideration in this domain. Since many of the existing agile practices support this concept they can easily be adopted within MD SD. For example, object oriented programming, componentisation, and modularisation all assist in encapsulating code which can isolate potential software failures and thereby make future code changes less critical to the overall code base. Specifically in relation to the waste of software defects, the lean concept of Poka-Yoke (mistake proofing) and the agile approach of Test Driven Development achieve the fast identification and elimination of defects in the code. Again these support the ongoing verification of the source code and therefore regulatory requirements. Figure 8-5, generated from the research, presents an approach to MD SD which is both lean and regulatory compliant in nature.
280
Figure 8-60 The SaLeS-4-MD Process Model Overview
What this research has shown is that lean guiding principles can complement the objectives of the MD regulations while allowing companies pursue cost saving SD initiatives. Specific practices, be they lean, agile or simply industry best practices, can support these principles but many require some tailoring in order to suit the specifics of the regulations, the context of individual companies, and the risk averse nature of the MD industry.
8.5
Contributions
This study answered the research questions above and in so doing generated results which I believe are beneficial to both industry and academia. The main thesis contributions are: 1. an addition to the body of knowledge through the published mapping study 2. the Conceptual Framework 3. the SaLeS-4-MD process model. At a more detailed level the contributions can be viewed from both a theoretical and practical perspective.
281
8.5.1 Theoretical Contributions:
The literature review in Chapter 2 will be useful to those who are new to the subject area, introducing them to the concepts of risk management and device classification, which form the cornerstone of the MD regulations. Academics may benefit from the description of MD regulation presented which is brought up to date to include the latest revisions of the IEC 62304 international standard. The pedigree of the SaLeS-4-MD process model is also demonstrated through the description of the Medi-SPICE framework.
The concept of lean software development continues to be a somewhat undefined phenomenon. In particular its relationship with the more well known agile methodologies is something which has proven to be rather nebulous. Chapter 2 provides clarity around the underlying principles of lean SD and takes a clear position on how lean and agile are related. This is useful for practitioners who can use this information to gain a clearer understanding of a subject which has been gaining momentum as being the next stage of agile evolution. It is useful for academics since it draws together reference material which crosses over between the worlds of manufacturing and software engineering, and builds a coherent view of lean software development, thus addressing a gap identified within the literature.
The effects of regulation on software development, is another area which has received little attention in main stream academic research. Since regulation is becoming more relevant in many industries, this research benefits many different types of readers since it has reported on the consequences of regulations on the SD process from multiple sources. The effects of following a lean thinking mindset within a regulated context are combined in the Conceptual Framework presented in
282
Chapter 7, something which academics should find useful if pursuing this line of enquiry further.
The review of safety-critical software development generated a state of the art report in terms of the adoption of particular SD practices within those industries. This may be useful to practitioners in these fields looking to investigate the opportunities in adopting such SD approaches. From an academic perspective, this identifies a gap in the literature highlighting the need for further research into the suitability of agile SDMs within such domains, but particularly the applicability of a lean SD approach.
The design of the research process itself and some of its constituent activities is something which may be of interest to other researchers. The design science approach to iterative model development melds well with a hermeneutical approach to research, in that knowledge is incrementally gathered from multiple sources and synthesised to produce a more holistic understanding.
8.5.2 Practical Contributions: The Medical Device industry should be the primary beneficiaries from this research since the major practical output, the SaLeS-4-MD process model, is focused specifically on that domain. It does not dictate any specific SD practices but advocates principles to follow and offers a selection of practices to choose from, all of which are understood to be context dependent. I therefore consider it very accessible to a general audience and indeed possibly transferable to other safety-critical domains. From an academic perspective, the SaLeS-4-MD model structure is adapted from existing peer-reviewed work and backward references all the specific practices to the originating sources. This therefore facilitates a more in-depth exploration of the model and possible future model extension and refinement.
283
8.5.2.1
Utilising the SaLeS-4-MD process model
There are two types of end-user who can benefit from the process model: (1) lean or agile SD companies who are new to the MD regulations, and (2) existing MD companies who are interested in adopting a leaner, cost effective SD process. Category (1) companies may be looking to enter the MD SD market, or, by a change in the definition of a MD, might be companies who find themselves newly subject to the regulations. In both cases such companies have existing lean or agile SD processes but are unsure how their existing practices fit within the expectations of the regulations. What the SaLeS-4-MD model can offer them is a tool to examine their SD practices in light of how they may or may not satisfy the regulations. In many cases they will be able to see how they would need to alter their existing practices in order to be compliant. For category (2) companies, the usefulness of the process model will be in suggesting how they might move away from a heavyweight waterfall SDLC to a less plan-based and more iterative approach. They will be able to use the tool to review the suggested lean SD practices appropriate to the different life cycle stages, and understand how they can support the relevant regulations. This allows companies to then consider how they might adopt or adapt the practices within their own context. In both category (1) and (2) companies, they will be able to use the SaLeS-4MD process model to check the sources of the suggested practices, thereby allowing them to delve further into the suitability of these practices for their own situation. 8.5.2.2
SaLeS-4-MD and Medi-SPICE
Medi-SPICE is a software process improvement framework initiated by the SPICE user community specifically for the MD industry. Existing companies who are looking to ensure that their SD processes are in compliance with the relevant regulations and best practices can perform assessments on their processes to determine their level of conformance. They can use the Medi-SPICE
284
reference model to see the activities that are recommended and adjust their own processes as needed. SaLeS-4-MD is designed to complement the Medi-SPICE framework. It has been structured around the Medi-SPICE reference model and so has a natural fit with the recommended specific practices of Medi-SPICE. Companies examining their SD processes through the Medi-SPICE framework, can therefore easily use the SaLeS-4-MD model to identify potentially suitable lean SD practices which support the Medi-SPICE model. Since both Medi-SPICE and SaLeS-4-MD are both built around existing regulations, companies can be confident in the pedigree of the recommendations.
8.6
Limitations of the Research
The research presented in this thesis is concerned with the development of software within the medical device industry. However, the primary empirical data gathering was through a single case study company, and therefore the question may arise as to the generalisability of the findings. Unfortunately it was not feasible to extend the research to multiple MD companies due to the lack of availability of other MD companies and industry experts. While this is acknowledged as a weakness, the following is offered in support. The findings which have been generated were constructed from empirical evidence within a MD company and from a rich literature search of relevant MD publications. That case study can be viewed as typical of the industry in that all companies in this domain are subject to the same regulations, and it is the regulatory aspect which has been central to the investigation. Reflecting on my own work experience helped my understanding of the consequences of regulation on the software development process. Since the objective of all regulation is to control human behaviour, this adds to the broader relevance of the findings. Lastly, the final SaLeS-4-MD model was the result of scrutiny by 11 independent MD industry experts from different companies, and so adds weight to the applicability of the model in a more general sense. This thesis does not claim to
285
propose a panacea for applying lean within this domain, however, the contributing research elements combine to build a stronger case for the model’s wider applicability. Additionally, in many instances the model offers a selection of specific lean practices which may widen the field of use to a larger number of companies. The SaLeS-4-MD model structure uses the eleven specific practices from the Medi-SPICE model. As clearly stated, the Med-SPICE project was still ongoing at the time of model development and so had not been finalised. This poses the question as to the accuracy and completeness of the model’s objectives. This was indeed a concern when deciding to adopt the eleven specific practices. However, a discussion with the Medi-SPICE principal investigator and a senior researcher concluded that the version of Medi-SPICE at the time was relatively stable and unlikely to change in any significant way. This was supported by the following: -
much of the Medi-SPICE model drew on closely related models already established for the automotive industry,
-
the Medi-SPICE work to-date had already been peer-reviewed at international conferences
-
the
extended
international
Medi-SPICE
project
panel
were
periodically reviewing the project progress and feedback had been positive to-date. Due to the large number of process areas and specific practices, the model was narrowed down to the construction phase of the SDLC, and subsequently 53 of the 97 suggested practices were validated through the expert panel. This raises the question as to the validity of the rest of the model. This weakness is clearly highlighted in the thesis, however, the validation process used here is similar to that used by (McLoughlin, 2010). The validation phase delivers a high level of confidence in those 53 practices which represents over 50% of the model. Thus the thesis does not claim validity of the entire model but rather
286
presents evidence of a robust research approach and a validation process which supported a majority of the model, and therefore an increased probability of validity of the rest of the model. This therefore is identified as a primary candidate for future research. The maxim “easier said than done” might be an accusation levelled against many process improvement models, and with respect to the effort and cost involved in implementing some of the practices proposed by this model, that limitation is recognised. Indeed some expert feedback has highlighted this as a concern. Therefore acknowledgement is given to the fact that the model does not provide any indication of the cost involved in implementing the various practices. For certain companies, it may potentially be cost prohibitive to employ some of the model’s practices, or that the cost of implementation would negate the possible gains in efficiency.
8.7
Research Quality Review
Recalling the credibility and quality considerations presented in section 3.6, I briefly review the quality attributes from Table 3-8 and how they have been addressed within the research. Objectivity/Confirmability This research is the result of empirically grounded data with particular attention given to ensure an unbiased account. This was achieved by having key processes, plans and data independently reviewed by my academic supervisors and colleagues, for example the mapping study and case study protocols and findings. Having taken a design science approach, there is a certain amount of researcher bias which has been acknowledged. This bias was intentional and enhanced the research, however, care was taken to use this knowledge in a controlled and open fashion. In terms of confirmability, this dissertation presents a detailed view of a majority of the research data both in terms of how the data was generated and also in full final form within the appendices. Only confidential participant data has been withheld. In addition some of the data has been peer reviewed and
287
published, clearly identifying myself as the author and providing a means for contact about the research. Reliability/Dependability/Auditability In section 3.7 I outline a number of concerns prevalent within qualitative data analysis. The aim of that section was to instil in the reader that reasonable care was planned with every step of the research. This was indeed achieved through the following techniques: -
performing a mapping study to ensure all relevant material was systematically captured
-
careful scheduling of the ... interviews and subsequent transcription and coding
-
triangulating, where possible, new model practices with previous data sources
-
an interim mini survey to sense-check the emerging data
-
providing validation experts with a written description of the SaLeS4-MD model prior to interview to aid in clarification
-
ensuring an adequate number and calibre of domain experts
-
periodically publishing peer-reviewed papers associated with this work.
All research artefacts have been preserved in a project folder and have been archived for review and further use. Internal Validity/Credibility/Authenticity Do the findings make sense? To answer this I performed an independent general review of the process model with an industry partner, academic colleagues and my supervisors. This led to the need to adjust the model and therefore added the clarity that was needed. In addition the final validation phase was performed on a panel of MD experts who reviewed the model and which consequently resulted in modifications to the model.
288
External Validity/Transferability/Fittingness This questions as to whether the conclusions are transferable to other contexts. The conceptual framework presented in Chapter 7 describes the landscape for lean software development within MD regulated environments. There are no specifics to a particular company or industry segment and in that sense could be considered applicable in the wider MD context. The SaLeS-4-MD model was presented to 11 independent MD experts from different countries, and they agreed that 53 of the 97 practices were valid practices within the MD domain in general. The ... case study participants, my own professional background, and the expert panel details have been clearly presented for those who may wish to use them in subsequent cross comparisons. These points strongly suggest that the model is transferable within the MD industry. Utilisation/Application/Action Orientation I have published some of the findings within this dissertation, and this document will be publicly available from the University of Limerick. In addition I plan on publishing the Conceptual Framework and validated SaLeS-4-MD model in appropriate journals. This will ensure that all relevant material is publicly available. Unfortunately it was beyond the scope of this research to trial the model to ascertain how it works in practice.
8.8
Potential for Future Research
Two focusing steps were performed to facilitate a validation phase to this study. The model focused on the software construction and unit testing SDLC phases, and the validation process concentrated on 53 of the 97 specific practices. Although a majority of the practices have been validated, clearly a next step would be to put the remaining 44 practices through a similar validation process.
289
In addition, the remaining SDLC phases need to be catered for within the model. Once the Medi-SPICE reference model is complete, it could be used to extend the SaLeS-4-MD model to the entire SDLC and allow for a mapping of lean practices to the additional process areas. While MD software development clearly has an important technical aspect to it, the non-technical aspects such as management and inter team collaboration could benefit from some focused exploration. In particular: -
There is a need for a mechanism to measure the process improvement of this lean model. Lean manufacturing is focused on measurement and statistical control, but is that something which can or should be applied within software development?
-
What are the cost implications of implementing such an approach? Do we as academics simply propose solutions and process improvement mechanisms without considering the expense it imposes?
-
Interesting feedback from the expert panel was that since it is more common nowadays to see MD companies outsourcing different pieces of the SDLC, what would an appropriate model look like in such a case? How might the SaLeS-4-MD model have to change?
-
In addition how does the concept of Global Software Development in general complicate things further here?
Finally, since the SaLeS-4-MD process model has been structured around the Medi-SPICE reference model, it may be possible for it to become an extension of the Medi-SPICE framework. Industry is looking for tools which can help them achieve in practice what theory is suggesting, and therefore the process model could be a useful tool to be offered with Medi-SPICE. Medi-SPICE is expected to be an internationally recognised process assessment framework for the MD industry, so it might be possible to have the SaLeS-4MD process model as an accompanying artefact available to the wider MD community.
290
8.9
Concluding Remarks
The development of software is a collaborative exercise which starts with a customer requirement and results in the delivery of a satisfactory product. People have to communicate, interact and share information and knowledge for that to happen. These are imperfect processes and have led to the increased use of research methods, typical of the social sciences, to investigate them. This social dimension is one of the reasons for arguing that there is no one-size-fitsall or “Silver Bullet” when it comes to software engineering (Brooks, 1987), but rather each situation or context may benefit from a combination of different development techniques. Consequently the software development model developed in this thesis suggests a collection of lean software development practices which are not a panacea for all MD SD but rather should be viewed as a collection of tools which may or may not be adopted by the target organisation, but dependent on their suitability within their organisational context.
291
References AAMI 2011. Guidance on the use of AGILE practices in the development of medical device software. Agile TIR SW Committee Draft 1.0. AAMI. ABRAHAMSSON, P., BABAR, M. A. & KRUCHTEN, P. 2010. Agility and Architecture: Can They Coexist? Software, IEEE, 27, 16-22. ABRAHAMSSON, P., SALO, O. & RONKAINEN, J. 2002. Agile software development methods—review and analysis. ABRAHAMSSON, P., WARSTA, J., SIPONEN, M. T. & RONKAINEN, J. 2003. New directions on agile methods: a comparative analysis. Proceedings of the 25th International Conference on Software Engineering. Portland, Oregon: IEEE Computer Society. AGILEALLIANCE. 2001. Manifesto for Agile Software Development [Online]. Available: www.agilemanifesto.org/ [Accessed 9/7/2012. ALLEMAN, G. B., HENDERSON, M. & SEGGELKE, R. Making agile development work in a government contracting environment-measuring velocity with earned value. Agile Development Conference, 2003. ADC 2003. Proceedings of the, 2003. 114-119. AMBLER, S. 2008a. Agile Adoption Survey 2008. AMBLER, S. 2008b. When IT Gets Cultural: Data Management and Agile Development. IT Professional, 10, 11-14. AMBLER, S. W. 2006. Imperfectly agile: You too can be agile! Dr. Dobb's Journal [Online], 31. Available: http://www.ddj.com/architect/192700252?pgno=1. AMBLER, S. W. Agile software development at scale. 2nd IFIP TC 2 Central and East European Conference on Software Engineering Techniques, CEE-SET 2007, October 10, 2007 - October 12, 2007, 2008c Poznan, Poland. Springer Verlag, 1-12. AMBLER, S. W. & KROLL, P. 2007. Best practices for lean development governance [Online]. IBM. Available: http://www.ibm.com/developerworks/rational/library/jun07/kroll/ [Accessed 22nd January 2010]. ANDERSON, D. J. Stretching agile to fit CMMI level 3 - the story of creating MSF for CMMI & process improvement at Microsoft corporation. Agile Conference, 2005. Proceedings, 2005. 193-201. ANDERSON, D. J. 2010. Kanban, Blue Hole Press. ANSI/AAMI 2001. SW68 Medical device software - Software life cycle processes. AAMI. ANSI/AAMI/IEC 2006. 62304:2006 Medical Device Software-Software life cycle processes. Association for the Advancement of Medical Instrumentation. AVELING, B. 2004. XP Lite Considered Harmful? Extreme Programming and Agile Processes in Software Engineering. BALLARD, G. 2000. Positive vs negative iteration in design. The 8th Conference of the International Group for Lean Construction. Brighton, U.K. BARNETT, L. 2009. An Agile Approach to Project Management - Leveraging Agile Practices to Improve PMO Effectiveness. [Accessed 26/11/2011]. BASILI, V. & TURNER, A. 1975. Iterative enhancement: A practical technique for software development. IEEE Transactions on Software Engineering, 4, 390-396. BASILI, V. R. The role of experimentation in software engineering: past, current, and future. Software Engineering, 1996., Proceedings of the 18th International Conference on, 25-29 Mar 1996 1996. 442-449. BASILI, V. R. & ROMBACH, H. D. Tailoring the software process to project goals and environments. The 9th international conference on Software Engineering (ICSE), 1987 Monterey, California, United States. 345-357.
292
BBC.
2002. Enron Scandal [Online]. Available: http://news.bbc.co.uk/2/hi/business/1780075.stm. BECK, K. 1999. Extreme Programming Explained: Embrace Change Addison-Wesley Professional. BECK, K. 2002. Test Driven Development: By Example Addison-Wesley Professional. BECK, K. & FOWLER, M. 2000. Planning Extreme Programming Addison-Wesley Professional BEECHAM, S., BADDOO, N., HALL, T., ROBINSON, H. & SHARP, H. 2008. Motivation in Software Engineering: A systematic literature review. Information and Software Technology, 50, 860-878. BEECHAM, S., HALL, T., BRITTON, C., COTTEE, M. & RAINER, A. 2005. Using an expert panel to validate a requirements process improvement model. Journal of Systems and Software, 76, 251-275. BEECHAM, S., NOLL, J., RICHARDSON, I. & DHUNGANA, D. A Decision Support System for Global Software Development. Global Software Engineering Workshop (ICGSEW), 2011 Sixth IEEE International Conference on, 15-18 Aug. 2011 2011. 48-53. BEGEL, A. & NAGAPPAN, N. Usage and Perceptions of Agile Software Development in an Industrial Context: An Exploratory Study. Empirical Software Engineering and Measurement, 2007. ESEM 2007. First International Symposium on, 2007. 255-264. BEN-MENACHEM, M. 2008. Towards management of software as assets: A literature review with additional sources. Information and Software Technology, 50, 241-258. BERTELSEN, O. W. 1997. Toward A Unified Field Of SE Research And Practice. Software, IEEE, 14, 87-88. BLACK, S., BOCA, P. B., BOWEN, J. P., GORMAN, J. & HINCHEY, M. 2009. Formal Versus Agile: Survival of the Fittest. IEEE, 42, 37-45. BOEHM, B. 1981. Software Engineering Economics Prentice Hall. BOEHM, B. 1996. Anchoring the software process. Software, IEEE, 13, 73-82. BOEHM, B. 2002. Get ready for agile methods, with care. Computer, 35, 64-69. BOEHM, B. A View of 20th and 21st Century Software Engineering. 28th International Conference on Software Engineering, 2006 Shanghai, China. BOEHM, B. & TURNER, R. 2003a. Balancing Agility and Discipline: A Guide for the Perplexed, Addison-Wesley Longman Publishing Co., Inc. BOEHM, B. & TURNER, R. 2003b. Using risk to balance agile and plan-driven methods. IEEE Computer Society, 36, 57-66. BOEHM, B. & TURNER, R. 2005. Management challenges to implementing agile processes in traditional development organizations. Software, IEEE, 22, 30-39. BOEHM, B. W. 1976. Software Engineering. Computers, IEEE Transactions on, C-25, 12261241. BOEHM, B. W. 1988. A spiral model of software development and enhancement. Computer, 21, 61-72. BOLAND, D. & FITZGERALD, B. 2004. Transitioning from a co-located to a globallydistributed software development team: a case study at Analog Devices Inc. IEE Seminar Digests, 2004, 4-7. BORRÁS, C. 2006. Overexposure of Radiation Theraphy in Panama: problem recognition and follow-up measures. Pan American Journal of Public Health, 20, 173-187. BOWER, J. L. & CHRISTENSEN, C. M. 1995. Disruptive Technologies: Catching the Wave. Harvard Business Review, 73, 43-53.
293
BRERETON, P., KITCHENHAM, B., BUDGEN, D. & LI, Z. 2008. Using a Protocol Template for Case Study Planning. 12th International Conference on Evaluation and Assessment in Software Engineering (EASE) BRERETON, P., KITCHENHAM, B. A., BUDGEN, D., TURNER, M. & KHALIL, M. 2007. Lessons from applying the systematic literature review process within the software engineering domain. Journal of Systems and Software, 80, 571-583. BROOKS, F. P., JR. 1987. No Silver Bullet Essence and Accidents of Software Engineering. Computer, 20, 10-19. BURN, J. M., COUGER, D., MA, L. & TOMPKINS, H. 1991. Motivating IT professionals-the Hong Kong challenge. System Sciences, 1991. Proceedings of the Twenty-Fourth Annual Hawaii International Conference on, 8-11 Jan 1991 1991. 524-529 vol.4. BUSH, W. R. 2007. Software, regulation, and domain specificity. Information and Software Technology, 49, 44-54. CAMPBELL, M. K. 2004. Regulations. IEEE Potentials, 23, 14-15. CARMEL, E. 1999. Global software teams: collaborating across borders and time zones, Prentice Hall PTR. CASEY, V. & RICHARDSON, I. Project Management within Virtual Software Teams. Global Software Engineering, 2006. ICGSE '06. International Conference on, 2006. 33-42. CAWLEY, O. & RICHARDSON, I. 2010. Lessons in Global Software Development – Local to Global Transition within a Regulated Environment. European Systems & Software Process Improvement and Innovation. Grenoble, France: Springer. CAWLEY, O., WANG, X. & RICHARDSON, I. 2010. Lean/Agile Software Development Methodologies in Regulated Environments – State of the Art. In: ABRAHAMSSON, P. & OZA, N., eds. Proceedings of Lean Enterprise Software and Systems, 19 October 2010 Helsinki, Finland. Springer Berlin Heidelberg, 31-36. CDRH 2002. General Principles of Software Validation; Final Guidance for Industry and FDA Staff. FDA. CDRH 2005. Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices. Guidance for Industry and FDA Staff. FDA. CHARETTE, R. 2007. Challenging the Fundamental Notions of Software Development. CHEN, L., MUHAMMAD, A. B. & CAWLEY, C. 2009. A Status Report on the Evaluation of Variability Management Approaches. Evaluation and Assessment in Software Engineering-EASE. CHEN, W. & HIRSCHHEIM, R. 2004. A paradigmatic and methodological examination of information systems research from 1991 to 2001. Information Systems Journal, 14, 197-235. CHISHOLM, R. A. 2007. Agile Software Development Methods and DO-178B Certification. MASTER OF APPLIED COMPUTER SCIENCE, Royal Military College of Canada. CHRISTOPHER, M. & TOWILL, D. R. 2000. Supply chain migration from lean and functional to agile and customised. Supply Chain Management, 5, 206-213. CIRI. 2007. Quality Function Deployment. Available: www.ciri.org.nz/downloads/Quality%20Function%20Deployment.pdf [Accessed 27/10/2011]. CMMI. 2010. Capability Maturity Model Integration V1.3 [Online]. Available: http://www.sei.cmu.edu/cmmi/ [Accessed 10-Aug-2011. COAD, P., LEFEBVRE, E. & DE LUCA, J. 1999. Java Modeling In Color With UML. COCKBURN, A. 2000. Selecting a project's methodology. Software, IEEE, 17, 64-71. COCKBURN, A. 2002. Agile Software Development, Addison-Wesley.
294
COHEN, D., LINDVALL, M. & COSTA, P. 2003. Agile Software Development - A DACS State-ofthe-Art Report. Data and Analysis Center for Software 775 Daedalian Dr. Rome, New York. COLEMAN, G. & O'CONNOR, R. 2007. Using grounded theory to understand software process improvement: A study of Irish software product companies. Information and Software Technology, 49, 654-667. CONBOY, K. 2006. A Framework of Method Agility in Information Systems Development. Doctorate, Limerick. CONNELLY, L. M. & YODER, L. H. 2000. Improving Qualitative Proposals: Common Problem Areas. Clinical Nurse Specialist, 14, 69-74. CONWAY, M. 1968. How Do Committees Invent. Datamation Magazine. COOPER, H. M., HEDGES, L. V. & VALENTINE, J. C. 2009. The handbook of research synthesis and meta-analysis, Russell Sage Foundation. CORDEIRO, L., BARRETO, R., BARCELOS, R., OLIVEIRA, M., LUCENA, V. & MACIEL, P. 2007. TXM: an agile HW/SW development methodology for building medical devices. ACM SIGSOFT Softw. Eng. Notes, 32, 4. CRESWELL, J. W. & MILLER, D. L. 2000. Determining Validity in Qualitative Inquiry. Theory into Practice, 39, 124-130. CUMBO, D., KLINE, D. E. & BUMGARDNER, M. S. 2006. Benchmarking performance measurement and lean manufacturing in the rough mill. Forest Products, 56, 25-30. DAGNINO, A. An evolutionary lifecycle model with Agile practices for software development at ABB. Engineering of Complex Computer Systems, 2002. Proceedings. Eighth IEEE International Conference on, 2-4 Dec. 2002 2002. 215223. DAMIAN, D. E. & ZOWGHI, D. 2002. The impact of stakeholders' geographical distribution on managing requirements in a multi-site organization. Requirements Engineering, 2002. Proceedings. IEEE Joint International Conference on, 2002. 319-328. DE CRESCENZO, L. 1990. The History of Greek Philospphy - Volume II Socrates and Beyond, London, Picador. DEMARCO, T. & BOEHM, B. 2002. The agile methods fray. Computer, 35, 90-92. DROBKA, J., NOFTZ, D. & REKHA, R. 2004. Piloting XP on four mission-critical projects. Software, IEEE, 21, 70-75. DURANTON, M., BLACK-SCHAFFER, D., DE BOSSCHERE, K. & MAEBE, J. 2013. The Hipeac vision for advanced computing in Horizon 2020. DYBA, T. & DINGSOYR, T. 2009. What Do We Know about Agile Software Development? Software, IEEE, 26, 6-9. DYBÅ, T. & DINGSØYR, T. 2008. Empirical studies of agile software development: A systematic review. Information and Software Technology, 50, 833-859. DYBÅ, T., KITCHENHAM, B. & JORGENSEN, M. 2005. Evidence-Based Software Engineering for Practitioners. IEEE Software, 22, 58-65. EASTERBROOK, S., SINGER, J., STOREY, M.-A. & DAMIAN, D. 2002. Selecting Empirical Methods for Software Engineering Research. Springer. EASTERBROOK, S., SINGER, J., STOREY, M.-A. & DAMIAN, D. 2008. Selecting Empirical Methods for Software Engineering Research. In: SHULL, F., SINGER, J. & SJØBERG, D. I. K. (eds.) Guide to Advanced Empirical Software Engineering. Springer. EDMONDSON, A. C. & MCMANUS, S. E. 2007. Methodological fit in organizational field research. Academy of Management Review, 32, 1155-1179.
295
EGGERMONT, L. 2002. Embedded Systems Roadmap 2002-Vision on technology for the future of PROGRESS. Dutch PROGRESS (PROGram for Research in Embedded Systems and Software). EISENHARDT, K. M. 1989. Building Theories from Case Study Research. The Academy of Management Review, 14, 532-550. EISENHARDT, K. M. & GRAEBNER, M. E. 2007. THEORY BUILDING FROM CASES: OPPORTUNITIES AND CHALLENGES. Academy of Management Journal, 50, 25-32. EMAM, K. E., QUINTIN, S. & MADHAVJI, N. H. 1996. User participation in the requirements engineering process: An empirical study. Requirements Engineering, 1, 4-26. ESA 1998. On the Construction of New-Generation On-Board Real-Time Systems. Integrated Computer-Aided Engineering 5, 227-244. EU 1990. Council Directive 90/385/EEC Relating to Active implantable medical devices Official Journal of the European Union. EU 1993. Council Directive 93/42/EEC Concerning Medical Devices. In: COUNCIL, E. (ed.). Official Journal of the European Union. EU 2007. Medical Device Directive 2007/47/EC Of The European Parlimant And Of The Council. Official Journal of the European Union. EU 2010. MEDICAL DEVICES: Guidance document - Classification of medical devices. GUIDELINES RELATING TO THE APPLICATION OF THE COUNCIL DIRECTIVE 93/42/EEC ON MEDICAL DEVICES. EUROPEAN COMMISSION. EVANS, G. 2011. Management and light-touch regulation blamed for RBS crisis. The Independent. FAGAN, M. E. 1976. Design and code inspections to reduce errors in program development. IBM Syst. J., 15, 182-211. FAIRLEY, R. E. & WILLSHIRE, M. J. 2005. Iterative rework: the good, the bad, and the ugly. Computer, 38, 34-41. FDA 1996. 21 CFR Parts 808, 812, and 820 Medical Devices; Current Good Manufacturing Practice (CGMP); Final Rule. FDA 2011. 21 CFR Part 860. Medical Device Classification Procedures. FDA April 2009. Code of Federal Regulations 21 CFR Part 820 - Quality System Regulation. In: ADMINISTRATION, U. F. A. D. (ed.). FELDMANN, R. L., SHULL, F., DENGER, C., HOST, M. & LINDHOLM, C. A Survey of Software Engineering Techniques in Medical Device Development. High Confidence Medical Devices, Software, and Systems and Medical Device Plug-and-Play Interoperability, 2007. HCMDSS-MDPnP. Joint Workshop on, 2007. 46-54. FITZGERALD, B. 1995. A Descriptive Framework for Investigating Problems in the Application of Systems Development Methodologies. Third Conference on Information Systems Methodologies. FITZGERALD, B., RUSSO, N. & STOLTERMAN, E. 2002. Information Systems Development: Methods-in-Action McGraw-Hill Education. FITZGERALD, B., RUSSO, N. L. & O'KANE, T. 2003. SOFTWARE DEVELOPMENT METHOD TAILORING AT MOTOROLA. Communications of the ACM, 46, 64-70. FORFÁS 2008. Future Skills Needs of the Irish Medical Devices Sector. FORSBERG, K. & MOOZ, H. 1991. The Relationship of System Engineering to the Project Cycle. National Council for Systems Engineering (NCOSE). Chattanooga, Tennessee, USA. FOWLER, M. 2000. Refactoring-Improving The Design Of Existing Code, Addison-Wesley Professional.
296
FOWLER, M. 2008. AgileVersusLean [Online]. Available: http://martinfowler.com/bliki/AgileVersusLean.html [Accessed 11-Aug-2011. FRANCA, A. C. C. & DA SILVA, F. Q. B. 2010. Designing motivation strategies for software engineering teams: an empirical study. Proceedings of the 2010 ICSE Workshop on Cooperative and Human Aspects of Software Engineering. Cape Town, South Africa: ACM. FRASER, S., RISING, L., AMBLER, S., COCKBURN, A., ECKSTEIN, J., HUSSMAN, D., MILLER, R., STRIEBECK, M. & THOMAS, D. 2006. A fishbowl with piranhas: coalescence, convergence or divergence? Companion to the 21st ACM SIGPLAN symposium on Object-oriented programming systems, languages, and applications. Portland, Oregon, USA: ACM. FRIEDMAN, T. L. 2005. The World Is Flat-A Brief History of the Twenty-first Century, Farrar Straus Giroux. GARY, K., ENQUOBAHRIE, A., IBANEZ, L., CHENG, P., YANIV, Z., CLEARY, K., KOKOORI, S., MUFFIH, B. & HEIDENREICH, J. 2011. Agile methods for open source safety-critical software. Software: Practice and Experience, 41, 945-962. GEE, T. 2009. Apple Targets Health Care with iPhone 3.0 OS. Available: http://medicalconnectivity.com/2009/03/19/apple-targets-health-care-withiphone-30-os/#comments [Accessed 29/11/2011]. GEORGE, M. L. 2002. Lean Six Sigma: Combining Six Sigma Quality with Lean Production Speed, McGraw-Hill Professional GILB, T. 1985. Evolutionary Delivery versus the "waterfall model". SIGSOFT Softw. Eng. Notes, 10, 49-61. GIVEN, L. M. 2008. The SAGE Encyclopedia of Qualitative Research Methods, SAGE Publications. GLASS, R. L. 2001. Agile Versus Traditional: Make Love, Not War. Cutter IT, 12-18. GLASS, R. L. & VESSEY, I. 1995. Contemporary application-domain taxonomies. Software, IEEE, 12, 63-76. GOLDRATT, E. M. 1997. Critical Chain, Aldershot : Gower. GRAAF, B., LORMANS, M. & TOETENEL, H. 2003a. Embedded software engineering: the state of the practice. Software, IEEE, 20, 61-69. GRAAF, B., LORMANS, M. & TOETENEL, H. 2003b. Embedded Software Engineering: The State of the Practice. In: MARCO, L. & HANS, T. (eds.). GRAY, D. E. 2004. Doing research in the real world, Sage Publications. GREGG, D. G., KULKARNI, U. R. & VINZÉ, A. S. 2001. Understanding the Philosophical Underpinnings of Software Engineering Research in Information Systems. Information Systems Frontiers, 3, 169-183. GRENNING, J. 2002. Extreme programming and embedded software development. Embedded Systems Conference. Chicago. GROUT, J. R. & DOWNS, B. T. 2012. A Brief Tutorial on Mistake-proofing, Poka-Yoke, and ZQC. Available: http://facultyweb.berry.edu/jgrout/tutorial.html [Accessed 9/7/2012]. GUBA, E. & LINCOLN, Y. 1994. Competing Paragigms in Qualitative Research. Handbook of qualitative research. Sage Publications. HAKIM, C. 1987. Research Design: Strategies and Choices in the Design of Social Research, Routledge. HAMEL, G. 2006. Thw Why, What, and How of Management Innovation. (cover story). Harvard Business Review, 84, 72-84.
297
HANDY, C. B. 1985. Understanding organisations, Penguin Books. HARGADON, A. & SUTTON, R. I. 2000. Building an Innovation Factory. Harvard Business Review, 78, 157-166. HASTIE, S. 2009. Phil Abernathy on Agile Governance and Suncorp's Agile Transition [Online]. Available: http://www.infoq.com/articles/phil-abernathy-governance [Accessed 12/5/2010. HEALY, M. & PERRY, C. 2000. Comprehensive criteria to judge validity and reliability of qualitative research within the realism paradigm. Qualitative Market Research: An International Journal, 3. HERBSLEB, J. D. & GRINTER, R. E. 1999. Architectures, coordination, and distance: Conway's law and beyond. IEEE Software, 16, 63-70. HERBSLEB, J. D., MOCKUS, A., FINHOLT, T. A. & GRINTER, R. E. 2001. An empirical study of global software development: distance and speed. Proceedings of the 23rd International Conference on Software Engineering. Toronto, Ontario, Canada: IEEE Computer Society. HEVNER, A. R., MARCH, S. T., PARK, J. & RAM, S. 2004. Design science in information systems research. MIS Quarterly, 28, 75-105. HIBBS, C., JEWETT, S. C. & SULLIVAN, M. 2009. The Art of Lean Software Development, O'Reilly Media. HIGHSMITH, J. 2002. Agile Software Development Ecosystems, Addison Wesley. HINES, P., HOLWEG, M. & RICH, N. 2004. Learning to evolve: A review of contemporary lean thinking. International Journal of Operations & Production Management, 24, 9941011. HIRANABE, K. 2008. Kanban Applied to Software Development: from Agile to Lean [Online]. Available: http://www.infoq.com/articles/hiranabe-lean-agile-kanban [Accessed 12/1/2010 2010]. HNEIF, M. & HOCK OW, S. 2009. Review of Agile Methodologies in Software Development. International Journal of Research and Reviews in Applied Sciences, 1. HOSSAIN, E., BARBAR, M. A. & PAIK, H.-Y. 2009. Using Scrum in Global Software Development: A systematic Literature Review. ICGSE. HÖST, M. & RUNESON, P. Checklists for Software Engineering Case Study Research. Empirical Software Engineering and Measurement, 2007. ESEM 2007. First International Symposium on, 20-21 Sept. 2007 2007. 479-481. HOU, A. 1995. Toward Lean / Software System Development: Evaluation of Selected Complex Electronic system Development Methodologies. Massachusetts Institute of Technology. HOVE, S. E. & ANDA, B. Experiences from conducting semi-structured interviews in empirical software engineering research. Software Metrics, 2005. 11th IEEE International Symposium, 1-1 Sept. 2005 2005. 10 pp.-23. HUFFMAN HAYES, J., DEKHTYAR, A. & JANZEN, D. S. 2009. Towards traceable test-driven development. Proceedings of the 2009 ICSE Workshop on Traceability in Emerging Forms of Software Engineering. IEEE Computer Society. HUSTON, L. & SAKKAB, N. 2006. CONNECT AND DEVELOP. (cover story). Harvard Business Review, 84, 58-66. IEEE 2008. IEEE Reliability Society - Annual Technical Report 2008. IEEE. INCOSE. 1990. International Council on Systems Engineering [Online]. Available: www.incose.org [Accessed 8-Aug-2011.
298
ISO 2003. ISO 13485:2003 Medical devices -- Quality management systems -- Requirements for regulatory purposes International Organisation for Standardisation. ISO 2007. ISO 14971:2007 Medical devices -- Application of risk management to medical devices. International Organisation for Standardisation. ISO 2009. ISO/DIS 26262 - Functional Safety for Road Vehicles. International Standards Organisation. ISO/IEC 2004. ISO/IEC 15504-1:2004 Information technology -- Process assessment -- Part 1: Concepts and vocabulary. International Organisation for Standardisation & International Electrotechnical Commission. JACOBSON, I. keynote at EuroSPI 2010. EuroSPI, 2010 Grenoble, France. JALALI, S. & WOHLIN, C. Agile Practices in Global Software Engineering - A Systematic Map. Global Software Engineering (ICGSE), 2010 5th IEEE International Conference on, 23-26 Aug. 2010 2010. 45-54. JAYARATNA, N. 1986. NIMSAD: a framework for understanding and evaluating methodologies. Journal of Applied Systems Analysis, 73-78. JEFFRIES, R., ANDERSON, A. & HENDRICKSON, C. 2001. Extreme Programming Installed, Addison-Wesley. JORDAN, P. 2005. Medical device software standards. IEE Seminar Digests, 2005, 6-6. KAJKO-MATTSSON, M. Problems in agile trenches. Proceedings of the Second ACMIEEE international symposium on Empirical software engineering and measurement ESEM 08 (2008), 2008 Kaiserslautern, Germany. ACM Press. KANE, D. W., HOHMAN, M. M., CERAMI, E. G., MCCORMICK, M. W., KUHLMMAN, K. F. & BYRD, J. A. 2006. Agile methods in biomedical software development: a multi-site experience report. BMC Bioinformatics, 7, 273-12. KAROLAK, D. W. 1999. Global Software Development: Managing Virtual Teams and Environments, IEEE Computer Society Press. KAROW, M., PFEIFFER, D. & RÄCKERS, M. 2008. Empirical-Based Construction of Reference Models in Public Administrations. European Research Center for Information Systems. KETTUNEN, P. & LAANTI, M. 2005. How to steer an embedded software project: tactics for selecting the software process model. Information and Software Technology, 47, 587-608. KHURANA, A. & ROSENTHAL, S. R. 1997. Sloan Management Review. Sloan Management Review. KIM, H. K. 2008. Aspect Oriented Testing Frameworks for Embedded Software. In: LEE, R. (ed.) Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing. KIM, J. & WILEMON, D. 2002. Focusing the fuzzy front–end in new product development. R&D Management, 32, 269-279. KITCHENHAM, B. & CHARTERS, S. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering KLEIN, H. K. & MYERS, M. D. 1999. A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems. MIS Quarterly, 23, 67-93. KONRAD, M. & OVER, J. W. 2005. Agile CMMI: No Oxymoron [Online]. Available: http://www.drdobbs.com/architect/184415287. KOTTER, J. 1995. Leading change: Why Transformation Efforts Fail. Harvard Business Review, 73.
299
KOTTER, J. P. & SCHLESINGER, L. A. 1979. Choosing strategies for change. Harvard Business Review, 57, 106-114. LANE, S. & RICHARDSON, I. 2011. Process models for service-based applications: A systematic literature review. Information and Software Technology, 53, 424-439. LEANSSC. 2009. Lean Software and Systems Consortium [Online]. Available: http://www.leanssc.org/ [Accessed 8-Aug-2011. LEVEN, E. 2007. Large-Scale Adoption of Agile Development-Lessons Learned. Citigroup. LEVESON, N. 1995. Medical Devices: The Therac-25. Safeware: System Safety and Computers. Addison-Wesley. LEVY, M. & HAZZAN, O. 2009. Knowledge management in practice: The case of agile software development. 2009 ICSE Workshop on Cooperative and Human Aspects on Software Engineering, CHASE 2009, May 17, 2009 - May 17, 2009, 2009 Vancouver, BC, Canada. IEEE Computer Society, 60-65. LEWIN, K. 1946. Action Research and Minority Problems. Journal of Social Issues, 2, 34-46. LIKER, J. 2003. The Toyota Way McGraw-Hill. LIN, W. & FAN, X. Software development practice for FDA-compliant medical devices. 2009 International Joint Conference on Computational Sciences and Optimization, CSO 2009, April 24, 2009 - April 26, 2009, 2009 Sanya, Hainan, China. IEEE Computer Society, 388-390. LINCOLN, Y. & GUBA, E. 1988. Criteria for Assessing Naturalistic Inquiries as Reports. Annual Meeting of the American Educational Research Association New Orleans, LA. LINDVALL, M., BASILI, V., BOEHM, B., COSTA, P., DANGLE, K., SHULL, F., TESORIERO, R., WILLIAMS, L. & ZELKOWITZ, M. 2002. Empirical Findings in Agile Methods. In: WELLS, D. & WILLIAMS, L. (eds.) Extreme Programming and Agile Methods — XP/Agile Universe 2002. Springer Berlin Heidelberg. LIVERMORE, J. A. Factors that impact implementing an agile software development methodology. SoutheastCon, 2007. Proceedings. IEEE, 2007. 82-86. MARCH, S. T. & SMITH, G. F. 1995. Design and natural science research on information technology. Decision Support Systems, 15, 251-266. MASTICOLA, S. P. & GALL, M. 2008. Vision: testing of mechatronics software using agile simulation. Proceedings of the 3rd international workshop on Automation of software test. Leipzig, Germany: ACM. MAXWELL, J. A. 1996. Qualitative research design: an interactive approach, Sage Publications. MCCAFFERY, F. & CASEY, V. 2010. Med-Adept: A Lightweight Assessment Method for the Irish Medical Device Software Industry. European Systems & Software Process Improvement and Innovation Conference, (EuroSPI. Grenoble, France. MCCAFFERY, F., CASEY, V., SIVAKUMAR, M. S., COLEMAN, G., DONNELLY, P., BURTON, J. & MARIEL SEIFERT, L. 2011. Medical Device Software Traceability. MCCAFFERY, F. & DORLING, A. 2009. Medi SPICE : an Overview. SPICE 2009, 9th International Spice Conference. Pisa, Italy;. MCCAFFERY, F., DORLING, A. & CASEY, V. 2010. Medi SPICE : an update. SPICE 2010, 10th International Spice Conference. Pisa, Italy;. MCCAFFERY, F., PIKKARAINEN, M. & RICHARDSON, I. 2008. Ahaa --agile, hybrid assessment method for automotive, safety critical smes. Proceedings of the 30th international conference on Software engineering. Leipzig, Germany: ACM. MCCONNELL, S. 1998. Problem programmers. Software, IEEE, 15, 128, 127, 126. MCCONNELL, S. 2000. The Best Influences on Software Engineering. IEEE Software. IEEE.
300
MCHUGH, M., MCCAFFERY, F. & CASEY, V. 2011. Standalone Software as an Active Medical Device. In: O'CONNOR, R., ROUT, T., MCCAFFERY, F. & DORLING, A. (eds.) 11th International Conference on Software Process Improvement and Capability Determination. Dublin, Ireland: Springer. MCLOUGHLIN, F. 2010. The Rosetta Stone Methodology – A Benefits Driven Approach to Software Process Improvement. PhD, University of Limerick. MEDI-SPICE. 2011. Medi-SPICE [Online]. The Spice User Group. Available: http://medispice.ning.com/. MIDDLETON, P. 2001. Lean Software Development: Two Case Studies. Software Quality Journal, 9, 241-252. MIDDLETON, P., FLAXEL, A. & COOKSON, A. 2005. Lean Software Management Case Study: Timberline Inc. Extreme Programming and Agile Processes in Software Engineering. MILES, M. B. & HUBERMAN, M. A. 1994. Qualitative Data Analysis: An Expanded Sourcebook (2nd Edition), Sage Publications, Inc; 2nd edition (1994) MILLER, G. G. The Characteristics of Agile Software Processes. 39th Int’l Conf. and Exhibition on Technology of Object-Oriented Languages and Systems (TOOLS’01), 2001. IEEE. MORGAN, T. 1998. Lean Manufacturing Techniques Applied to Software Development Masters, Massachusetts Institute of Technology MORSE, J. M., BARRETT, M., MAYAN, M., OLSON, K. & SPIERS, J. 2002. Verification Strategies for Establishing Reliability and Validity in Qualitative Research. International Journal of Qualitative Methods, 1. MUELLER, G. & BORZUCHOWSKI, J. 2002. Extreme embedded a report from the front line. OOPSLA 2002 Practitioners Reports. Seattle, Washington: ACM. MULLINS, L. 2004. Management and Organisational Behaviour Prentice Hall. NAYLOR, B. J., MOHAMED, N. M. & DANNY, B. 1999. Leagility: Integrating the lean and agile manufacturing paradigms in the total supply chain. International Journal of Production Economics, 62, 107-118. NIELSEN, J. & MCMUNN, D. The agile journey -Adopting XP in a large financial services organization. 6th International Conference on Extreme Programming and Agile Processes in Software Engineering, XP 2005, June 18, 2005 - June 23, 2005, 2005 Sheffield, United kingdom. Springer Verlag, 28-37. NITRD. 2011. NITRD Homepage [Online]. Available: www.nitrd.gov [Accessed 10-Aug-2011. O'LEARY, P. 2010. Towards a Product Derivation Process Reference Model for Software Product Line Organisations. Doctorate Ph.D., Limerick. OAKLEIGH CONSULTING. 2007. Benefits of Lean and Agile Compared [Online]. Available: http://www.oakleigh.co.uk/page/3341/White-Papers/WhitepaperArticles/Benefits-of-Lean-and-Agile-Compared [Accessed 12/1/2010 2010]. OHNO, T. 1988. Toyota Production System: Beyond Large-Scale Production Productivity Press. OPEN-DO. 2011. Open-DO Initiative [Online]. Available: http://www.open-do.org/about/ [Accessed 17th October 2011]. OPPENHEIM, A. N. 2000. Questionnaire design, interviewing and attitude measurement, Continuum. OPPENHEIM, B. W., MURMAN, E. M. & SECOR, D. A. 2011. Lean Enablers for Systems Engineering. Systems Engineering, 14, 29-55.
301
ORLIKOWSKI, W. J. & BAROUDI, J. J. 1991. Studying Information Technology in Organizations: Research Approaches and Assumptions. INFORMATION SYSTEMS RESEARCH, 2, 1-28. PAIGE, R. F., CHARALAMBOUS, R., GE, X. & BROOKE, P. J. 2008. Towards Agile Engineering of High-Integrity Systems. Computer Safety, Reliability, and Security, 27th International Conference, SAFECOMP 2008. Newcastle upon Tyne, UK. PANG, C. K., NG, T. S., LEWIS, F. L. & LEE, T. H. 2012. Managing Complex Mechatronics R&D: A Systems Design Approach. Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions on, 42, 57-67. PARNAS, D. L. & CLEMENTS, P. C. 1986. A Rational Design process: how and why to fake it. IEEE Transactions on Software Engineering, 1986, 251-257. PARNELL-KLABO, E. 2006. Introducing Lean Principles with Agile Practices at a Fortune 500 Company. Proceedings of the conference on AGILE 2006. IEEE Computer Society. PETERSEN, K. & WOHLIN, C. 2010. Software process improvement through the Lean Measurement (SPI-LEAM) method. Journal of Systems and Software, 83, 12751287. PETERSEN, K. & WOHLIN, C. 2011. Measuring the flow in lean software development. Software: Practice and Experience, 41, 975-996. PETTICREW, M. & ROBERTS, H. 2006. Systematic Reviews in the Social Sciences: A Practical Guide Blackwell Pub. PFLEEGER, S. L. & ATLEE, J. M. 2010. Software engineering: theory and practice, Prentice Hall. POPPENDIECK, M. & POPPENDIECK, T. 2003. Lean Software Development: An Agile Toolkit Addison-Wesley Professional POPPENDIECK, M. & POPPENDIECK, T. 2006. Implementing Lean Software Development From Concept to Cash, Addison-Wesley Professional PORTER, M. E. 1998. Competitive advantage: creating and sustaining superior performance : with a new introduction, Free Press. PRAHALAD, C. K. & HAMEL, G. 1990. The Core Competence of the Corporation. Harvard Business Review, 68, 79-91. PUNTER, T., CIOLKOWSKI, M., FREIMUT, B. & JOHN, I. Conducting on-line surveys in software engineering. Empirical Software Engineering, 2003. ISESE 2003. Proceedings. 2003 International Symposium on, 30 Sept.-1 Oct. 2003 2003. 80-88. PYETT, P. M. 2003. Validation of Qualitative Research in the “Real World”. Qualitative Health Research, 13, 1170-1179. QUINN PATTON, M. 2002. Qualitative Research & Evaluation Methods, Sage Publications. R, T. 2011. Untangling intangible assets. OECD Observer, 13-15. RAMAN, S. Lean software development: is it feasible? Digital Avionics Systems Conference, 1998. Proceedings., 17th DASC. The AIAA/IEEE/SAE, 1998. C13/1-C13/8 vol.1. RASMUSSEN, R., HUGHES, T., JENKS, J. R. & SKACH, J. 2009. Adopting Agile in an FDA Regulated Environment. Agile Conference. REICH, B. H. 2007. MANAGING KNOWLEDGE AND LEARNING IN IT PROJECTS: A CONCEPTUAL FRAMEWORK AND GUIDELINES FOR PRACTICE. Project Management Journal, 38, 5-17. REID, S. E. & DE BRENTANI, U. 2004. The Fuzzy Front End of New Product Development for Discontinuous Innovations: A Theoretical Model. Journal of Product Innovation Management, 21, 170-184.
302
REINERTSEN, D. G. 2009. The Principles of Product Development Flow: Second Generation Lean Product Development Celeritas Publishing. RICHARDSON, I., AVRAM, G., DESHPANDE, S. & CASEY, V. 2008. Having a Foot on Each Shore - Bridging Global Software Development in the Case of SMEs. IEEE International Conference on Global Software Engineering. Bangalore, India. ROBERTS, T. L., JR., GIBSON, M. L., FIELDS, K. T. & RAINER, R. K., JR. 1998. Factors that impact implementing a system development methodology. Software Engineering, IEEE Transactions on, 24, 640-649. ROBILLARD, P. N. 1999. The role of knowledge in software development. Commun. ACM, 42, 87-92. ROBINSON, H. 1997a. Using Poka-Yoke Techniques for Early Defect Detection Sixth International Conference on Software Testing Analysis and Review ROBINSON, S. 1997b. Simulation Model Verification And Validation: Increasing The Users’ Confidence. In: ANDRADÓTTIR, S., HEALY, K. J., WITHERS, D. H. & NELSON, B. L. (eds.) Proceedings of the 1997 Winter Simulation Conference. RONKAINEN, J. & ABRAHAMSSON, P. Software development under stringent hardware constraints: do agile methods have a chance? Extreme Programming and Agile Processes in Software Engineering. 4th International Conference, XP 2003. Proceedings, 25-29 May 2003, 2003 Berlin, Germany. Springer-Verlag, 73-9. ROSEMANN, M. & SCHÜTTE, R. 1999. Multi-Perspective Reference Modelling. In: BECKER, J., ROSEMANN, M. & SCHÜTTE, R. (eds.) Referenzmodellierung: State-of-the-Art und Entwicklungsperspektiven. Heidelberg: Physica-Verlag. ROTTIER, P. A. & RODRIGUES, V. Agile Development in a Medical Device Company. Agile, 2008. AGILE '08. Conference, 2008. 218-223. ROYCE, W. W. Managing the Development of Large Software Systems: Concepts and Techniques. Western Electronic Show and Convention (WesCon), 1970 Los Angeles, USA. RTCA. 2011. RTCA Home Page [Online]. Radio Technical Commission for Aeronautics. Available: http://www.rtca.org/ [Accessed 10-Aug-2011. RTCA January 1992. DO-178B, Software Considerations in Airborne Systems and Equipment Certification. RTCA (Radio Technical Commission for Aeronautics). RUNESON, P. & HOST, M. 2009. Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering 14. RYAN, J. J., JR. & SCUDIERE, R. The price of agile is eternal vigilance. Agile 2008, 4-9 Aug. 2008, 2008 Piscataway, NJ, USA. IEEE, 125-8. RYAN, R. M. & DECI, E. L. 2006. Self-Regulation and the Problem of Human Autonomy: Does Psychology Need Choice, Self-Determination, and Will? Journal of Personality, 74, 1557-1586. SABETZADEH, M., NEJATI, S., BRIAND, L. & MILLS, A.-H. E. Using SysML for Modeling of Safety-Critical Software-Hardware Interfaces: Guidelines and Industry Experience. High-Assurance Systems Engineering (HASE), 2011 IEEE 13th International Symposium on, 10-12 Nov. 2011 2011. 193-201. SALO, O. & ABRAHAMSSON, P. 2008. Agile methods in European embedded software development organisations: A survey on the actual use and usefulness of Extreme Programming and Scrum. IET Software, 2, 58-64. SALO, O., KOLEHMAINEN, K., KYLLÖNEN, P., LÖTHMAN, J., SALMIJÄRVI, S. & ABRAHAMSSON, P. 2004. Self-Adaptability of Agile Software Processes: A Case Study on Post-iteration Workshops. In: ECKSTEIN, J. & BAUMEISTER, H. (eds.)
303
Extreme Programming and Agile Processes in Software Engineering. Springer Berlin / Heidelberg. SCHULMEYER, G. G. 1990. Zero Defect Software, Mcgraw Hill. SCHWABER, K. 2004. Agile Project Management with Scrum, Microsoft Press. SCHWABER, K. & BEEDLE, M. 2001. Agile Software Development with Scrum Prentice Hall. SCIPLORE. 2011. Mindmap Tool [Online]. http://www.sciplore.org/. Available: http://www.sciplore.org [Accessed 13/12/2011 2011]. SEO, J., SUNG, A., CHOI, B. & KANG, S. 2007. Automating Embedded Software Testing on an Emulated Target Board. Proceedings of the Second International Workshop on Automation of Software Test. IEEE Computer Society. SHARP, H. & HALL, T. 2009. An initial investigation of software practitioners' motivation. Proceedings of the 2009 ICSE Workshop on Cooperative and Human Aspects on Software Engineering. IEEE Computer Society. SHOEMAKER, B. & VAN SCHOOENDERWOERT, N. 2010. When It Just Has to Work: Agile Development in Safety-Critical Environments. SIDKY, A. & ARTHUR, J. 2007. Determining the Applicability of Agile Practices to Mission and Life-Critical Systems. Proceedings of the 31st IEEE Software Engineering Workshop. IEEE Computer Society. SIDKY, A., ARTHUR, J. & BOHNER, S. 2007. A disciplined approach to adopting agile practices: the agile adoption framework. Innovations in Systems and Software Engineering, 3, 203-216. SIGGELKOW, N. 2007. PERSUASION WITH CASE STUDIES. Academy of Management Journal, 50, 20-24. SIMON, J.-M. 1996. SPICE: Overview for software process improvement. Journal of Systems Architecture, 42, 633-641. SMALL, A. W. & DOWNEY, E. A. 2001. Managing change: some important aspects. Change Management and the New Industrial Revolution, 2001. IEMC '01 Proceedings., 2001 2001. 50-57. ŠMITE, D., WOHLIN, C., GORSCHEK, T. & FELDT, R. 2009. Empirical evidence in global software engineering: a systematic review. Empirical Software Engineering. SMITH, P. G. & REINERTSEN, D. G. 1998. Developing Products in Half the Time: New Rules, New Tools, Wiley. SMITS, H. & RILLIET, K. Agile Experience Report: Transition and Complexity at Cisco Voice Technology Group. AGILE Conference (AGILE), 2011, 7-13 Aug. 2011 2011. 274278. SOX 2002. Sarbanes-Oxley Act of 2002. In: CONGRESS, T. (ed.). SPENCE, J. W. There has to be a better way! AGILE Conference 2005, July 24, 2005 - July 29, 2005, 2005 Denver, CO, United states. Inst. of Elec. and Elec. Eng. Computer Society, 272-278. SPERRY, R. & JETTER, A. Theoretical framework for managing the front end of innovation under uncertainty. Management of Engineering & Technology, 2009. PICMET 2009. Portland International Conference on, 2-6 Aug. 2009 2009. 2021-2028. SRINIVASAN, J., DOBRIN, R. & LUNDQVIST, K. 'State of the Art' in Using Agile Methods for Embedded Systems Development. Computer Software and Applications Conference, 2009. COMPSAC '09. 33rd Annual IEEE International, 2009. 522-527. STAKE, R. E. 1995. The Art Of Case Study Research, Sage Publications, Inc.
304
STANDARD, I. E. 1995. IEEE/EIA Standard Industry Implementation of International Standard ISO/IEC 12207: 1995 (ISO/IEC 12207) Standard for Information Technology Software Life Cycle Processes. IEEE/EIA 12207.0-1996, i-75. STAPLETON, J. & CONSTABLE, P. 1997. Dynamic Systems Development Method, AddisonWesley Professional STENBACKA, C. 2001. Qualitative research requires quality concepts of its own. Management Decision, 39, 551-556. STRAUSS, A. & CORBIN, J. 1990. Basics of qualitative research: grounded theory procedures and techniques, Sage Publications. SUGIMORI, Y., KUSUNOKI, K., CHO, F. & UCHIKAWA, S. 1977. Toyota production system and Kanban system Materialization of just-in-time and respect-for-human system. International Journal of Production Research, 15, 553 - 564. SUPERVISION, B. C. O. B. 2004. Basel II Accord. Bank for International Settlements SUTTON, J. M. Lean software for the lean aircraft. Digital Avionics Systems Conference, 1996., 15th AIAA/IEEE, 27-31 Oct 1996 1996. 49-54. TAYLOR, P. S., GREER, D., COLEMAN, G., MCDAID, K. & KEENAN, F. 2008. Preparing small software companies for tailored agile method adoption: Minimally intrusive risk assessment. Software Process: Improvement and Practice, 13, 421-437. TESSEM, B. & MAURER, F. 2007. Job satisfaction and motivation in a large agile team. Proceedings of the 8th international conference on Agile processes in software engineering and extreme programming. Como, Italy: Springer-Verlag. THE-FREE-DICTIONARY 2011. Regulations. THE-IRISH-EXAMINER. 2011. Quinn Group - ‘Light touch’ regulation was madness. The Irish Examiner, 15-Apr-2011. THE-WORLD-BANK 2004. Doing Business in 2004. The World Bank. THIMBLEBY, H. 1988. Delaying commitment [programming strategy]. Software, IEEE, 5, 7886. TIERNERY, J. 1993. From Deming to Shingo: using Japanese Quality Assurance principles to eradicate problems from your software development process. Microsoft Corporation. TREUDE, C., BARZILAY, O. & STOREY, M.-A. 2011. How do programmers ask and answer questions on the web? (NIER track). Proceedings of the 33rd International Conference on Software Engineering. Waikiki, Honolulu, HI, USA: ACM. TSAI, W.-T., PAUL, R., YU, L. & WEI, X. 2005. Rapid Pattern-Oriented Scenario-Based Testing for Embedded Systems. In: YANG, H. (ed.) Software Evolution with UML and XML. IGI Global. VÄHÄNIITTY, J. & RAUTIAINEN, K. 2008. Towards a Conceptual Framework and Tool Support for Linking Long-term Product and Business Planning with Agile Software Development. The 1st international workshop on Software development governance New York, USA: ACM. VAISHNAVI, V. K. & KUECHLER, W. 2007. Design Science Research Methods and Patterns: Innovating Information and Communication Technology, Taylor & Francis. VAN SCHOOENDERWOERT, N. Embedded agile project by the numbers with newbies. AGILE Conference, 2006, July 23, 2006 - July 28, 2006, 2006 Minneapolis, MN, United states. Inst. of Elec. and Elec. Eng. Computer Society, 351-363. VAN SCHOOENDERWOERT, N. 2008. Safety-Critical Applications Built via Agile Discipline. Available: http://www.boston-spin.org/slides/boston_spin_slides_2008_09.pdf.
305
VAN SCHOOENDERWOERT, N. & MORSICATO, R. Taming the embedded tiger - Agile test techniques for embedded software. Proceedings of the Agile Development Conference, ADC 2004, June 22, 2004 - June 26, 2004, 2004 Salt Lake City, UT, United states. IEEE Computer Society, 120-126. VANDERLEEST, S. H. & BUTER, A. Escape the waterfall: Agile for aerospace. Digital Avionics Systems Conference, 2009. DASC '09. IEEE/AIAA 28th, 2009. 6.D.3-1-6.D.3-16. VARDANEGA, T. & VAN KATWIJK, J. 1998. Productive engineering of predictable embedded real-time systems: the road to maturity. Information and Software Technology, 40, 745-764. VARDEMAN, S. B., KANN, C., BUMBLAUSKAS, D., HECK, K. & PATNAIK, A. 2002. The Impact of Dr. Shigeo Shingo on Modern Manufacturing Practices. Available: http://www.public.iastate.edu/~vardeman/IE361/f02mini/bumblauskas.pdf. VERNER, J. M., SAMPSON, J., TOSIC, V., BAKAR, N. A. A. & KITCHENHAM, B. A. Guidelines for industrially-based multiple case studies in software engineering. Research Challenges in Information Science, 2009. RCIS 2009. Third International Conference on, 22-24 April 2009 2009. 313-324. VOGEL, D. A. 2010. Medical Device Software Verification, Validation and Compliance Artech House Publishers. WANG, X. 2011. The Combination of Agile and Lean in Software Development: An Experience Report Analysis. Agile 2011. Salt Lake City, Utah, USA. WANG, X., CONBOY, K. & CAWLEY, O. 2012. “Leagile” software development: An experience report analysis of the application of lean approaches in agile software development. Journal of Systems and Software, 85, 1287-1299. WEICK, K. E. 1995. What Theory Is Not, Theorizing Is. Administrative Science Quarterly, 40, 385-390. WEYRAUCH, K. What are we arguing about? A framework for defining agile in our organization. Agile Conference, 2006, 2006. 8 pp.-220. WILLIAMS, L. & COCKBURN, A. 2003. Guest Editors' Introduction: Agile Software Development: It's about Feedback and Change. Computer, 36, 39-43. WILS, A., VAN BAELEN, S., HOLVOET, T. & DE VLAMINCK, K. Agility in the avionics software world. 7th International Conference on Extreme Programming and Agile Processes in Software Engineering, XP 2006, June 17, 2006 - June 22, 2006, 2006a Oulu, Finland. Springer Verlag, 123-132. WILS, A., VAN BAELEN, S., HOLVOET, T. & DE VLAMINCK, K. 2006b. Agility in the Avionics Software World. Extreme Programming and Agile Processes in Software Engineering, 7th International Conference, XP 2006. Oulu, Finland, June 17-22, 2006. WINTER, G. 2000. A Comparative Discussion of the Notion of 'Validity' in Qualitative and Quantitative Research The Qualitative Report, 4. WIRTH, N. 1995. A plea for lean software. IEEE Computer, 28, 64-68. WOMACK, J. P. & JONES, D. T. 1994. From Lean Production to the Lean Enterprise. (cover story). Harvard Business Review, 72, 93-103. WOMACK, J. P. & JONES, D. T. 1996. Lean Thinking : Banish Waste and Create Wealth in Your Corporation Simon & Schuster. WOMACK, J. P., JONES, D. T. & ROOS, D. 1990. The Machine That Changed The World: How lean production revolutionized the global car wars, Simon & Schuster Ltd. WOMACK, J. P., JONES, D. T. & ROOS, D. 2007. The Machine That Changed The World: How lean production revolutionized the global car wars, Simon & Schuster Ltd.
306
WOODALL, P. & BRERETON, P. 2006. Conducting a Systematic Literature Review from the Perspective of a Ph.D. Student. 10th International Conference on Evaluation and Assessment in Software Engineering. Keele University, Staffordshire, UK. YABUUCHI, Y., KOCAOGLU, D. & WATADA, J. Analysis of Project Management in Software Development. Technology Management for the Global Future, 2006. PICMET 2006, July 2006 2006. 2809-2814. YE, Y., NAKAKOJI, K. & YAMAMOTO, Y. 2008. The economy of collective attention for situated knowledge collaboration in software development. Proceedings of the 2008 international workshop on Cooperative and human aspects of software engineering. Leipzig, Germany: ACM. YIN, R. K. 2009. Case Study Research: Design and Methods Sage Publications, Inc. YOO, J., CHA, S., KIM, C. H. & SONG, D. Y. 2005. Synthesis of FBD-based PLC design from NuSCR formal specification. Reliability Engineering & System Safety, 87, 287-294. ZIEGLER, A. S. 2006. Regulation. Annals of the New York Academy of Sciences, 1093, 339349.
307
Appendix A – The Mapping Study Process
From an overall perspective this process involves three main steps:
Figure A-61 The Mapping Study Process Steps
By approaching the review in such a systematic manner and demonstrating the rigor applied to each step, a higher level of confidence is built into the conclusions. In particular, the first step has several component steps which are critical to this. They are:
Specify Research Questions
Develop a Review Protocol
Evaluate the Review Protocol
Figure A-62 Planning the Mapping Study
These set the boundaries of the search and help to keep the researcher focused as they progress. A protocol was developed containing a full breakdown of the research approach. Specifically included was the following a priori assumption in order to make any bias clearly identifiable: - Some practices/aspects of lean/agile software development do not adequately support all the requirements of the regulations for safetycritical software development, such as those laid down by the US/EU regulatory bodies. The main tasks described in the protocol are: identifying research questions, identifying data sources, building search strings, performing pilot search, adjusting search criteria, exporting results for citation management, applying inclusion/exclusion criteria, quality assessment, data extraction, and data synthesis. The full protocol can be seen in Appendix B. While this offers a very positive approach in terms of providing a clear and unambiguous path through the literature, other important contributions were available from industry sources such as non-academic books, reports, and other online resources. This ‘grey literature’ assisted in addressing publication bias (Cooper et al., 2009) Chapter 23, (Kitchenham and Charters, 2007) pg 15, but it is also highlighted as a limitation to the review findings.
308
Mapping Study Research Questions In posing the research questions cognisance was taken of lessons learned from other SLRs and advice from prominent researchers. For example: (Kitchenham and Charters, 2007) warn against having too broad a scope in a mapping study in order to ensure the research has a good chance of completing within the time allowed. Since the normal literature review had shown a lack of literature on my specific area of interest, the area of focus was widened for the mapping study to cover safety-critical domains in general. This would also include the MD domain since all the associated domains are regulated to a similar degree. The final research questions for the mapping study therefore became:
Mapping Study Question 1:
What is currently known about the use of lean or agile software development methodologies within companies developing safety-critical embedded software which must comply with regulatory standards?
Mapping Study Question 2:
How is the use of lean or agile software development within these environments being restricted?
Mapping Study Question 3:
How can such restrictions be circumvented in support of lean or agile methods without compromising regulatory compliance?
Search Sources and Strings Taking the mapping study guidelines as a starting point and from looking at published SLRs (Chen et al., 2009), (Šmite et al., 2009), (Hossain et al., 2009), (Lane and Richardson, 2011), the electronic sources noted in Table A-1 were used. An important lesson in the practice of searching electronic databases is that each database search engine is different as highlighted by (Brereton et al., 2007). The specific search strings were formed following the PICOC (Population, Intervention, Comparison, Outcomes, Context) criteria suggested by (Petticrew and Roberts, 2006). The generic query used can be written as follows: “any document containing the phrase ‘medical device’ or ‘embedded software’ or the word stem ‘regulat’ AND containing any of the following phrases (‘software development’, ‘software process’, ‘software life-cycle’, ‘manufacturing software’) AND containing any of the following words/phrases (‘agile’, ‘lean’, ‘scrum’, ‘XP’, ‘extreme programming ‘, ‘crystal’,).
309
Table A-32 The Mapping Study Literature Sources SOURCE Acm Compendex IEEE INSPEC ScienceDirect Web of Science Misc.
URL http://portal.acm.org/portal.cfm http://www.engineeringvillage2.org http://ieeexplore.ieee.org/Xplore/ http://www.engineeringvillage2.org http://www.sciencedirect.com/ http://apps.isiknowledge.com Book Sections, Thesis, Industry Reports, Web sites
Inclusion and Exclusion criteria
When conducting a mapping study, the guidelines tell us that explicit conditions should be documented for the inclusion and exclusion of studies in/from the results. The inclusion criteria I used were: - Peer reviewed research papers - Grey literature from experienced practitioners - Relevant to safety-critical software development - Focus on lean or agile practices Additionally, the exclusion criteria included: - Non-English language - Content too general (for example a high level review of agile practices) - Duplicated work - Off topic (does not concentrate on expected subject matter)
Quality Attributes
As described by (Kitchenham and Charters, 2007), an integral part of a mapping study is to assess the quality of the primary studies. This aids in determining the relevant importance of a study by capturing, for example: - the possible effects of bias - the importance of a study in the body of literature - the relevance to your particular research question(s) In order to determine the value of each paper, a series of questions was asked of each one (Table A-2) and the answer recorded on the data extraction sheet. Following this two more attributes to each entry were added; (1) an overall rating [1-poor to 5-excellent], indicating the importance of the contribution made to the research questions, and (2) a comment to explicitly state the limitations of the contribution. This was particularly important step since many of the papers were not empirically based. The average overall rating assigned was 3.33, with the lowest rating of 2 being given to 4 of the 21 papers. No papers were excluded based on their quality rating.
310
Table A-33 Researcher’s Questions to Solicit Quality of Paper Does the study predominantly relate to software development within a safetycritical, regulated setting? Which particular lean/agile methodology does it look at in depth? Does it report on real-life case studies where a lean/agile methodology was used? Does the study find in favour of the applicability of lean/agile software development methodologies within a safety-critical, regulated setting? What ‘flavour’ of lean/agile does the study promote? Lean (specifically refers to a lean approach) Agile (the practices of a single agile methodology e.g. SCRUM, XP, CRYSTAL) Agile-Agile (a combination of different agile methodologies) Agile-Planned (a combination of agile and traditional methodologies) None (it does not recommend agile methodologies)
Data Extraction
Following the technique used by (Dybå and Dingsøyr, 2008) for citation management, the results of the searches were exported to an EndNote library (a bibliography tool). Within EndNote, the publications underwent an initial inclusion/exclusion analysis. This was carried out by the primary reviewer and validation performed by secondary reviewers (researcher’s supervisors). A two step approach to the inclusion/exclusion of papers was carried out (Figure A-3). First, papers were excluded on the basis of their title and abstract. All remaining papers were then read in full and irrelevant papers excluded. The remaining entries formed the basis of the review.
Figure A-63 The two Stage Review Process
The relevant data was extracted into excel (data extraction template), where subsequent information, for example, inclusion/exclusion justification, quality attributes and a short note on the limitations of each publication, were recorded. From an initial count of 160 results, the data set was reduced down to 49 in the first review, and then down to 21 in the second review. The complete list is included in Appendix C, and a truncated version of the data extraction sheet in Appendix D.
311
Appendix B – The Mapping Study Protocol Research Question
The area I am interested in investigating is the use of Lean/Agile software development methodologies within the medical device industry. There is some debate about whether in fact there is any difference between Lean and Agile software development, but for the purpose of this review is does not matter since I will be searching for both terms. This chosen area is interesting because the industry is heavily regulated to ensure the highest level of safety in the devices is attained. As a result, the regulations stipulate quite a number of practices which must be implemented and require an audit trail be kept to show that this is happening. What is worth investigating is whether this type of formal regulation can be satisfied with the more flexible and iterative development practices espoused by the Lean and Agile software development supporters. It is important to state that I am aware of two software development scenarios within this area. The first is the software which gets embedded in the medical device, and the second is the software which is used to assist/control the manufacture of the devices. Both scenarios are governed by the regulations but clearly they operate within different constraints and have different dependencies. I am interested in both scenarios and for the purpose of this review will not draw a distinction between them. I will however be mindful of this as I review the literature. In order to be clear on where my personal bias is leaning I will state my hypothesis/assumption: - “Some practices/aspects of Lean/Agile software development do not adequately support all the requirements of the regulations for medical device software development, such as those laid down by the US/EU regulatory bodies” With this in mind my research questions are: 1. What is currently known about the use of Lean/Agile software development methodologies operating within companies which must comply with regulatory standards governing software development for medical devices? 2. What are the factors which restrict the use of Lean/Agile within such environments? 3. What practices are being used to address such restrictions in support of Lean/Agile methods? 4. Are there classes/categories of medical devices for which Lean/Agile software development methodologies do and do not work? 5. Do such Lean/Agile approaches deliver the expected return on investment? (Kitchenham and Charters, 2007) warn against having too broad a scope in a mapping study in order to ensure the research has a good chance of completing
312
within the time allowed. Also, (Woodall and Brereton, 2006) suggest that the research questions for a lone PhD student do not have to be as detailed as when a group of researchers are involved. Therefore my revised questions have been simplified as follows: 1. What is currently known about the use of Lean/Agile software development methodologies operating within companies which must comply with regulatory standards governing software development for medical devices? 2. What are the factors which restrict the use of Lean/Agile within these environments? 3. What practices are being used to address such restrictions in support of Lean/Agile methods? In anticipation that the above questions return too few papers, as the area of Lean/Agile development specifically to medical devices may not be very well covered, then the following questions can be used: 1. What is currently known about the use of Lean/Agile software development methodologies operating within companies developing embedded safety-critical software which must comply with regulatory standards? 2. What are the factors which restrict the use of Lean/Agile within these environments? 3. What practices are being used to address such restrictions in support of Lean/Agile methods?
Building the search strings
Using the PICOC criteria (Petticrew and Roberts, 2006), I have formulated my keywords as follows: - Population: o Medical Device Manufacturers o Software Development Professionals - Intervention: o Lean Software Development o Agile Software Development - Comparison: o Traditional plan-based Software Development models e.g. Waterfall (I do not assume that the waterfall model works particularly well in a regulated environment, but it is a methodology that has been historically, and is still the widest used in software development projects (Dyba and Dingsoyr, 2009)) - Outcomes: o More efficient and cost effective software development lifecycles while maintaining regulatory compliance - Context: o Regulated Industry
313
Keywords: - Medical Device o Also to catch papers referring to regulatory compliance: Regulatory compliance Regulated industry o Initial searches are limited in their results so expanding to include: Embedded software - Software Development o Because there are many variations of this I will also include: software process software life-cycle manufacturing software software engineering - Agile o Because ‘Agile’ is more of an umbrella term for many similar methods which follow a common philosophy, I will include some of the more common methods in my search, specifically: SCRUM XP and “Extreme Programming” CRYSTAL - Lean The generic search string logic can be expressed as follows: Any document containing the phrase ‘medical device’ or ‘embedded software’ or the word stem ‘regulat’ AND containing any of the following phrases (‘software development’, ‘software process’, ‘software life-cycle’, ‘manufacturing software’) AND containing any of the following words/phrases (‘agile’, ‘lean’, ‘scrum’, ‘XP’, ‘extreme programming ‘, ‘crystal’,).
Sources As suggested in the guidelines (Kitchenham and Charters, 2007), I will use the following digital sources: o IEEExplore (http://ieeexplore.ieee.org/Xplore/) o ACM Digital library (http://portal.acm.org/portal.cfm) o Inspec (http://www.engineeringvillage2.org) o ScienceDirect (www.sciencedirect.com) o EI Compendex (http://www.engineeringvillage2.org) From other SRs I have looked at (Chen et al., 2009), (Hossain et al., 2009), I have decided to include the following sources: o Web of Science http://apps.isiknowledge.com o University of Limerick Library http://193.1.100.111/TalisPrism
314
An important lesson in the practice of searching electronic databases is that each database search engine is different as highlighted by (Brereton et al., 2007). An exception is obviously Inspec and Compendex in this case. We see in Table B-1 below the search strings which although contain the same keywords, are structured differently and in some cases wild-card characters are used. A caveat for anyone developing search strings and using Microsoft Word is that cutting and pasting the strings which include quotation marks can be problematic. In this case I kept my master search strings in a normal text file separate to this protocol. Table B-34 Search Sources and Results Source
Search string
Comment
IEEExplore
(((medical device or regulat* or embedded software) and (agile or lean or scrum or xp or extreme program* or crystal) and (software development or software process or software life cycle or software engineering) )metadata) ("medical device" or "medical devices" or "regulated industry" or "regulatory compliance" or "embedded software") AND (lean or agile or xp or scrum or crystal or "extreme programming") AND ("software development" or "software process" or "software life-cycle" or "software life cycle" or "software engineering") ("medical device" or "medical devices" or regulat* or "embedded software" WN All) AND (lean or agile or scrum or xp or "extreme programming" or crystal WN ALL) AND ("software development" or "software process" or "software life-cycle" or "software life cycle" or "software engineering" WN ALL) ("medical device" or "medical devices" or regulat* or "embedded software" WN All)
Publications: All selected Other Resources: All selected Year: All to Present
ACM Digital library
EV Inspec
EV Compendex
315
Count
Valid
8
6
58
13
Limit by: All Document Types All treatment types All disciplines All languages 1969-2010 Autostemming: ON
15
5
Limit by: All Document Types
35
10
ScienceDirect
AND (lean or agile or scrum or xp or "extreme programming" or crystal WN ALL) AND ("software development" or "software process" or "software life-cycle" or "software life cycle" or "software engineering" WN ALL) ("medical device" or "medical devices" or "embedded software") AND (regulat*)AND (agile or lean or xp or crystal or "extreme programming") AND ("software development" or "software process" or "software life-cycle" or "software life cycle" or "software engineering")
All treatment types All disciplines All languages 1969-2010 Autostemming: ON
Include: Only Journals selected. Sources: All Sources Subject: “Business, Management and Accounting”, “Computer Science”, Decision Sciences”, “Economics, Econometrics and Finance”, “Engineering”, “Medicine and Dentistry”, “Nursing and Health Professions” Dates: All Years Note that the tag FT stands for “Funding Text” not “Full Text” Timespan=All Years. Databases=SCIEXPANDED, SSCI, A&HCI, CPCI-S, CPCI-SSH.
28
3
5
0
Web of Science
TS=("medical device" or "medical devices" or regulat* or "embedded software") AND TS=(lean or agile or scrum or xp or "extreme programming") AND TS=("software development" or "software process" or "software life-cycle" or "software life cycle" or "software engineering")
UL Library catalog
1
1
UL Library catalog
Keyword search “software development regulation” Keyword search “lean software development”
2
2
UL Library catalog
Keyword search “agile software development”
3
0
UL Library catalog
Keyword search “medical device software development”
1
0
316
Procedure
An initial pilot test will be carried out and the search strings/sources adjusted as appropriate. Following the technique used by (Dybå and Dingsøyr, 2008) for citation management, the results will be exported to an EndNote Library. Within EndNote, the publications will undergo an initial inclusion/exclusion analysis. This will be carried out by the primary reviewer and validation performed by the secondary reviewers.
Inclusion/Exclusion criteria At the outset no data sources are being specifically excluded. Preliminary searches have shown a limited amount of published material in this specific area, therefore we are not going to limit ourselves to any specific publication types. However the data extraction will include publication type for clarity. There will be a two step approach to the inclusion/exclusion of papers. First, papers will be excluded on the basis of their title and abstract. Next all remaining papers will be read in full and irrelevant papers excluded. Reasons for excluding at this stage will be: - Non-English language - Conference proceedings - Too general - Duplicated work - Off topic (does not concentrate on expected topic) The remaining entries will form the basis of the review. The relevant data will then be extracted into excel, by means of the data extraction form below, where any subsequent information e.g. inclusion/exclusion criteria, will be recorded. Search Results 1
Valid Tile and Abstract?
Search Results 2
Valid Full Text?
Search Results 3
Data Extraction
Figure B-64 The Data Extraction Process
The data synthesis will be performed by the primary reviewer and will be a descriptive summary of the literature without any formal meta-analysis being performed. Relevant data will be tabulated and presented in a manner consistent with the research questions.
317
Study Quality assessment
As described by (Kitchenham and Charters, 2007), an integral part of a SR is to assess the quality of the primary studies. This aids in determining the relevant importance of a study by capturing, for example: - the possible effects of bias - the importance of a study in the body of literature - the relevance to your particular research question(s) Following an initial inclusion/exclusion analysis I will mark each paper against the following questions Table B-35 Quality Assessment Questions
Quality Attribute
Value
Does the study predominantly relate to software development for medical devices? Which particular Lean/Agile methodologies does it look at in depth? Does it report on real-life case studies where a Lean/Agile methodology was used? Does the study find in favour of the applicability of Lean/Agile software development methodologies within the medical device industry? What ‘flavour’ of Lean/Agile does the study promote? - Agile (the complete practices of any single Agile methodology e.g. SCRUM, XP, CRYSTAL) - Agile-Agile (a combination of different agile models) - Agile-Planned (a combination of Agile and traditional models) - None (it does not recommend Agile methodologies)
Data Extraction Form
Since the quality criteria are not being used for separate inclusion/exclusion decisions, they will form part a single data extraction form. For ease of storage, tracking and review, all extracted data will be recorded in one spreadsheet instead of a form per paper/article. See template on next page. Some of the fields will be drop down lists as follows: Table B-36 Sample of Data Extraction Fields
Type
Recommendati on
Journal Article
Agile
Conference Paper
Agile-Agile
Primary Seconda ry
Workshop Paper
Agile-Planned
Tertiary
Book Section Technical Paper
None
Sources
318
Study Type Controlled experiment
Domain
Quasi experiment
Aerospace Medical Devices
Case study Exploratory case study Experience report
Aviation
Nuclear Automotive
Web Site Article Conference Proceedings Thesis Report Magazine Article Other
Meta-analysis Example application Survey Discussion Undefined
319
Satellites Financial Services Multiple Other
Table B-37 Data Extraction Template
Exclud e
Meta-Data
Cou nt
Name of Review er
Date of Data Extract ion
Titl e
Author s
Typ e of Arti cle
Exclusion Reason
Description
Study Sourc es
Stud y Typ e
Study Doma in
1 2 3 4 5 6 7 8 9 10
320
Quality Attributes
Does the study predomina ntly relate to software developme nt for medical devices?
Which particular Lean/Agile methodolog ies does it look at in depth?
Does it report on real-life case studies where a Lean/Agile methodolo gy was used?
Does the study find in favour of the applicabilit y of Lean/Agile software developmen t methodolog ies within the medical device industry?
What ‘flavour’ of Lean/Ag ile does the study promote ?
What are the limitatio ns of the study?
Appendix C – Selected Literature from Mapping Study [S1] [S2] [S3]
[S4] [S5] [S6] [S7]
[S8] [S9]
[S10]
[S11]
[S12] [S13] [S14]
Aiken, J. 2004. Technical and human perspectives on pair programming. SIGSOFT Softw. Eng. Notes, 29, 1-14. Ambler, S. W. 2006. Imperfectly agile: You too can be agile! Dr. Dobb's Journal, 31, 82-84. Bosch, T. 2007. Medical Device Software Development—Going Agile [Online]. MDDI (Medical Device and Diagnostic Industry). Available: http://www.mddionline.com/article/medical-device-softwaredevelopmentmdashgoing-agile [Last Accessed 17th June 2011]. Chisholm, R. A. 2007. Agile Software Development Methods And DO178B Certification. Master Of Applied Computer Science, Royal Military College Of Canada. Cordeiro, L., Barreto, R., Barcelos, R., Oliveira, M., Lucena, V. & Maciel, P. 2007. TXM: an agile HW/SW development methodology for building medical devices. ACM SIGSOFT Softw. Eng. Notes, 32, 4. Graaf, B., Lormans, M. & Toetenel, H. 2003. Embedded Software Engineering: The State of the Practice. In: Marco, L. & Hans, T. (eds.). Huffman Hayes, J., Dekhtyar, A. & Janzen, D. S. 2009. Towards traceable test-driven development. Proceedings of the 2009 ICSE Workshop on Traceability in Emerging Forms of Software Engineering. IEEE Computer Society. Kettunen, P. & Laanti, M. 2005. How to steer an embedded software project: tactics for selecting the software process model. Information and Software Technology, 47, 587-608. Lin, W. & Fan, X. 2009. Software development practice for FDA-compliant medical devices. In: International Joint Conference on Computational Sciences and Optimization, CSO 2009, April 24 - April 26, 2009, Sanya, Hainan, China. IEEE Computer Society, 388-390. Manhart, P. & Schneider, K. 2004. Breaking the ice for agile development of embedded software: an industry experience report. In: Proceedings of the 26th International Conference on Software Engineering, 2004. 378386. McCaffery, F., Pikkaraainen, M. & Richardson, I. 2008. Ahaa --agile, hybrid assessment method for automotive, safety critical smes. In: Proceedings of the 30th international conference on Software engineering. Leipzig, Germany 2008. Mueller, G. & Borzuchowski, J. 2002. Extreme embedded a report from the front line. OOPSLA 2002 Practitioners Reports. Seattle, Washington. Rasmussen, R., Hughes, T., Jenks, J. R. & Skach, J. 2009. Adopting Agile in an FDA Regulated Environment. In: Agile Conference, 2009. AGILE ’09, 151-155. Ronkainen, J. & Abrahamsson, P. 2003. Software development under stringent hardware constraints: do agile methods have a chance? In: Extreme Programming and Agile Processes in Software Engineering. 4th
321
[S15] [S16]
[S17] [S18]
[S19]
[S20] [S21]
International Conference, XP 2003. 25-29 May 2003. Berlin, Germany. Springer-Verlag, 73-9. Rottier, P. A. & Rodrigues, V. 2008. Agile Development in a Medical Device Company. In: Agile, 2008. AGILE '08. 218-223. Salo, O. & Abrahamsson, P. 2008. Agile methods in European embedded software development organisations: A survey on the actual use and usefulness of Extreme Programming and Scrum. IET Software 2008, 2, 58-64. Spence, J. W. 2005. There has to be a better way! In: AGILE Conference 2005, July 24 - July 29, 2005, Denver, CO, United states. Inst. of Elec. and Elec. Eng. Computer Society, 272-278. Van Schooenderwoert, N. 2008. Safety-Critical Applications Built via Agile Discipline.[Online] Bonston SPIN (Software Process Improvement Network). Available: http://www.bostonspin.org/slides/boston_spin_slides_2008_09.pdf. [Last accessed 17 June 2011] Van Schooenderwoert, N. & Morsicato, R. 2004. Taming the embedded tiger - Agile test techniques for embedded software. In: Proceedings of the Agile Development Conference, ADC 2004, June 22 - June 26, 2004, Salt Lake City, UT, United states. IEEE Computer Society, 120-126. Vanderleest, S. H. & Buter, A. 2009. Escape the waterfall: Agile for aerospace. In: Digital Avionics Systems Conference, 2009. DASC '09. IEEE/AIAA 28th, 6.D.3-1-6.D.3-16. Wils, A., Van Baelen, S., Holvoet, T. & De Vlaminck, K. 2006. Agility in the avionics software world. In: 7th International Conference on Extreme Programming and Agile Processes in Software Engineering, XP 2006, June 17- June 22, 2006, Oulu, Finland. Springer Verlag, 123-132.
322
Appendix D – Data Extraction Sheet (truncated version)
Figure D-65 Snapshot of the Data Extraction Sheet from the Mapping Study
323
Appendix E – Case Study Protocol Overview This study is intended to explore how Medical Device companies manufacture their products from a software development perspective. The aspect which makes this domain particularly interesting is that the entire development process, both hardware and software, is subject to tight regulatory controls. The demonstration of adherence to high quality and risk management procedures, together with a well defined and documented development process are some of the critical indicators the regulators look for. These regulations affect the way these companies develop their products, which is typically a multi-year lifecycle requiring significant investment. Anything which can help reduce either the duration or the cost, while maintaining regulatory compliance, would be extremely beneficial, especially during the tough economic times of the present day. Since its inception in the Toyota Production System (TPS), Lean manufacturing – which revolutionised the automobile manufacturing industry and saw the Japanese become one of the dominant forces in the industry worldwide - has been copied and extended around the world. The primary focus and guiding principle of ‘Lean’ is the identification and elimination of waste from the process, as opposed to a detailed and regimented list of activities that must be followed which is typical of other manufacturing improvement initiatives. By applying these Lean principles within the area of developing software (Lean Software Development), a different focus is achieved and the potential for more efficient and shorter product lifecycles is presented. The challenge, which this study is trying to address, is to see how suitable Lean Software Development is within the Medical Device domain.
Motivation Many of the largest Medical Device manufacturers in the world have a presence in Ireland. However, very little of the software that is required for the manufacture of these devices, is developed in Ireland. As mentioned above, cost could be seen as a deterrent to existing Medical Device companies developing the capability in Ireland, and the regulations could be seen as a deterrent to indigenous Irish software development companies entering the market. Our aim is to examine the possibility of applying Lean Software Development practices within this domain thereby achieving less costly development processes while ensuring regulatory compliance. If this is successful then perhaps we can develop a roadmap to entice new entrants into this
324
market segment and/or encourage existing Medical Device companies to invest in a Lean Software Development capability in Ireland.
Background There has been some previous research carried out on embedded software development which highlights the different challenges faced when the software is developed in conjunction with the hardware platform. Modern software development methodologies such as those described as Agile, can be seen as supportive practices of a Lean philosophy. Much research is also available on Agile software development practices, and methods of Agile adoption. However, as we found, there is very little published on the specific area of software development for regulated environments, and even less when this is refined to the medical device domain. Our initial understanding was that Lean or Agile approaches to developing software for Medical Devices would not suit the tight regulations that governed the domain. However, a literature review showed that there do exist a few reports on the success of implementing Agile practices for medical devices and also within similar safety-critical domains e.g. aviation. So our research focus is to investigate the applicability of Lean Software Development (as opposed to purely Agile development practices) within the domain and to what extent the regulations affect implementation.
Main Research Question With the above in mind, our main research question becomes: RQ - How applicable is a Lean Software Development methodology in the medical device manufacturing industry?
Case study Roles Primary researcher:
Oisín Cawley – Doctoral Researcher (University of Limerick)
Research Supervisors:
Dr. Ita. Richardson (University of Limerick) Dr. Xiaofeng Wang (University of Limerick)
Case study location:
---
Case study contact:
---
325
Design The approach to this research study is to perform case studies across multiple medical device companies. This comparative approach will give us rich detailed data (both quantitative and qualitative) and help us better understand the intricacies of developing software within such a regulatory context.
Propositions (P) and Assumptions (A) It is important to state the following propositions/assumptions up front prior to beginning the study. These have been formed from the literature review: -
A1: Lean Software Development leads to fewer defects, shorter development times, cheaper development costs and a more engaged workforce
-
P1: Medical Device regulatory controls force companies to perform tasks they would otherwise not perform and/or prevents companies from performing other tasks the way they would prefer
-
P2: Some form of Lean Software Development is still possible within such a regulated environment Although we state two propositions above, our aim is not to prove them in the positivist
sense, but rather use them to guide the case-study in an interpretative way.
Conceptual Framework Below is a depiction of the lean framework used for focusing the cases study. This is a typical depiction of what is called the “house of lean” as developed within the company Toyota. Its main features depict a roof, supported by two pillars which are standing on a solid foundation. By achieving a balance between the various features, a strong and steady structure is maintained. Within a SD context, the individual practices will obviously be different to those in a manufacturing environment, however, the underlying lean concepts, principles and values remain the same.
326
Figure E-66 The Lean Framework as a Focusing Tool
Detailed Research Questions To give more focus to our research the following sub-questions are provided which roll-up to answer the main research question: Within the context of Software Development for Medical Devices which are governed by various regulations: RQ1: How are existing medical device companies developing their software given the regulations they must adhere to? RQ2: What is currently known about the use of Lean Software Development practices within such companies? RQ3: How is the use of Lean Software Development practices within these companies being restricted? RQ4: How can such restrictions be circumvented in support of Lean methods without compromising regulatory compliance?
Unit of analysis The object of study is the software development process with some necessary focus on the software development practices. By examining existing processes and practices within a Lean frame of reference, we aim to gain some insight into the main research question above. The main RQ is admittedly quite broad and encompasses the complete software development
327
lifecycle which will need to be focused as we progress. So, as with any research undertaking, we must be able to be flexible in terms of where the research takes us, and because we are interested in the real-life context and the issues experienced by industry, we fully expect that some useful direction will be obtained in the initial case study. We therefore state that: -
The initial case study may suggest a change of focus depending on whether something particularly interesting comes to light or indeed is highlighted by the company as a significant issue which needs addressing in this context.
-
The focus of the initial case study may also change, for example, if it becomes apparent that a specific project should become the focal point of our enquiry.
Case Selection In choosing the cases for this study is important that the companies are operating within the appropriate context i.e. they are developing software which is subject to the relevant medical device regulations. Company size is not particularly relevant, however we aim to include different size organisation in order to identify any issues specific to team size/dispersion.
Data collection The data gathered will be of a quantitative and qualitative nature. Such a mixed method approach allows the convergence of information from different sources and helps build a stronger study than either on their own. The quantitative data will be in the form of: -
Survey results from practitioners
-
Metrics such as: o
Overall project durations
o
Project stage durations
o
Number of project team members
o
Change request lifecycle (value chain mapping)
-
Development methodology used
-
Documentation o
Corporate and departmental policies/procedures
o
Progress reports, agendas, announcements, meeting minutes
o
E-mail correspondence, news items
-
Archival records
-
Physical artefacts
328
-
Outsource/offshore arrangements The qualitative data will be gathered through semi-structured interviews framed around a
pre-defined set of precise and exploratory questions. The interviewees will be from the different departments involved in a typical software development project, but typically consist of the following stakeholders: -
Developers (may have varied roles e.g. coding, testing etc.)
-
Quality Engineers
-
End-users
With permission of the interviewees, the interviews will be recorded for data analysis.
Analysis The intention is use a grounded theory approach to the analysis of the interview data. By using such an approach we aim to find common key themes/attributes/issues appearing. This will help us formulate theory around the development process within the case. By performing this after each interview, it enables us to refine the questions for, or change direction in, the follow-on interviews. In this manner, we incrementally build a more accurate view of the case. The information gathered from other sources will then be used to form a convergent view of the data. This triangulation will help develop a stronger case study analysis.
Plan Validity We use the following five, somewhat overlapping, attributes by which the study quality should be judged. We include some guiding key questions (KQ) to ask in determining the strength of each attribute:
Objectivity/Confirmability KQ - How free is the study from the influence and biases of the researcher? KQ – Is all the data presented in detail and is the study data readily available?
Reliability/Dependability/Auditability KQ - Have things been done with reasonable care?
Internal Validity/Credibility/Authenticity KQ - Do the findings of the study make sense?
329
KQ - Does the story seem convincing and did the original informants think our findings were accurate?
External Validity/Transferability/Fittingness KQ - Are the conclusions transferable to other contexts? KQ - Are the original sample of peoples and variables described enough to permit comparisons?
Utilisation/Application/Action Orientation KQ - Are the findings intellectually and physically accessible to potential users?
Reporting The intended audience of the results of this cases study will be: a.
Interested parties within the case study organisation a.
This will take the format of report of the findings with possible recommendations of how Lean practices might be adopted for specific purposes
b.
Academicians a.
Peer reviewers from any publications which result
b.
Academic examiners from a PhD perspective
Administrative Concerns The following general points will be of interest to various parties:
Ethics -
The research proposal has been submitted to and approved by the Science and Engineering Ethics committee of the University of Limerick.
-
Each participant will be provided with a participant’s information sheet and an informed consent form prior to interview.
Legal -
A non-disclosure-agreement (NDA) is in the process of being drafted between ... and Lero.
-
The confidentiality of all company specific data gathered or reviewed, and identity of participants will be ensured by the research team at all times.
330
Schedule -
The researchers are conscious of the busy schedule the participants have, and will endeavour to be punctual, stick to agreed timeslots and generally be as unobtrusive as possible.
-
The interview timeslots have been provided by ... and accepted by the lead researcher
Publication -
The research study may lead to the publication of academic papers, however, any such publications which identify the company will be presented to the company in advance for review.
Appendices N/A
331
Appendix F – Participants’ Information Sheet 1. Title of study:
Lean Software Development within the regulated medical device manufacturing industry.
2. Introduction:
The purpose of this research project is to examine the suitability/applicability of Lean Software Development to the software development lifecycle within the medical device industry. This industry is heavily regulated by such bodies as the Food and Drugs Administration (FDA) in the US, and in the EU under the Medical Device Directive (MDD), and it is within this context that we are specifically interested. The study will involve researching the body of knowledge (academic and nonacademic), an examination of the current practices within ... and the relevant regulatory controls to which ... must conform.
3. Procedures:
To gain the most complete understanding of the above, we will be interviewing ... employees to solicit their input on ...’s current practices and also solicit feedback to the ongoing research at various stages of its progression. We plan to do this by means of 1 to 1.5 hour semi-structured interviews. The interviews will be audiotaped by the researcher and later transcribed for the purpose of data analysis. The interviews will be conducted at a setting that is mutually agreeable to the participant and the researcher. All participants’ identities will remain confidential and all input received will be used for the sole purpose of progressing the research. The criteria for participant selection is that they should be actively engaged in the process of project managing, designing, creating and/or configuring the manufacturing lines/software/equipment for medical devices within the ... plant, for example: software engineers, process engineers, technicians, quality controllers, end-users. Participants will be sought by ... management. Participants may be asked to modify their work practices, for example stop performing certain tasks, start doing new tasks, change the order of tasks, communicate differently, change desk location.
4. Benefits:
The intention of this research is to develop a software development framework based around the principles of Lean Software Development. The participants will become familiar with these concepts and will contribute to developing work practices that are more value adding, less wasteful and consequently more cost effective to the company.
5. Risks:
We do not foresee any physical risks, however it may be useful to trial changes to some work practices and/or inter-department collaboration in a manner participants currently do not perform. Risks will be discussed with ... management.
332
6. Exclusion from participation:
You cannot participate in this study if any of the following are true: Not actively engaged in the process of project managing, designing, creating and/or configuring the manufacturing lines/software/equipment for medical devices within the ... plant.
7. Confidentiality:
Your identity will remain confidential. Your name will not be published and will not be disclosed to anyone outside the study group.
8. Participation:
Participation in the study by individuals will be agreed with ... management.
9. Withdrawal without Prejudice
Each participant is free to withdraw consent and discontinue participation in this project at any time without prejudice.
Permission:
This research project has been approved by the Faculty of Science and Engineering Ethics Committee at the University of Limerick, and is being supervised by Dr. Ita Richardson and Dr. Xiaofeng Wang from Lero - the Irish Software Engineering Research Centre, Limerick University.
Further information:
You can get more information or answers to your questions about the study, your participation in the study, and your rights, from lead researcher Oisín Cawley (
[email protected]). If the study team learns of important new information that might affect your desire to remain in the study, ... management will be informed at once.
333
Appendix G – Case Study Interview Guide Interview Context IEC 62304:2006 – Medical Device Software life-cycle processes -
-
Software Development Process o
Planning
o
Analysis
o
Architectural Design
o
Detailed Design
o
Unit Implementation & Verification
o
Integration & Integration Testing
o
System Testing
o
Release
Types of Waste o
Over Production (Unnecessary Code)
o
Waiting (Delays)
o
Unnecessary Transport (Handoffs)
o
Over-processing or Incorrect Processing
o
Excess Inventory (Partially completed work)
o
Unnecessary Movement (Task switching/Travelling)
o
Defects (Bugs)
o
Unused employee Creativity
Introduction 1. What is your role? 2. How long have you been working? a.
How long in a medical device context?
3. What are the typical tasks you perform? 4. Has the software development process changed in recent years? 5. When working on a project what groups do you interact act with and how/how often? 6. How conscious are you of the Medical Device regulations a. Any difference between US (FDA) and EU (MDD)?
334
b. How does it manifest itself in your daily work? 7. How would your work practices be different if you were not working in a regulated environment? 8. For software engineers: a. How is your work organised: i. Who assigns your tasks? ii. How many tasks might you be working on concurrently? iii. Do you get pulled away to work on “urgent” items? If so how often? b. What types of software defects are typical? c. How often do your requirements change over a project’s lifetime? d. Do you spend time gathering data/generating reports to comply with regulations? If so how much time? e. What sort of tool support do you rely on? f.
How do you test your code?
g. How do you validate and verify your code? h. Who is your customer? i.
How much interaction do you have with the customer during a project?
j.
What areas of expertise do you think your group is lacking/low in?
k. When working with embedded software: i. How much interaction is there with the hardware teams? ii. How do hardware issues affect the software development process? iii. How could the HW/SW groups’ integration be improved? 9. How does the software development process fit in with the various Lean initiatives ongoing throughout the company? 10. With regard to the current processes: a. In your opinion what processes work well? b. In your opinion what processes do not work well? c. Is there anything you would change, bearing in mind the regulations?
335
Appendix H – Industry Questionnaire Questionnaire – Lean Software Development for Medical Devices The questions below are set in the context of developing medical device software and consequentially all the ramifications of having to adhere to the relevant regulations should be kept in mind. 1. Do you see any problems with taking an iterative approach to developing medical device software (as opposed to the typical waterfall or V models)? 2. A lean approach would be to project plan “just enough” upfront followed by incremental planning for each iteration. What should be kept in mind if taking this approach? 3. Requirements specified as user stores would initially be high level and expanded on once they are prioritised for an iteration. This would lower the overhead up front and ensure high value requirements are done first. Any concerns with such an approach? 4. Each iteration, apart from developing code, would include many activities aimed at incrementally fulfilling the regulatory requirements for requirement specification, traceability, verification, validation, risk analysis etc. Where might the regulators have concerns here? 5. From a validation point of view, the project plan would designate specific iterations (not all) to be fully validated, with a full system validation at the end. Would you see any problems with this? 6. Eliminate waste (bugs) by taking a test driven development approach. Any concerns? 7. What would concern you with taking an automated testing and integration approach? 8. A lean approach to development would be the use of pair-programming to support knowledge sharing and which would also support continuous review. Would you have any concerns with this? 9. Refactoring is the process of reviewing and improving existing code. Some concern has been expressed in relation to this practice introducing bugs into other, dependent parts of the code base. What are your thoughts? 10. Eliminating delays is a core lean practice. Within the software development context such things as face-to-face communication, close and easy access to the customer, hardware abstraction etc. can be used to make the process flow better. What would you suggest are things which need to be considered in this domain which would affect such an approach?
336
Appendix I – The SaLeS-4-MD Model Detail Table I-38 The SaLeS-4-MD Model Detail
Safe and Lean Software for Medical Devices (SaLeS-4-MD)
1
Medi-SPICE
2
3
MediSPICE Activity
4
5.5
5
ENG.5.5. SP1
Lean Practices
Requir Potenti Specific ed for Lean Lean al Issue Practice Device Princip Practic Descriptio in MD Mitigatio Sourc Description Class le e n Domain n e The purpose of the Software Unit Implementation and Verification Activity is to produce verified software units that properly reflect the software design. R&D phase not Regulat governed or may by the seek regulations some Eliminat Conside , but clearly high e Waste r if state when level High level Define and (Avoid perform your R&D indicati verificatio establish a unit Overing an phase on of n report verification Processi R&D starts and verificat is Case strategy B,C ng) phase ends. ion prudent Study
337
Eliminat e Waste (Avoid OverProcessi ng)
6 SP1.1 Where verification is done by testing Test procedures shall be evaluated for correctness
7
8
9
ENG.5.5. SP2
Define and establish software unit verification criteria
B,C
Decide on level of concern
Different LOCs require different levels of detail
There is risk involve d in doing too little
Choose the level of risk/conc ern you are comforta ble with and verify according ly
Just enough docume ntation
Don't write documents that nobody reads. Try and automate the documenta tion process.
Differen t stakehol ders may require differen t levels of docume ntation
Strike a balance
Rassm ussen et al 2009
Case Study
Hibbs et al 2009
Lin & Fan 2009
TBD
TBD
B,C
Eliminat e Waste (Avoid OverProcessi ng)
338
Case Study
Reflective Analysis
10
ENG.5.5. SP3
Define and establish software unit acceptance criteria
B,C
Eliminat e Waste (Avoid OverProcessi ng)
Re-use existing artefact s
User stories become the requireme nts but should be expanded on during the iteration (see ROW 11)
11
12 13 14
SP3.1 Additional Acceptance Criteria When present in the design -Proper event sequence
C
TBD TBD TBD
339
May not have enough detail for regulato r
Include clear condition s of satisfactio n Each iteration gathers extra detail for its user stories. Care should be given to ensure this is at a software unit level.
Shoem aker & Van Shooe nderw oert 2010
Shoem aker & Van Shooe nderw oert 2010
-Data and control flow -Planned resource allocation -Fault handling (error definition, isolation, and recovery) -Initialization of variables
15 16
17 18 19
21
23
TBD
TBD TBD
-Self-diagnostics -Memory management and memory overflows -Boundary conditions
20
22
TBD
ENG. 5.5.SP4
Develop software units
TBD
TBD
A,B,C
Eliminat e Waste (Avoid OverProduct ion)
Only develop for current iteratio n
Concentrat e only on what is required for this iteration Regulat ory focus will be on V&V activitie s
Develop software units
340
Perform V&V during and after each iteration
Hibbs et al 2009
Reflect ive Analys is
Rassm ussen et al 2009
Reflect ive Analys is
24
Develop software units
Develop software units
Deliver Fast
Short iteratio ns
26
Develop software units
Create Knowle dge
Build a prototy pe
Use short iterations to get feedback quickly Allow customer to 'play with' prototype so they can refine requireme nts
27
Develop software units
Continu ous
Don't overloa
Perform load
25
341
Problem s may arise when working with traditio nal teams who want an all or nothing approac h Typical agile iteratio n lengths may be too short
Define common skeleton interfaces which can be worked on in pieces
Drobk a et al 2004
Use iteration lengths appropria te to your situation
Rottier and Rodrig ues 2008
Rassm ussen et al 2009
Case Study
Reflect ive Analys is
Case Study
Reflect ive
Case Study
Reflective Analysis
Flow
28
29
30
Develop software units
Build Quality In
d team member s
Re-use existing artefact s
Develop software units
Develop software units
Build Quality In
Take an object oriented approac h
levelling (Heijunka) Re-use existing code/comp onents/OT S application s speeds up the developme nt process. Keep in mind importance of Risk Analysis (see ROWS 97 & 98) Builds confidence in the new functionalit y Component isation, modularisa tion, parameteri sation hides away complexity and limits
342
Analys is
Tenden cy to lightly test or not test at all
Include as part of validation plan
Vogel 2010
Graaf et al 2003
Case Study
Reflect ive Analys is
Poppe ndeick & Poppe ndieck 2003
Case Study
Case Study
Reflecti ve Analysis
Reflective Analysis
the effects of changes
31
32
Assists in a Distributed Software Developme nt configurati on
Develop software units
Develop software units
Build Quality In
Maintai n Code Clarity
Use an intuitive naming convention
33
Develop software units
Use a common language
34
Develop software units
Use simple notation
343
Case Study Poppe ndeick & Poppe ndieck 2003 Poppe ndeick & Poppe ndieck 2003 Poppe ndeick & Poppe ndieck 2003
Reflect ive Analys is
Hibbs et al 2009
Jeffries et al 2001
Case Study
Reflective Analysis
Hibbs et al 2009
Jeffries et al 2001
Case Study
Reflective Analysis
Hibbs et al 2009
Jeffries et al 2001
Case Study
Reflective Analysis
35
36
37
38
Make incode comments clear and focused
Develop software units
Build Quality In
Use a good source code control tool
Develop software units
Create Knowle dge
Use a good source code control tool
Develop software units
Create Knowle dge
In-code docume ntation
Develop software units
Maintain source code integrity Daily check in and out so developers all have latest changes. Decide if only tested and integration ready code should be checked in. Put explanator y low-level detail in the source code itself as opposed to a written
344
Poppe ndeick & Poppe ndieck 2003
Hibbs et al 2009
Case Study
Reflect ive Analys is
Hibbs et al 2009
Poppe ndeick & Poppe ndieck 2003
Vogel 2010
Reflect ive Analys is
Jeffries et al 2001
Case Study
Reflective Analysis
Ambler & Kroll 2007
Case Study
Reflective Analysis
document
39
40
41
Develop software units
Try and organise code to resemble the SRS and SDS documents For less complex systems, deliverable s can be grouped into a single document
Develop software units
Test driven developme nt will support finding defects early
Develop software units
Eliminat e Waste (Avoid OverProcessi ng)
Eliminat e Waste (Elimin ate Defects)
Docume ntation groupin g
Use a TDD approac h
345
Vogel 2010
Case Study
Reflect ive Analys is
Hibbs et al 2009
Poppe ndeick & Poppe ndieck 2003
Cordeir o et al 2007
Kane et al 2006
Van Schooende rwoert & Morsicato 2004
42
43
44
45
Use mocks, stubs or comment out h/w calls
Hibbs et al 2009
Cordei ro et al 2007
Kane et al 2006
Develop software units
Use a simulator
Mastic olla & Gall 2008
Muelle r and Borzuc howsk i 2002
Gary et al 2011
Develop software units
Use a h/w abstractio n layer
Mastic olla & Gall 2008
Have a formal document ation process
Lin & Fan 2009
Hardwa re may be unavaila ble
Develop software units
Develop software units
Develop a test as soon as a defect is found
346
Defects must be traced back to require ments
Van Schooend erwoert & Morsicato 2004
Eliminat e Waste (Avoid OverProduct ion)
Periodicall y reviewing and improving existing code. Carefull attention should be given to the potential issues listed below (ROWs 4753)
Refactor ing in later iteratio ns may cause regulato ry issues
Limit refactorin g to early iterations
46
Develop software units
47
Develop software units
Constant testing
Wils et al 2006 Chisho lm 2007
Develop software units
Perform pair program ming
Chisho lm 2007
Develop software units
Use a SCM system which includes the hardware version
Ronkai nen and Abrah amsso n 2003
48
49
Refactor Code
Potentia lly hazardo us to hardwa re
347
Reflect ive Analys is
50
Develop software units
51
Develop software units
52
53
Employ system level impact analysis methods
Develop software units
Develop software units
348
Changes may break existing code
Relentles s testing Create a source code branch and integrate only when all tests pass
Legacy code may pose a problem
Add build scripts and unit tests as modificati ons are requested
Ronkai nen and Abrah amsso n 2003 Ronkai nen and Abrah amsso n 2003
Cordei ro et al 2007
Hibbs et al 2009
54
55
56
57
Develop software units
Eliminat e Waste (Elimin ate Defects)
Explore the use of BDD
Develop software units
Eliminat e Waste (Elimin ate Defects)
Continu ous Softwar e Integrat ion
Develop software units
Develop software units
Behaviour driven developme nt helps eliminate misunderst andings between stakeholde rs Continually integrating small changes catches issues sooner Avoids the "Big Bang" approach to integration
Continually integrating small changes catches issues sooner
349
Tool support may be poor
May require new softwar e/script s that will require validati on
Revert to TDD
Scripts should be controlle d in a SCM system
Hibbs et al 2009
Hibbs et al 2009
Gary et al 2011
Rassm ussen et al 2009
Hibbs et al 2009
Hibbs et al 2009
58
59
60
61
May require extra hardwa re for building the softwar e
Develop software units
Develop software units
Eliminat e Waste (Elimin ate Defects)
Frequen t S/W and H/W integrati on
Make the appropria te investme nt if required
Provides early feedback to the hardware team
Rassm ussen et al 2009 Regulat or will still insist on a full final integrati on Legacy code can be problem atic
Develop software units
Develop software units
350
Hibbs et al 2009
Final verificatio n much faster and less troubleso me
Reflect ive Analys is
Rottier and Rodrig ues 2008 Graaf et al 2003
Rottie r and Rodrig ues 2008
Develop software units
62
63
Develop software units
64
Ensure consistency and bilateral traceability to system and software requirements
65
Create Knowle dge
ENG. 5.5.SP5
A,B,C
Create Knowle dge Build Quality In / Eliminat e Waste (Avoid OverProcessi ng) Eliminat e Waste (Avoid OverProcessi ng)
Use a visual display
Intercha nge Develop er Rolls
Use similar docume ntation structur e Docume ntation groupin g
Open display of work in process informs everyone of status & issues Developers should interchang e rolls in order to increase broad expertise Documents that resemble each other are easier to review and relate Map sections instead of every single detail
351
Critical areas should be assigne d to experie nced develop ers
Could be seen as cutting corners
Assign experienc ed developer s to critical areas
Don't make sections too big
Case Study
Reflect ive Analys is
Case Study
Reflect ive Analys is
Vogel 2010
Drobk a et al 2004
Case Study
Vogel 2010
Case Study
Egyed et al 2009
MediSPICE Assessme nt 1
Reflective Analysis
66
ENG. 5.5.SP6
Ensure consistency and bilateral traceability to detailed design
A,B,C
67
68
69
ENG. 5.5.SP7
Ensure consistency and bilateral traceability to test specification of software units Ensure consistency and bilateral traceability to test specification of software units
A,B,C
Build Quality In / Eliminat e Waste (Avoid OverProcessi ng) Eliminat e Waste (Avoid OverProcessi ng) Build Quality In / Eliminat e Waste (Avoid OverProcessi ng) Eliminat e Waste (Avoid OverProcessi ng)
Docume ntation groupin g
Documents that resemble each other are easier to review and relate Map sections instead of every single detail
Use similar docume ntation structur e
Documents that resemble each other are easier to review and relate
Docume ntation groupin g
Map sections instead of every single detail
Use similar docume ntation structur e
352
Could be seen as cutting corners
Could be seen as cutting corners
Don't make sections too big
Don't make sections too big
Vogel 2010
Drobk a et al 2004
Case Study
Vogel 2010
Case Study
Egyed et al 2009
Vogel 2010
Drobk a et al 2004
Case Study
Vogel 2010
Case Study
Egyed et al 2009
MediSPICE Assessme nt 1
Reflective Analysis
MediSPICE Assessme nt 1
Reflective Analysis
Ensure consistency and bilateral traceability to test specification of software units Ensure consistency and bilateral traceability to test specification of software units
70
71
72
73
ENG. 5.5.SP8
Verify software units
Verify software units
Eliminat e Waste (Avoid OverProcessi ng)
B,C
Eliminat e Waste (Avoid OverProcessi ng)
Build on a TDD approac h
Add references to automated tests
Huffm an Hayes et al 2009
Conside r if perform ing an R&D phase
R&D phase not governed by the regulations
Refactor ing may create or break links Regulat or may seek some high level indicati on of verificat ion
Decide on level of concern
Different LOCs require different levels of detail
There is risk involve d in doing too little
353
Failing tests should trigger an update of the RTM
High level verificatio n report is prudent Choose the level of risk/conc ern you are comforta ble with and verify according
Huffm an Hayes et al 2009
Case Study
Rassm ussen et al 2009
Case Study
ly
74
75
76
Verify software units
Eliminat e Waste (Elimin ate Defects)
Verify software units
Verify software units
Build Quality In
Use PokaYoke Employ formal model checkin g techniq ues
Use state machine s
Employ defect prevention and detection techniques Models can assist in the verification of correctness of software State machines support safety-bydesign, by restricting the allowable behaviours of a component
354
Robins on 1997
Reflect ive Analys is
Models may not have been created during design
Reverse engineer models from the source code
Tsai et al 2005
Jetley et al 2006
All possible compon ent states should be docume nted
Docuemn t and review the statemachine catalog
Gary et al 2011
Reflect ive Analys is
Gary et al 2011
77
78
79
80
Verify software units
Use Standar ds
Use coding standards and guidelines Standar ds are not enforce d
Verify software units
Verify software units
Root cause analysis
Incorpora te check of standards into code review process
Ensure issues are fully understood by using techniques such as the 5 whys approach
355
Hibbs et al 2009
Case Study
Case Study
Vogel 2010
Reflecti ve Analysis
Poppe ndeick & Poppe ndieck 2003 Not rigorous enough. Impact of issues needs to be rigorous ly assesse d.
Verify software units
Poppe ndeick & Poppe ndieck 2003
Conduct proper risk analysis as required. See SP10 and SP11.
Case Study
Reflective Analysis
81
82
Verify software units
Re-use existing artefact s
Verify software units
Use experie nced develop ers for critical areas
83
Verify software units
84
Verify software units
Perform automat ed testing Use a testing framew ork such as Xunit
Re-use existing code/comp onents/OT S application s
Use experience d developers for critical areas Supports GPSV's guidelines of "consistent ly fulfilling" requireme nts Use a testing framework such as Xunit
356
Tenden cy to lightly test or not test at all Potentia l to develop resourc e bottlene cks if many critical areas
Hardwa re may be unavaila ble
Include as part of validation plan
Vogel 2010
Graaf et al 2003
Case Study
Reflective Analysis
Build up developer s' experienc e
Poppe ndeick & Poppe ndieck 2003
Case Study
Reflecti ve Analysis
Reflective Analysis
Cordei ro et al 2007
Hibbs et al 2009
Shoema ker & Van Shooen derwoer t 2010
Jeffries et al 2001
Hibbs et al 2009
See SP4 (TDD approach )
Vogel 2010
Poppendei ck & Poppendie ck 2003
85
86
UI testing can be difficult to automat e
Verify software units
Verify software units
Maintai n Clarity
Use an intuitive naming convention
87
Verify software units
Use a common language
88
Verify software units
Use simple notation
Verify software units
Make incode comments clear and focused
89
357
Separate frontend from backend and use tool support
Jeffries et al 2001 Poppe ndeick & Poppe ndieck 2003 Poppe ndeick & Poppe ndieck 2003 Poppe ndeick & Poppe ndieck 2003 Poppe ndeick & Poppe ndieck 2003
Hibbs et al 2009
Hibbs et al 2009
Jeffries et al 2001
Case Study
Reflective Analysis
Hibbs et al 2009
Jeffries et al 2001
Case Study
Reflective Analysis
Hibbs et al 2009
Jeffries et al 2001
Case Study
Reflective Analysis
Hibbs et al 2009
Jeffries et al 2001
Case Study
Reflective Analysis
90
91
92
93
Don't overdocume nt
Verify software units
Verify software units
Build Quality In & Create Knowle dge
Pair Progra mming
Keep maintenan ce documenta tion to a minimum Use pair programmi ng as a means of continuous review
Verify software units
Verify software units
Integrat ed Problem solving
Use cross functional teams to solve problems
358
Product s have long life times, so system needs to be support able by new people Can be difficult to pick pairs Familiar ity breeds lax practice s
Have non-team members evaluate the document ation's adequacy
Good oversight can make this work well
Drobk a et al 2004
Vogel 2010
Reflecti ve Analysis
Drobk a et al 2004
Levy & Hazza n 2009
Beck 1999
Vogel 2010 Poppe ndeick & Poppe ndieck 2003
Raymo nd 2001
Case Study
MediSPICE Assessme nt 1
Reflective Analysis
94
ENG. 5.5.SP9
Document software unit verification results
B,C
95
Create Knowle dge Eliminat e Waste (Avoid OverProcessi ng)
96
97
98
ENG. 5.5.SP10
Conduct risk analysis
A,B,C
Eliminat e Waste (Avoid OverProduct ion)
Don't overdocume nt Use predefined templat es Use automat ed techniq ues
Re-use existing artefact s
Standardis e documenta tion to facilitate ease of commun ication Eases capture of verification results Automated test results can be automatica lly logged Use risk analysis from a similar device as a starting point Differen ces betwee n devices
Conduct risk analysis
359
Perform systemati c evaluatio n of difference s
Drobk a et al 2004
Case Study
Case Study
Reflect ive Analys is
Gary et al 2011
Reflect ive Analys is
ISO 14971: 2007 4.1 Note 1
Burton , John PhD Thesis 2008
ISO 14971: 2007 4.1 Note 1
Burton , John PhD Thesis 2008
Reflecti ve Analysis
MediSPICE Assessm ent 1
99
100
101
102
Decide on level of concern
Conduct risk analysis
Less rigour required for "low" LOC Choosig n the wrong LOC
Conduct risk analysis
Conduct risk analysis
Build Quality In
Use establis hed risk analysis techniq ues
Use established risk analysis techniques
Conduct risk analysis
103
Conduct risk analysis
104
Conduct risk analysis
Case Study
360
Analysis techniq ues need to be rigorous
Have a clear process for deciding Use Failure Mode and Effects Analysis (FMEA) Perform a System FMEA only for high or moderate LOCs Use Failure Mode, Effects and Criticality Analysis (FMECA) Use Fault Tree Analysis (FTA)
Case Study
Feldm an et al 2007
Case Study
Case Study
Ronkai nen and Abrah amsso n 2003
Shoem aker & Van Shooe nderw oert 2010 Feldm an et al 2007
MediSPICE Assessm ent 1
105
106
107
108
Conduct risk analysis
Perform Risk Review
Review and revise risk/hazar d analysis Mitigati ons need to be fully traceabl e
Conduct risk analysis
Conduct risk analysis
Conduct risk analysis
Use Cross Functio nal Teams
Assists by getting perspective s from multiple stakeholde rs
Use predefined templat es
Eases capture of risk data and assists in traceability
361
Create "negative " user stories
Shoem aker & Van Shooe nderw oert 2010 Shoem aker & Van Shooe nderw oert 2010
Case Study
Reflect ive Analys is
Case Study
Reflect ive Analys is
109
Conduct risk analysis
Use a docume ntation manage ment tool
110
Conduct risk evaluation and risk control activities
Use Cross Functio nal Teams
ENG. 5.5.SP11
B,C
Build Quality In
Used to secure documents and automate workflow, including remote collaborati on Cross functional teams assists in evaluating risk from various perspective s (ISO 14971 pg 40)
Mitigati ons need to be fully traceabl e
111
362
Add or change relevant document ation, including the risk managem ent file. Record who attends meetings
Reflect ive Analys is
Case Study
Case Study
Reflect ive Analys is
Shoem aker & Van Shooe nderw oert 2010
Case Study
Appendix J – The Industry Report
Image removed..
Lean/Agile Software Development Processes in a Regulated Industry Oisín Cawley January 2010
Short Description This document is a report into the practices of Lean/Agile Software Development methodologies. Its specific focus is to discover how relevant these methodologies are to the domain of medical device manufacturing. This is achieved by reviewing as much as possible of the academic literature which has reported on this subject and which includes many case studies and industry reports. To augment this, material from relevant industry experts has been included. A list of specific questions was agreed with ..., and an attempt is made to answer them in the later part of this document. This report is the first stage of the research project : “Lean/Agile Software Development within the regulated medical device manufacturing industry”. Following on from this it is intended to review the software development practices and procedures within ..., together with the relevant regulatory documentation, to produce
363
a suggested mapping onto appropriate Lean/Agile software development practices. The intention is to help ... reap the benefits of becoming Lean and Agile from a software development perspective, while ensuring any new practices support the relevant regulations.
Introduction Since its inception in the Toyota Production System around the 1940s (some of the concepts were actually used prior to this by the Ford Automobile company)8, Lean manufacturing has been copied and extended around the world. This has not been confined to manufacturing processes either but has been applied in other disciplines too. In particular I examined what has been published on the practice of applying Lean concepts to software development and in particular within a Medical Device context. There was limited published literature available on this exact area and so I widened my search criteria to include other closely related domains which involve embedded-software development. These included the Automotive, Aviation and Aerospace industries among others. At this point it is worthwhile to very briefly restate the core principles of ‘Leanness’. The Lean concept is more a way of thinking or a mental approach you take to a particular business process or set of processes. The primary focus is on the identification and elimination of waste from the process, as opposed to a regimented list of activities that must be followed. There are five main modern Lean principles: Value: It is defined by the customer and it is paramount to have a clear understanding of what that is; Value Stream: A map that identifies every step in the process and categorises each step in terms of the value it adds; Flow: It is important that the production process flows continuously; Pull: Customer orders pull product, ensuring nothing is built before it is needed; Perfection: You strive for perfection in your process by continuously identifying and removing waste. So, an obvious question to ask is how do Lean manufacturing practices, which routinely produce the same product as output, relate to software development activities which always produce something different? This I think is where Lean concepts meet Agile software development practices. is.
For completeness let’s look very briefly at what ‘Agile software development’
8
http://www.leanmanufacturingconcepts.com/HistoryOfLeanManufacturing.htm
364
Agile software development is seen as an attempt to address, what some people see as, the weaknesses (lack of flexibility, failure to meet the customer’s expectations, late product delivery) in the traditional plan-based or ‘waterfall’ type approach to software development. To combat these, practices such as closer collaboration between developers and end-users throughout the complete development process is something that Agilists would say can provide assistance in ensuring the end product matches the usability and functional requirements of the users (Dittrich and Lindeberg, 2004). In 2001, the various advocates of alternatives to the traditional waterfall approach met and agreed to an umbrella term called ‘Agile’ and to a set of underlying values and principles, defined in their Manifesto9. There are many Agile methods in existence today such as: XP (eXtreme Programming), Scrum, Crystal, Feature Driven Development, Dynamic Systems Development Method, Rational Unified Process, and each has a different set of specific practices. Some are very prescriptive in terms of the software development activities that should be performed e.g. pair programming in XP, while others have a different focus, for example Scrum mainly looks at project management type activities. Academic literature has numerous examples of successful Agile implementations (Dingsoyr and Dyba, 2009), but indications show there is a lack of empirical support of the success of such methods. This is not to say that the reports are not true, and in general, it is accepted that Agile methods when implemented appropriately do deliver better quality software in shorter timeframes. Agile Methodologies are not without their detractors. There has been much debate over the years about the suitability of Agile methods in certain circumstances. (Abrahamsson et al., 2003) performed a good comparative study of many of the Agile methods, and found a majority had little support for true project management. Some reports have indicated that Agile methodologies such as XP do not scale to larger projects, while on the other hand some have reported success in rolling out a common set of Agile practices across multiple teams (Nielsen and McMunn, 2005) or even an entire organisation (Ryan and Scudiere, 2008). (Ambler, 2008) even provides a walkthrough of one such scaling strategy. Further, dissidents point to issues such as: project cost estimating when dealing with an iterative process, the “wanton” disregard for thorough documentation, and uncontrolled developers, as fundamental weaknesses of the Agile approach. (Longstreet, 2007) goes so far as to suggest that: “In the not too distant future the software development industry will wake up and realize the move to Agile Methods was a mistake” However, there is no doubt that Agile methods are still gaining in popularity. A 2008 internet survey found that 69% of organisations are using Agile approaches in some form or another (Ambler, 2008). The method of adoption can be quite varied, with some companies jumping right in with a big-bang approach and others, which seems to be more usual and indeed advisable, 9
http://agilemanifesto.org/
365
taking a slow and gradual approach (Aveling, 2004). Particularly in the modern business landscape where product life-cycles are getting shorter, competition fierce and cost pressures high, the promise of shorter development timelines, fewer quality issues and a happy customer at the end continue to fuel this movement. To the un-initiated, it can be difficult to decide which Agile method might be the best one to adopt. I think there is an inherent problem with that quest because there is no quick answer, “It depends”. What makes one Agile method better than another depends on the context in which you want to implement it. Indeed an important question to ask would be whether all the different Agile methods are simply variations of each other, or if they are all converging towards a common method. Interestingly, (Fraser et al., 2006) solicited the opinion on this from some key Agile practitioners, including Alistair Cockburn (inventor of the Crystal Agile method), and the general feedback can be summarised in two points: 1. No agile method is implemented “out-of –the-box”. Some tailoring always happens. 2. Agile methods are still evolving as enhancements are needed to cover new domains and circumstances.
Lean or Agile or Both?
There seems to be much debate around the distinction between Lean and Agile, with some Agile practitioners saying there is no difference, and Lean software development practitioners viewing Agile methods as supportingpractices of the Lean philosophy10. Alan Kelly11 (an Agile and Lean software development consultant) positions many of the relevant practices relative to each other in Figure J-1 below:
10
http://www.infoq.com/news/2008/09/Not-Agile-Vs-Lean
11
http://allankelly.blogspot.com/2009/02/agile-and-lean-same-but-different.html
366
Figure J-67 Software Development Philosophies and Methods
For the purpose of this report two things are very interesting in this representation. First, Agile (the philosophy) and consequently Agile methods are depicted as focused supporting elements of Lean thinking. Second, we see the addition of the ‘Kanban’ methodology, referred to as “the new kid on the Agile block”. It is purposefully shown to straddle the Lean and Agile philosophies but also extends into the Agile methods level indicating that it contains specific practices. It will be worthwhile taking a closer look at Kanban software development later, as it seems to have a more visibly Lean origin that other Agile methods and also breaks from the hallmark fixed-iteration practice of typical Agile methods. Some of the early proponents of “Lean software Development” are Mary and Tom Poppendieck, and in their book (Poppendieck and Poppendieck, 2003), they mapped the core Lean principles to software development practices. These Lean principles are: 1. Eliminate Waste 2. Build Quality In 3. Create Knowledge 4. Defer Commitment 5. Deliver Fast 6. Respect People 7. Optimise the Whole
367
Many of the Agile practices support these principles. However, the key difference observed when compiling this report is between the focuses of Lean and Agile. Agile practices are mainly concerned with delivering working software as quickly as possible, while Lean’s primary focus is on eliminating waste in the context of customer value. (Hibbs et al., 2009) suggest that a good starting point if pursuing a Lean software development direction would be to pick and implement an Agile methodology and apply other Lean tools from there. They would recommend SCRUM as it seems to be the easiest to adopt. As a point of interest, (Kettunen, 2009) takes a close look at how much commonality there is between the concepts of ‘Agile Manufacturing’ and Agile Software Development. He draws our attention to the fact that there is no one universally agreed definition of either. However, from a business perspective, it is important that both the manufacturing and software development spaces both have the ability to be agile. He suggests that as Agile manufacturing has evolved over a long period and encompassed the wider business context, so too does the newcomer ‘agile software development’ need to evolve and adapt to modern circumstances. As we can see there is the wider company context to include when dealing with the Lean concept. The management or governance of the lean practices and the people performing them is an integral thread, and in our context, the development group especially. (Ambler and Kroll, 2007) wrote a three part article for IBM, where they identified a collection of practices for Lean governance of software development projects. These were: Adapt the process (repeatable results not necessarily repeatable processes), Align stakeholders policies with IT values, Align team structure with architecture, and Integrated lifecycle environment (automate the drudge-work). They identify 18 specific practices which support this strategy. Ambler (the creator of Agile Modelling), puts it this way: “The heart of the Lean Development Governance framework … is that good governance should motivate, and then enable, IT professionals to do what your organization believes to be the correct behaviours.”
Regulation
Developing software for embedding in devices is quite different to nonembedded software development. In 2003 (Graaf et al., 2003) performed a review of embedded-software development practices, and found that the existing methodologies did not take into consideration the specific needs/constraints of embedded system development nor did they indicate how to apply them to the different domain areas. The increasing complexity of devices is making the hardware and software processes much more interlinked and so the software development methodology must take that into consideration. In many cases projects were being driven by the hardware development due to longer lead-times. Software architects were not being consulted at the system level design resulting in issues that had to be resolved
368
in the software layer instead of the hardware layer. More flexible methodologies are needed and it is important to remember that groups outside the IT department need to be involved fully in the process. In particular the development of medical devices, as would be expected, is highly regulated in order to ascertain the effectiveness of a company’s manufacturing and software development practices at ensuring device safety and reliability. Indeed as technology progresses and software and hardware become more interwoven and easier to connect, the definition of a Medical Device is continuously evolving. The regulatory standards govern the manufacturing of the devices or software application which remotely/wirelessly connects to the device, equally as much as the software which is embedded on the device. While these standards are quite rigorous and require detailed record keeping, they are not necessarily prescriptive. For example, IEC-62304 defines the various software development life-cycle processes and activities. It acknowledges that there are different software development methodologies but does not stipulate which ones to use. The important thing is that the processes, activities and tasks, as identified by the regulations, are being implemented. One particular aspect of regulated industries such as Medical Devices is the need for an auditable trail of documentation, and this is typically where a lot of suspicion exists around Agile methods. Indeed, the Agile Manifesto states that they value “Working software over comprehensive documentation”. The literature, however, does help us clear-up this misconception and provides us with many examples of how companies have achieved these regulatory requirements and still remained Agile. For example, Abbott implemented Agile on a pilot project in 2004 (Rasmussen et al., 2009). As part of each iteration they delivered key artefacts which they were able to use to demonstrate their regulatory compliance. They did find however that, iterations less than 6 weeks were not efficient due to the large documentation requirements of the FDA. They reported overall lower cost and shorter project duration. Software development in the avionics industry has equally stringent regulatory watchdogs to ensure that the risk of injury to humans is minimal. In the US, the Federal Aviation Authority uses the DO-178B standard to determine safety and reliability when software certification is sought. Similar to ISO 13485:2003, the standard does not impose any particular software development methodology but rather specifies objectives, descriptions of activities, and descriptions of evidence that must be satisfied. (Wils et al., 2006) performed a mapping of the core Agile principles against the requirements of DO-178B and found they needed to re-interrupt 3 of the principles. They found that similar to other embedded-software domains, the further on in the lifecycle you are, the less agility it is possible to maintain. The final stage of certification is where they see the least amount of agility possible. Similarly
369
(Chisholm, 2007) carried out research to investigate whether Agile methods could satisfy the objectives of DO-178B. His findings supported this hypothesis, however, he has specific suggestions as how some practices will need to be modified or supplemented in order to fully satisfy DO-178B. In the 1990s it was becoming apparent that within the field of satellite systems, the future need to reduce development schedules from 3-4 years down to 18-24 months with more software-intensive systems meant that the prevailing ‘V’ (waterfall) methodology was becoming a hindrance. (Vardanega and van Katwijk, 1998) suggested that practices such as more concentrated testing earlier in the process, among others, could help speed up the process. (Nielsen and McMunn, 2005) discuss the first implementation and subsequent rollout of XP as the preferred software development method for a large financial services organisation. By outsourcing the first Agile project and then building on small tangible successful initiatives they progressed over a multi-year timeframe to a position where the internal developers became fullfledged Agile experts. There would have been little success were they to have attempted to effect large scale change all at once. The use of an outside contract company had two advantages. First, they had a fixed price contract (although they don’t give any detail on how that price was estimated upfront within an Agile context), and second, the internal business and IT community got controlled exposure to agile development (they were largely shielded from the contractor’s internal agile processes). The implemented agile methodology was a tailored version of XP.
In Review
In an effort to examine how the software development process within ... might more fully adopt Lean approaches, we have examined how the practices of Agile software development methodologies, as opposed to traditional methods, could support this aim. We have seen how the concepts of Lean and Agile are very closely aligned and that specific Agile methods are supportive of those philosophies. Our main focus is to ensure that the medical device regulations can still be satisfied by these methods. We note that while the regulations are strict, they are not prescriptive, and that gives us the freedom to examine different software development methodologies. We continue below to try and answer some specific questions which ... were interested in.
Specific Questions which are relevant to ...: 1. What does it mean to be Lean/Agile and how can you leverage Agile practices to achieve a Lean software development process? In software development terms I think the introduction helps to show the relationship between Lean and Agile. Lean software development is considered a component of the overall product development process. As with a Lean product development process you uphold the same principles. The seven principles again are: 1. Eliminate Waste
370
2. Build Quality In 3. Create Knowledge 4. Defer Commitment 5. Deliver Fast 6. Respect People 7. Optimise the Whole Each of these principles gets translated into specific software development related principles. For example the first Lean principle ‘Eliminate Waste’ identifies 8 types of waste. One of these is ‘Defects’, which you should strive to prevent from occurring instead of finding and then fixing later. The cost of fixing something later in a process is much higher. In software development terms this would translate to bugs in the code. To bring in an Agile context here, you might use a typical Agile development methodology to tackle that specific issue, for example Test Driven Development (TDD). TDD calls for the creation of a test for a piece of code before the code is written. The aim is to increase the likelihood that the piece of code will work correctly first time, and to ensure that any subsequent changes to the code that introduce a new bug, are caught quickly by re-executing the test(s). Each of the other principles has a similar mapping, where some method from the Agile toolkit is used to directly address that issue. In this way, the Agile practices “on the ground” so to speak, are supporting the Lean principle governing it. 2. How well do Lean/Agile practices fit together with a large number and size of requirements documents? The Agile approach to requirements is one of the major departures from the traditional ‘waterfall’ development process. We need to take a step back and understand that from an Agile perspective, requirements or “user stories” at the beginning are ideally high level and not a fully loaded list of all possible features that the users might ever need. The iterative approach to the development, where requirements are refined and extended as needed is where the real power of Agile comes into play. There does not appear to be a limit to the number of user stories, but the aim is to keep them as simple as possible (Beck and Fowler, 2000). This of course is where regulated industry has problems, since documentation for traceability purposes is paramount. However, the literature does have reports on how this can be addressed. The key is to systematically keep the main artefacts produced at each iteration as an audit trail. Some of these artefacts may need to be expanded slightly if the team feel they are not adequate enough. For example, (Lin and Fan, 2009) decided that in order to balance agility with discipline, they would setup their own documentation template which involved more overhead at certain points in the process. This hybrid methodology maintained discipline around certain practices such as documentation, while allowing the development process to remain Agile.
371
3. Which architectural prerequisites exist for the application of Lean/Agile? There are no prescriptive architectural conditions, software or hardware, that the Agilists promote for an effective Agile implementation. This is possibly why embedded-software development is discovering some short comings with Agile. As Agile methods are getting adopted in the different software development domains, new constraints and dependencies are emerging (see question 4). As you proceed through an Agile project, you may find that as a result of the need to constantly test code (sometimes even before you write it!), the practice of continuous integration, and the sharing of code, that you will tend towards an architecture that lends itself to these practices. For example you may impose coding standards and/or you may insist that every component has a built-in test function that can be externally invoked. From a hardware perspective (Srinivasan et al., 2009) conducted a ‘state of the art review’ of Agile methods within embedded-systems development. Some of the main findings are that the choice of methodology is very dependent on the hardware architecture and physical restrictions imposed. They suggest that one of the major challenges is to find the right tools to support test-driven development without compromising existing efficient practices. 4. How well do Lean/Agile practices support development of software with external constraints? e.g. hardware development schedule This is typically one of the major concerns around Agile practices for embedded-software development. The fear is that Agile methods were designed and work well for computer based applications, but not when the software needs to run on a hardware device. (Grenning, 2002) identifies some of the constraints that embedded-software projects may experience, such as hardware not being available until late in the process, timing issues, the lack of human-computer interface, limited memory space or processing power, and safety concerns. He also performs an informative step through an XP project for the development of an embedded-software device. Since, at the start there is no hardware for testing, an interface needs to be created and its behaviour simulated. An interesting observation he makes is that it is often advantageous not to have the hardware available at the start since it forces the developers to look at the problem more abstractly and so helps to isolate the hardware interface from the application. He suggests that safety critical systems can benefit from the application of XP techniques but practitioners need to evolve them to meet their own specific needs. (Ronkainen and Abrahamsson, 2003) take a specific look at projects which have very specific hardware constraints such as projects within
372
the telecommunications industry. They examine how Agile methods are suitable. As opposed to (Grenning, 2002) there hardware is available from the start and so the co-design happens in parallel. Memory and power consumptions are factors which may change during the life of the project and they therefore suggest that the software development methodology needs to be something that can cope with changing requirements. Some issues they highlight are developing the bare minimum functionality needed, and also refactoring which they say can be hazardous in a time-sensitive device. As the project tends towards completion there are many stakeholders who are reliant on the code, and changes at this stage have a wide impact. In their opinion the development disciplines need to become more rigid at this point, something which agile practices do not cater for. They tabulate four areas where they believe further work is needed in making Agile methodologies more compatible with the requirements of embeddedsoftware development. There is also the bringing together of the software and hardware teams. When taking an Agile approach, both teams need to be integral elements in the process. The fostering of closer working relationships through the Agile practices can be used to help identify possible issues ahead of time instead of first when the software meets the hardware. Driving down defects is a key Lean principle. 5. How well do Lean/Agile aspects fit with organizational factors typical to commercial environments? Relevant aspects include: project management; skill development; career possibilities; communication (e.g. in distributed development projects); motivation of project members; authority, developer status, responsibility, freedom of choice; fluctuation What clearly emerges from the literature is that any Lean or Agile strategy can only work if the people doing it are organised and motivated appropriately. In the pursuit of Japanese efficiency “...it was only after American carmakers had exhausted every other explanation for Toyota’s success – an undervalued yen, a docile workforce, Japanese culture, superior automation– that they were finally able to admit that Toyota’s real advantage was its ability to harness the intellect of “ordinary” employees.” (Hamel, 2006). As mentioned earlier, (Ambler and Kroll, 2007) suggest that good Lean governance is essential for success. One of the seven core principles of Lean software development is People (Poppendieck and Poppendieck, 2006). A succinct point from their book is the following: “With people, the things that matter are skill, pride, expertise, confidence, and cooperation”. (Srinivasan et al., 2009) make six recommendations for using Agile methods in embedded-systems development. Three of them refer to people. They are to supply:
373
organizational support infrastructure ranging from new tools and methods, to refined communication/collaboration approaches - mechanisms and incentives to support knowledge transfer and sharing - the management of culture change, which is even more critical in the case of agile methods, since it emphasizes the ‘soft factors’ governing organizational work From a distributed team perspective, (Cusumano, 2008) suggests some core practices that are needed when implementing Agile software development methods in a global team environment. Commonly accepted development processes are essential. Splitting the team into sub-groups allows for parallel development of uncoupled modules. A strong project manager who can lead the effort is necessary. It is also important to have good tools which can support daily builds, continuous regression and integration testing, version control and configuration management. -
6. How does the clear separation between developer and user community affect the applicability of Lean/Agile practices? The immediate point to make is that, within the context of Lean/Agile software development, one of the goals is to minimise this distance i.e. bring the developer and customer face-to-face as often as possible. This reduces misunderstandings, speeds up feedback on questions and testing, and builds a collaborative relationship between developer and customer. It is often not possible to have the customer on site or commit to such close working conditions, and in that case it is recommended that an internal representative is appointed who will speak on behalf of the customer. However that person should be someone independent of the development team, fully versed with what the customer values, and senior/strong enough to represent the best interest of the customer within your organisation. (Spence, 2005) highlight an issue they encountered with this when it turned out that the internal representative just didn’t have enough time to play that role adequately. 7. Which aspects of Lean/Agile practices would affect the customer in a regulated environment (i.e. where would they need to adapt)? The customer gets affected in a number of ways. First, as we touched on above, it is expected that a much closer working relationship with the developers is achieved. For commercial applications this may be more difficult than when working within your own organisation. Customer requirements are given at a much higher level and throughout the duration of the project they would be expected to be available for questions, clarifications, and to prioritise the features they want delivered. Second, they would be required to perform testing at various intervals throughout the project. The results of the testing feed back to the developers immediately and issues get resolved for the next iteration. This way they would see the positive results of their
374
involvement. Also, since they prioritise the features to develop, they get to see the most valuable functionality delivered first. Third, because of the closer involvement in the project they share the project ownership which helps drive towards a successful end. Fourth, which is more relevant in a commercial setting, is that there would possibly need to be some re-thinking around the contractual arrangement, since ideally the true scope and duration of the project emerges as the project progresses. 8. What do case studies have to report on the effectiveness/feasibility of introducing Lean/Agile practices into a regulated environment? There are many reports supportive of use of Lean/Agile methodologies in embedded-software development, including regulated environments such as medical devices. Within the Aerospace industry, for example, (VanderLeest and Buter, 2009) found that nearly all Agile practices can be mapped to the DO-178B standard and yet the Aerospace industry has been slow in adopting them. They compare a typical waterfall type process to a more agile one and comment on the impact to the validation and verification process. They show that, as a result, the regulation authority (FAA) has an increased level of participation in the process since they perform “stage of involvement” (SOI) audits. These SOI audits are increased in the Agile process due to its iterative nature. This they suggest is offset by shorter SOI audits and fewer unforeseen issues arising in the final certification review. They do however highlight some key difficulties with the practice of fixed length iterations, such as: lockstep gateways (stage gates), complex subcontractor relationships and Legacy artefacts. The DO-178B standard is in the process of being revised (DO-178C) mainly to take into account advances in technology and software languages. Within the software development community, the Open-DO Initiative 12 is calling for a more open-source approach to aviation software development. They state that “By leveraging on lean approaches and agility we aim, within the OpenDO initiative, to shift the focus of safety-critical software development to more continuous and incremental certification approaches.” (Yoo et al., 2005) operating within the domain of Nuclear Engineering Applications, demonstrate NuSCR, a formal specification language, to document the requirements for real-time embedded software. This can generate equivalent Function Block Diagram (FBD) code from which, with the aid of CASE tools, executable code can be automatically generated for the programmable logic controllers (PLCs). This they say makes it easier to validate correctness when demonstrating software safety to the regulators.
12
http://www.open-do.org/about/
375
The software process department at Daimler-Benz often have to develop customer specific add-ons to be implemented in hours or days (Manhart and Schneider, 2004). They found that no “out-of-the-box” Agile method to be suitable as is, so they started with Agile practice of automated Unit Testing and ‘Test First’ as a very tentative first step which helped to foster the acceptance and success of the Agile process improvements. Finally, within the financial services area, any publicly trading US company must adhere to relatively new financial accounting and reporting standards as specified in the Sarbanes-Oxley Act of 2002 (SOX)13. These SOX controls have been
successfully mapped to agile practices (Gupta, 2008). One of the issues encountered was to get the various parties on board and performing activities they wouldn’t have done before. However, he points out that some formality had to be introduced to augment the ‘off-the-shelf’ practices. 9. Is there evidence that the medical device industry is tending towards a move to Lean/Agile software development practices? I don’t think it is possible to answer this in the broad sense. There is certainly strong evidence to say that medical device companies have been and continue to look to the promises that Agile practices hold. The literature has reports of successful Agile implementations but, as with the embedded-software domain in general, there are caveats which have led to ‘flavours’ of Agile methods being implemented. Cochlear (manufacturers of hearing devices), introduced some agile practices in 2001 (Rottier and Rodrigues, 2008). They took a cautious approach, just a few practices to start with and then in 2005 they implemented SCRUM. They used the practice of user-stories from XP to maintain traceability for regulatory compliance. Other groups within Cochlear have now also started using SCRUM. (Lin and Fan, 2009) describe their experiences developing software for a Digital Subtraction Angiography (DSA) medical device. An experienced Agile software development house, this was their first medical device software project. They note the most important thing for FDA approval is the need to perform formal review and approval steps. They implemented a hybrid Planned-Agile methodology in order to get the benefits of agility while maintaining discipline around certain areas such as documentation. (Spence, 2005) brings us through their journey to implement Agile (XP and Scrum) into a company developing class III medical devices. They found that the practices of pair-programming and test-driven development provided early feedback and quality. They reported lots of positive feedback from all members including making it faster for new 13
http://www.sec.gov/about/laws.shtml#sox2002
376
developers to get up to speed. Again this has led to other internal software teams becoming interested in Agile methods and looking to them for assistance. (Rasmussen et al., 2009) discuss the successful implementation of Agile practices within Abbott’s diagnostic division. They teamed up with a custom software engineering firm who specialised in Agile development to help guide them through the process. Overall they reported lower costs and shorter project duration.
References
ABRAHAMSSON, P., WARSTA, J., SIPONEN, M. T. & RONKAINEN, J. 2003. New directions on agile methods: a comparative analysis. Proceedings of the 25th International Conference on Software Engineering. Portland, Oregon: IEEE Computer Society. AMBLER, S. W. Year. Agile software development at scale. In: 2nd IFIP TC 2 Central and East European Conference on Software Engineering Techniques, CEE-SET 2007, October 10, 2007 - October 12, 2007, 2008 Poznan, Poland. Springer Verlag, 1-12. AMBLER, S. W. & KROLL, P. 2007. Best practices for lean development governance [Online]. IBM. Available: http://www.ibm.com/developerworks/rational/library/jun07/kroll/ [Accessed 22nd January 2010]. AVELING, B. 2004. XP Lite Considered Harmful? BECK, K. & FOWLER, M. 2000. Planning Extreme Programming Addison-Wesley Professional CHISHOLM, R. A. 2007. AGILE SOFTWARE DEVELOPMENT METHODS AND DO-178B CERTIFICATION. MASTER OF APPLIED COMPUTER SCIENCE, ROYAL MILITARY COLLEGE OF CANADA. CUSUMANO, M. A. 2008. Managing software development in globally distributed teams. Commun. ACM, 51, 15-17. DINGSOYR, T. & DYBA, T. 2009. What Do We Know about Agile Software Development? Software, IEEE, 26, 6-9. DITTRICH, Y. & LINDEBERG, O. 2004. How use-oriented development can take place. Information and Software Technology, 46, 603-617. FRASER, S., RISING, L., AMBLER, S., COCKBURN, A., ECKSTEIN, J., HUSSMAN, D., MILLER, R., STRIEBECK, M. & THOMAS, D. 2006. A fishbowl with piranhas: coalescence, convergence or divergence? Companion to the 21st ACM SIGPLAN symposium on Objectoriented programming systems, languages, and applications. Portland, Oregon, USA: ACM. GRAAF, B., LORMANS, M. & TOETENEL, H. 2003. Embedded Software Engineering: The State of the Practice. In: MARCO, L. & HANS, T. (eds.). GRENNING, J. Year. Extreme programming and embedded software development. In, 2002. GUPTA, S. Year. SOX compliant agile processes. In: Agile 2008, 4-9 Aug. 2008, 2008 Piscataway, NJ, USA. IEEE, 140-3. HAMEL, G. 2006. THE WHY, WHAT, AND HOW OF MANAGEMENT INNOVATION. (cover story). Harvard Business Review, 84, 72-84. HIBBS, C., JEWETT, S. C. & SULLIVAN, M. 2009. The Art of Lean Software Development, O'Reilly Media. KETTUNEN, P. 2009. Adopting key lessons from agile manufacturing to agile software product development--A comparative study. Technovation, 29, 408-422. LIN, W. & FAN, X. Year. Software development practice for FDA-compliant medical devices. In: 2009 International Joint Conference on Computational Sciences and Optimization, CSO 2009, April 24, 2009 - April 26, 2009, 2009 Sanya, Hainan, China. IEEE Computer Society, 388-390. LONGSTREET, D. 2007. The Agile Method and Other Fairy Tales [Online]. Available: http://www.softwaremetrics.com/Agile/Agile%20Paper.pdf [Accessed].
377
MANHART, P. & SCHNEIDER, K. Year. Breaking the ice for agile development of embedded software: an industry experience report. In: Software Engineering, 2004. ICSE 2004. Proceedings. 26th International Conference on, 2004. 378-386. NIELSEN, J. & MCMUNN, D. Year. The agile journey -Adopting XP in a large financial services organization. In: 6th International Conference on Extreme Programming and Agile Processes in Software Engineering, XP 2005, June 18, 2005 - June 23, 2005, 2005 Sheffield, United kingdom. Springer Verlag, 28-37. POPPENDIECK, M. & POPPENDIECK, T. 2003. Lean Software Development: An Agile Toolkit Addison-Wesley Professional POPPENDIECK, M. & POPPENDIECK, T. 2006. Implementing Lean Software Development From Concept to Cash, Addison-Wesley Professional RASMUSSEN, R., HUGHES, T., JENKS, J. R. & SKACH, J. Year. Adopting Agile in an FDA Regulated Environment. In: Agile Conference, 2009. AGILE '09., 2009. 151-155. RONKAINEN, J. & ABRAHAMSSON, P. Year. Software development under stringent hardware constraints: do agile methods have a chance? In: Extreme Programming and Agile Processes in Software Engineering. 4th International Conference, XP 2003. Proceedings, 25-29 May 2003, 2003 Berlin, Germany. Springer-Verlag, 73-9. ROTTIER, P. A. & RODRIGUES, V. Year. Agile Development in a Medical Device Company. In: Agile, 2008. AGILE '08. Conference, 2008. 218-223. RYAN, J. J., JR. & SCUDIERE, R. Year. The price of agile is eternal vigilance. In: Agile 2008, 4-9 Aug. 2008, 2008 Piscataway, NJ, USA. IEEE, 125-8. SPENCE, J. W. Year. There has to be a better way! In: AGILE Conference 2005, July 24, 2005 July 29, 2005, 2005 Denver, CO, United states. Inst. of Elec. and Elec. Eng. Computer Society, 272-278. SRINIVASAN, J., DOBRIN, R. & LUNDQVIST, K. Year. 'State of the Art' in Using Agile Methods for Embedded Systems Development. In: Computer Software and Applications Conference, 2009. COMPSAC '09. 33rd Annual IEEE International, 2009. 522-527. VANDERLEEST, S. H. & BUTER, A. Year. Escape the waterfall: Agile for aerospace. In: Digital Avionics Systems Conference, 2009. DASC '09. IEEE/AIAA 28th, 2009. 6.D.3-1-6.D.3-16. VARDANEGA, T. & VAN KATWIJK, J. 1998. Productive engineering of predictable embedded realtime systems: the road to maturity. Information and Software Technology, 40, 745-764. WILS, A., VAN BAELEN, S., HOLVOET, T. & DE VLAMINCK, K. Year. Agility in the avionics software world. In: 7th International Conference on Extreme Programming and Agile Processes in Software Engineering, XP 2006, June 17, 2006 - June 22, 2006, 2006 Oulu, Finland. Springer Verlag, 123-132. YOO, J., CHA, S., KIM, C. H. & SONG, D. Y. 2005. Synthesis of FBD-based PLC design from NuSCR formal specification. Reliability Engineering & System Safety, 87, 287-294.
378
Appendix K – The SaLeS-4-MD Model Version 1 Table K-39 The SaLeS-4-MD Version 1
Lean Tool Description 1
S/W Dev. Context
Specific Practice (Lean/Agile)
Issue in SafetyCritical Embedded Domains
Find defects
Find Defects Early (TDD)
Hardware may be unavailable
Unit testing
Use a testing framework e.g.xUnit
Hardware may be unavailable
Use mocks/stubs/commentout Use an Hardware Abstraction Layer Use a simulator Use mocks/stubs/commentout
Device hardware may be unavailable.
Use or remove mocks/stubs/commentout depending on hardware availability
Specific Practice (Non-Agile)
Possible Remediation Technique
Seeing Waste -Defects
Bugs
Integration Testing
Continuous Integration
May require extra hardware for building the software. Regulations will insist on a final proof of full integration
379
Final verification should be much less trouble some, faster with a higher level of confidence
Non-functional testing Develop a test for each defect as soon as it is detected (TDD) Behaviour driven development Executable specifications
-Overproduction
Extra Features/Code
Hardware may be unavailable Each defect and fix must be traceable back to requirement Tool support may not be available
Use Automated tests
Hardware may be unavailable
User Interface testing
Automated UI testing
Difficult to automate
User Testing
User Testing Prioritise requirements for development. Features that don't create clear customer value should not be created
Remember 80/20 rule !
Follow a YAGNI approach to developing functionality Short (4-6 week) iterations with customer feedback
380
User test instructions too prescriptive
There may be increased risk mitigation by not always doing the most minimal solution. 4-6 weeks may be too short
Use mocks/stubs/commentout Develop a more formal documentation process Revert to TDD Use mocks/stubs/commentout Some tool support available for web frontends. Separate this code from backend Focus on the intent of the test rather than the execution
Use a combination of the YAGNI and BDUF approaches to requirements specification (SYDNI) Use iteration lengths appropriate to your situation
Use design patterns
Only develop for the current iteration Use design patterns and refactor design patterns Refactor
Refactoring in later iterations can cause regulatory problems
May break existing code Potentially Hazardous due to S/W changes affecting H/W e.g. timing changes
-Transportation
Handoffs
More detailed documentation
Face-to-face communication
Requirements must be documented in a traceable manner Risks must be documented in a traceable manner
Limit refactoring to early iterations Constant testing Pair Programming Create a source code branch in repository, and only integrate when all tests have passed Pervasive use of S/W configuration management which includes hardware versions. System level impact analysis methods Relentless testing Use TDD for automatic Traceability? Huffman Hayes, 2009
Avoid Handoffs wherever possible -Waiting
Delays
Close/easy access to customer
381
Access to customer may not be possible
Appoint a proxycustomer from within the company
Avoid task switching -Inventory
Partially Completed Work
-Motion
Task Switching
(Over)Processing
Unneeded processes
-Re-Planning
Over Planning
2
Value Stream Mapping
SDLC Process Map
17
Perceived Integrity
Customer Perception Of Developers
No un-coded requirements No un-tested code No un-documented code No un-deployed code No un-fixed bugs One feature at a time through to completion Use Kanban and Limited WIP Avoid task switching
Draw out the SDLC process flow
One feature at a time through to completion Don’t write documents that nobody reads Automate Manual tasks (Jidoka) Eliminate Procedures that accomplish nothing Eliminate Procedures that make simple tasks hard Only plan enough for the next iteration Generate value stream map of development process Target biggest non-value add step Use MDD or BDD to provide clarity for all
382
Regulations will insist on a level of documentation
knowledge -maintain perceived integrity
Deliver an automated test suite with code base Deliver high-level overview model
18
Conceptual Integrity
A sound effective architecture
19
Refactoring / Simplicity
Simplify complex code
20
Testing
Finding Bugs
Component re-use (proven software) Integrated Problem Solving Root cause analysis Use architectural layers Hide things that will vary Using coding standards Use experienced developers in critical areas Master developer to act as a leader Automated Testing
Perform software tests
Refactor Maintain clarity Check performance levels Look out for identical code Remove Old/un-needed code Automated test suites
383
Full system integration
Continuous Integration
Device hardware may be unavailable. May require new software/scripts capable of continuous integration which may need certification.
Legacy systems need careful consideration
Governance
6
Risk Assessment Set-Based Development
7
Options Thinking
Governance
Assess risk of failure Concurrent Development Minimise irreversible design decisions
Closely manage resources
Carry out FTA or FMEA
Embed compliance in process
Carry out FTA or FMEA Develop multiple solutions Don't tie the design down when faced with uncertainty Delay making decisions if uncertainty exists Make plans but not predictions
384
Use or remove mocks/stubs/commentout depending on hardware availability
Script should controlled in SCM Create build tests, add unit tests as modifications/refactoring happens, follow an agile approach for new modules e.g. TDD
8
The Last Responsible Moment
Delay making decisions until the last responsible moment
Short (4-6 week) iterations with customer feedback Share partially complete design information Organise for direct worker-to-worker collaboration Develop a sense of what is critically important in the domain A mechanism to cause decisions to be made before it is too late Develop a quick response capability
Develop a sense of when decisions must be made
Develop a sense for how to absorb code change
Develop tactics to delay commitment
Use modules Use interfaces Use parameters Use abstractions Avoid sequential programming Beware of custom tool building Avoid repetition Separate concerns Encapsulate variation Defer implementation of future capabilities Avoid extra features
385
4-6 weeks may be too short
Use iteration lengths appropriate to your situation
9
5
Making Decisions
Decisions should be made by the appropriate person
Synchronisation
Building the complete application with all sources included
Breadth-first v Depth-first Front-line developers better positioned to make some decisions Specify a simple set of rules as guidelines Harness the 'swarm intelligence' Have a good source code management tool Frequent check in and out of code by all developers
Frequent build of full application
Daily check in and out of code by all developers. Only check in functional code for building Use coding standards Daily build of full application Daily run of automated test suite Use Spanning Use a Matrix approach
??
Knowledge Sharing
Team Knowledge
Face-to-face communication Collaborative workspace Everyone involved in presenting at end of
386
Traceability required for build scripts
Build scripts should be under SCM
Hardware may not be available Interfacing Issues between components Requirements must be documented in a traceable manner
Develop interfaces first Use TDD for automatic Traceability? Huffman Hayes, 2009
iteration Everyone assigned a role Stand-up meetings Pair-Programming Add comments to code after code analysis Assign a tracker for 'Measures' Create a team definition of 'Done' 3
4
Feedback Loops
Iterations
Feedback on how the code is achieving its goal
Iterative Development
Run tests as soon as code is written Write code instead of more documentation or detailed planning Propose user interfaces and get feedback instead of more detailed requirements Test top 3 tools in house instead of trying to pick the right ONE Every developer should know who their immediate customer is Short (4-6 week) iterations with customer feedback Feature driven development Measure Velocity
387
Hardware may be unavailable
Access to customer may not be possible 4-6 weeks may be too short
Use mocks/stubs/commentout
Appoint a proxycustomer from within the company Use iteration lengths appropriate to your situation
11
Queuing Theory
12
Cost Of Delay
10
Pull Systems
Display burn-down charts Small teams with the necessary expertise
Organisational Support Order and quantity of WIP
Project review meetings Small work packages Move variability down stream Address bottlenecks Don't max out the testing department Build some slack into the team Use an economic project model (p/l)
Assigning development tasks
13
SelfDetermination
Development teams have responsibility
14
Motivation
Motivation
Kanban (just-in-time) Use Kanban chart for visibility Short (4-6 week) iterations with customer feedback Short daily meetings (15 minutes)
Develop a clear and compelling purpose
Empower the team Delegate power down Develop teams decision making capabilities Self-Organising Teams Ensure all participants are adequately trained in chosen Agile
388
4-6 weeks may be too short
Use iteration lengths appropriate to your situation
methodology. Be sure the purpose is achievable
15
Leadership
Leadership
16
Expertise
Building and sharing expertise
Give team access to customers Team makes its own commitments Management tackles interference Keep sceptics away Develop a sense of 'teamness' Promote a 'safe' environment Instil an air of competence Make project status highly visible Make progress visible Encourage moderation over heroism Master developer to act as a leader Project leaders not managers
Form communities of expertise Pair-Programming Peer reviewing Survey the group and ask what expertise is lacking
21
Measurements
Measuring performance
Focus on the entire process
389
Can be problematic when the rest of the organisation is using a more waterfall like
approach
Avoid too much local optimisation
Use information measurements 22
Contracts
Contracts
Assign Risk to the appropriate place
Time and material Multistage Delivered Feature
390
Can be problematic when the rest of the organisation is using a more waterfall like approach Can be problematic when the rest of the organisation is using a more waterfall like approach
Appendix L – The SaLeS-4-MD Model Version 2 Table L-40 The SaLeS-4-MD Version 2
Safe and Lean Software for Medical Devices (SaLeS-4-MD) Medi-SPICE Me diSPI CE Acti vity
5.5
ENG .5.5. SP1
Specific Practice Description
Lean Practices
Source
Require d for Device Class
Lean Principl e
Lean Practi ce
Descri ption
Pote ntial Issu e in MD Dom ain
Mitig ation
Sour ce
R&D phase not govern ed by the regulat ions
Regu lator may seek some high level indic ation of verifi catio n
High level verifi catio n repo rt is prud ent
Case Stud y
The purpose of the Software Unit Implementation and Verification Activity is to produce verified software units that properly reflect the software design.
Define and establish a unit verification strategy
ASPICE , FDA GPSV, ISO 13485 & IEC 62304 MB
Eliminat e Waste (Avoid OverProcessi ng)
B,C
391
Consi der if perfor ming an R&D phase
Eliminat e Waste (Avoid OverProcessi ng) SP1.1 Where verification is done by testing
IEC 62304 MB
Decid e on level of concer n
Differe nt LOCs require differe nt levels of detail
Ther e is risk invol ved in doin g too little
Choo se the level of risk/ conc ern you are comf ortab le with and verif y accor dingl y
Don't write docum ents that nobody reads
Diffe rent stake hold ers may requi re diffe rent level s of
Strik ea balan ce
Case Stud y
Hibb s et al 2009
B,C
Test procedures shall be evaluated for correctness
ENG .5.5. SP2
Define and establish software unit verification criteria
ASPICE , FDA GPSV, ISO 13485 & IEC 62304 MB, MC
Eliminat e Waste (Avoid OverProcessi ng)
B,C
392
Just enoug h docu menta tion
Lin & Fan 200 9
Case Stud y
Refle ctive Analy sis
docu ment ation
ENG .5.5. SP3
Define and establish software unit acceptance criteria
SP3.1 Additional Acceptance Criteria
FDA GPSV, ISO 13485 & IEC 62304 MB, MC
IEC 62304 MC
Eliminat e Waste (Avoid OverProcessi ng)
B,C
C
When present in the design -Proper event sequence -Data and control flow
393
Reuse existin g artefa cts
User stories becom e the require ments
May not have enou gh detai l for regul ator
Inclu de clear condi tions of satisf actio n Each iterat ion gathe rs extra detai l for its user stori es
Shoe make r& Van Shoo ende rwoe rt 2010
Shoe make r& Van Shoo ende rwoe rt 2010
-Planned resource allocation -Fault handling (error definition, isolation, and recovery) -Initialization of variables -Self-diagnostics -Memory management and memory overflows -Boundary conditions
ENG .5.5. SP4
Develop software units
ASPICE , FDA GPSV, ISO 13485 & IEC 62304 MA
Eliminat e Waste (Avoid OverProducti on)
A,B,C
Only develo p for curren t iterati on
Concen trate only on what is require d for this iteratio n Regu lator y focus will be on V&V activ ities
394
Perfo rm V&V durin g and after each iterat ion
Hibb s et al 2009
Refl ecti ve Anal ysis
Rasm usse n et al 2009
Refl ecti ve Anal ysis
Deliver Fast
395
Short iterati ons
Use short iteratio ns to get feedbac k quickly
Prob lems may arise whe n work ing with tradi tiona l team s who want an all or nothi ng appr oach Typi cal agile itera tion lengt hs may be too short
Defin e com mon skele ton interf aces whic h can be work ed on in piece s Use iterat ion lengt hs appr opria te to your situa tion
Drob ka et al 2004
Rotti er and Rodri gues 2008
Ras mus sen et al 200 9
Case Stud y
Refle ctive Analy sis
Create Knowled ge
Build a protot ype
Maintain Flow
Don't overlo ad team memb ers
Build Quality In
396
Reuse existin g artefa cts
Allow custom er to 'play with' prototy pe so they can refine require ments Perfor m load levellin g (Heijun ka) Re-use existin g code/c ompon ents/O TS applica tions speeds up the develo pment process Builds confide nce in the new
Tend ency to lightl y test or not test at all
Inclu de as part of valid ation plan
Case Stud y
Refl ecti ve Anal ysis
Case Stud y
Refl ecti ve Anal ysis
Vogel 2010 Case Stud y
Gra af et al 200 3 Refl ecti ve Anal ysis
Case Stud y
Refle ctive Analy sis
functio nality
Build Quality In
397
Take an object orient ed appro ach
Compo nentisa tion, modula risation , parame terisati on hides away comple xity and limits the effects of change s Assists in a Distrib uted Softwa re Develo pment configu ration
Popp endei ck & Popp endie ck 2003
Cas e Stud y
Case Stud y
Refl ecti ve Anal ysis
Refle ctive Anal ysis
Build Quality In
Maint ain Code Clarity
Use an intuitiv e naming conven tion Use a commo n langua ge
Build Quality In
398
Use a good source code contro l tool
Use simple notatio n Make in-code comme nts clear and focused Maintai n source code integrit y
Popp endei ck & Popp endie ck 2003 Popp endei ck & Popp endie ck 2003 Popp endei ck & Popp endie ck 2003 Popp endei ck & Popp endie ck 2003
Case Stud y
Hib bs et al 200 9
Jeffri es et al 2001
Hib bs et al 200 9
Jeffri es et al 2001
Hib bs et al 200 9
Jeffri es et al 2001
Hib bs et al 200 9
Jeffri es et al 2001
Refl ecti ve Anal ysis
Case Study
Refle ctive Analy sis
Case Study
Refle ctive Analy sis
Case Study
Refle ctive Analy sis
Case Study
Refle ctive Analy sis
399
Create Knowled ge
Use a good source code contro l tool
Create Knowled ge
Incode docu menta tion
Eliminat e Waste (Avoid OverProcessi ng)
Docu menta tion groupi ng
Daily check in and out so develo pers all have latest change s Put explan atory lowlevel detail in the source code itself as oppose d to a written docum ent Try and organis e code to resemb le the SRS and SDS docum ents
Hibb s et al 2009
Pop pen deic k& Pop pen diec k 200 3
Vogel 2010
Refl ecti ve Anal ysis
Vogel 2010
Ambl er & Kroll 2007
Case Study
Refle ctive Analy sis
Eliminat e Waste (Eliminat e Defects)
Use a TDD appro ach
For less comple x system s, deliver ables can be groupe d into a single docum ent Test driven develo pment will suppor t finding defects early
Case Stud y
Hibb s et al 2009
Hard ware may be unav ailab le
400
Use mock s, stubs or com ment out h/w calls
Hibb s et al 2009
Refl ecti ve Anal ysis Pop pen deic k& Pop pen diec k 200 3
Cor deir o et al 200 7
Cord eiro et al 2007
Kane et al 2006
Kane et al 2006
Van Scho oend erwo ert & Morsi cato 2004
Van Scho oend erwo ert & Morsi cato 2004
Develo p a test as soon as a defect is found
401
Defe cts and fixes must be trace d back to requi reme nts
Use a simul ator
Masti colla & Gall 2008
Use a h/w abstr actio n layer
Masti colla & Gall 2008
Have a form al docu ment ation proc ess
Lin & Fan 2009
Mue ller and Bor zuc how ski 200 2
Gary et al 2011
Eliminat e Waste (Avoid OverProducti on)
402
Refact or Code
Periodi cally reviewi ng and improv ing existin g code
Refa ctori ng in later itera tions may caus e regul atory issue s
Limit refac torin g to early iterat ions
Pote ntiall y haza rdou s to hard ware
Cons tant testi ng Perfo rm pair prog ram ming Use a SCM syste m whic h inclu des the hard ware versi on
Wils et al 2006 Chish olm 2007
Chish olm 2007
Ronk ainen and Abra hams son 2003
Refl ecti ve Anal ysis
Empl oy syste m level impa ct analy sis meth ods
Chan ges may brea k existi ng code
403
Rele ntles s testi ng Creat ea sourc e code bran ch and integ rate only when all tests pass
Ronk ainen and Abra hams son 2003 Ronk ainen and Abra hams son 2003
Cord eiro et al 2007
404
Eliminat e Waste (Eliminat e Defects)
Explor e the use of BDD
Eliminat e Waste (Eliminat e Defects)
Contin uous Softw are Integr ation
Behavi our driven develo pment helps elimina te misund erstand ings betwee n stakeh olders Contin ually integra ting change s catches issues sooner
Lega cy code may pose a probl em
Add build scrip ts and unit tests as modi ficati ons are requ ested
Hibb s et al 2009
Tool supp ort may be poor
Reve rt to TDD
Hibb s et al 2009
Hibb s et al 2009
Gar y et al 201 1
Avoids the "Big Bang" approa ch to integra tion
Contin ually integra ting change s catches issues sooner
405
Rasm usse n et al 2009 May requi re new soft ware /scri pts that will requi re valid ation May requi re extra hard ware for build ing the soft ware
Scrip ts shoul d be contr olled in a SCM syste m
Hibb s et al 2009
Make the appr opria te inves tmen t if requi red
Hibb s et al 2009
Hib bs et al 200 9
Eliminat e Waste (Eliminat e Defects)
Frequ ent S/W and H/W integr ation
Provid es early feedbac k to the hardwa re team
Rasm usse n et al 2009 Regu lator will still insist on a full final integ ratio n Lega cy code can be probl emat ic
Create Knowled ge
406
Use a visual displa y
Open display of work in process inform s everyo ne of status & issues
Final verifi catio n much faste r and less troub leso me
Refl ecti ve Anal ysis
Rotti er and Rodri gues 2008
Graaf et al 2003
Rott ier and Rod rigu es 200 8
Case Stud y
Refl ecti ve Anal ysis
ENG .5.5. SP5
Ensure consistency and bilateral traceability to system and software requirements
ASPICE , FDA GPSV & ISO 13485
A,B,C
Create Knowled ge
Interc hange Devel oper Rolls
Build Quality In / Eliminat e Waste (Avoid OverProcessi ng)
Use simila r docu menta tion struct ure
Develo pers should interch ange rolls in order to increas e broad experti se Docum ents that resemb le each other are easier to review and relate
Docu menta tion groupi ng
Map section s instead of every single detail
Eliminat e Waste (Avoid OverProcessi ng)
407
Critic al areas shou ld be assig ned to expe rienc ed devel oper s
Coul d be seen as cutti ng corn ers
Assig n expe rienc ed devel oper s to critic al areas
Don't make secti ons too big
Case Stud y
Refl ecti ve Anal ysis
Vogel 2010
Dro bka et al 200 4
Case Stud y
Vogel 2010
Cas e Stud y
Egye d et al 2009
MediSPICE Asses smen t1
Refle ctive Analy sis
ENG .5.5. SP6
Ensure consistency and bilateral traceability to detailed design
ASPICE , FDA GPSV & ISO 13485
Build Quality In / Eliminat e Waste (Avoid OverProcessi ng)
A,B,C
Eliminat e Waste (Avoid OverProcessi ng)
ENG .5.5. SP7
Ensure consistency and bilateral traceability to test specification of software units
ASPICE , FDA GPSV & ISO 13485
Build Quality In / Eliminat e Waste (Avoid OverProcessi ng) Eliminat e Waste (Avoid OverProcessi ng)
A,B,C
408
Use simila r docu menta tion struct ure
Docu menta tion groupi ng
Use simila r docu menta tion struct ure Docu menta tion groupi ng
Docum ents that resemb le each other are easier to review and relate Map section s instead of every single detail Docum ents that resemb le each other are easier to review and relate Map section s instead of every
Coul d be seen as cutti ng corn ers
Coul d be seen as cutti ng
Don't make secti ons too big
Don't make secti ons too big
Vogel 2010
Dro bka et al 200 4
Case Stud y
Vogel 2010
Cas e Stud y
Egye d et al 2009
Vogel 2010
Dro bka et al 200 4
Case Stud y
Vogel 2010
Cas e Stud y
Egye d et al 2009
MediSPICE Asses smen t1
Refle ctive Analy sis
MediSPICE Asses smen t1
Refle ctive Analy sis
single detail
Eliminat e Waste (Avoid OverProcessi ng)
ENG .5.5. SP8
Verify software units
ASPICE , FDA GPSV, ISO 13485 & IEC 62304 MB
Eliminat e Waste (Avoid OverProcessi ng)
B,C
409
Build on a TDD appro ach
Consi der if perfor ming an R&D phase
corn ers
Add referen ces to automa ted tests
R&D phase not govern ed by the regulat ions
Huff man Haye s et al 2009 Refa ctori ng may creat e or brea k links Regu lator may seek some high level indic ation of verifi catio n
Faili ng tests shoul d trigg er an upda te of the RTM
Huff man Haye s et al 2009
High level verifi catio n repo rt is prud ent
Case Stud y
Decid e on level of concer n
Eliminat e Waste (Eliminat e Defects)
Use PokaYoke Emplo y formal model checki ng techni ques
410
Differe nt LOCs require differe nt levels of detail Employ defect preven tion and detecti on techniq ues Models can assist in the verifica tion of correct ness of softwar
Ther e is risk invol ved in doin g too little
Mod els may not have been creat ed duri
Choo se the level of risk/ conc ern you are comf ortab le with and verif y accor dingl y
Reve rse engin eer mod els from the sourc
Case Stud y
Robi nson 1997
Refl ecti ve Anal ysis
Tsai et al 2005
Jetle y et al 200 6
Gary et al 2011
e
Build Quality In
411
Use state machi nes
State machin es suppor t safetybydesign, by restrict ing the allowa ble behavi ours of a compo nent
Use Stand ards
Use coding standar ds and guideli nes
ng desig n
All possi ble com pone nt state s shou ld be docu ment ed
e code
Docu emnt and revie w the state mach ine catal og
Gary et al 2011 Popp endei ck & Popp endie ck 2003
Refl ecti ve Anal ysis Hib bs et al 200 9
Case Stud y
Refle ctive Analy sis
Stan dard s are not enfor ced
Root cause analys is
Ensure issues are fully unders tood by using techniq ues such as the 5 whys approa ch
Case Stud y
Popp endei ck & Popp endie ck 2003 Not rigor ous enou gh. Impa ct of issue s need s to
412
Incor porat e chec k of stand ards into code revie w proc ess
Cond uct prop er risk analy sis as requi red. See SP10
Case Stud y
Vog el 201 0
Refle ctive Anal ysis
be rigor ously asses sed.
413
Reuse existin g artefa cts
Re-use existin g code/c ompon ents/O TS applica tions
Use experi enced develo pers for critica l areas
Use experie nced develo pers for critical areas
Tend ency to lightl y test or not test at all Pote ntial to devel op reso urce bottl enec ks if man y critic al areas
and SP11.
Inclu de as part of valid ation plan
Build up devel oper s' expe rienc e
Vogel 2010
Gra af et al 200 3
Case Stud y
Popp endei ck & Popp endie ck 2003
Cas e Stud y
Refle ctive Anal ysis
Refle ctive Analy sis
Perfor m autom ated testin g
Suppor ts GPSV's guideli nes of "consis tently fulfillin g" require ments Use a testing framew ork such as Xunit
Hard ware may be unav ailab le
UI testi ng can be diffic ult to auto mate
Maint ain Clarity
414
Use an intuitiv e naming conven tion
See SP4 (TDD appr oach)
Sepa rate front end from back end and use tool supp ort
Cord eiro et al 2007
Hib bs et al 200 9
Jeffri es et al 2001
Hib bs et al 200 9
Jeffri es et al 2001 Popp endei ck & Popp endie ck 2003
Shoe make r& Van Shoo ende rwoe rt 2010
Vogel 2010
Popp endei ck & Popp endie ck 2003
Case Study
Refle ctive Analy sis
Hib bs et al 200 9 Hib bs et al 200 9
Jeffri es et al 2001
Popp endei ck & Popp endie ck 2003 Popp endei ck & Popp endie ck 2003 Popp endei ck & Popp endie ck 2003
Use a commo n langua ge
Use simple notatio n Make in-code comme nts clear and focused
Don't overdocu ment
415
Keep mainte nance docum entatio n to a minim um
Prod ucts have long life time s, so syste m need s to be supp ortab le by new peop
Have nonteam mem bers evalu ate the docu ment ation 's adeq uacy
Drob ka et al 2004
Hib bs et al 200 9
Jeffri es et al 2001
Hib bs et al 200 9
Jeffri es et al 2001
Hib bs et al 200 9
Jeffri es et al 2001
Vog el 201 0
Refle ctive Anal ysis
Case Study
Refle ctive Analy sis
Case Study
Refle ctive Analy sis
Case Study
Refle ctive Analy sis
le
Create Knowled ge
Pair Progr ammi ng
Integr ated Proble m solvin g
416
Use pair progra mming as a means of continu ous review
Use cross functio nal teams to solve proble ms
Can be diffic ult to pick pairs Fami liarit y bree ds lax pract ices
Drob ka et al 2004 Good overs ight can make this work well
Lev y& Haz zan 200 9
Beck 1999
Vogel 2010 Popp endei ck & Popp endie ck 2003
Ray mon d 200 1
Case Stud y
MediSPICE Asses smen t1
Refle ctive Analy sis
ENG .5.5. SP9
Document software unit verification results
ASPICE , FDA GPSV, ISO 13485 & IEC 62304 MB
Create Knowled ge Eliminat e Waste (Avoid OverProcessi ng)
B,C
Don't overdocu ment Use predefine d templ ates
Use autom ated techni ques
ENG .5.5. SP1 0
Conduct risk analysis
ISO 14971, IEC TR/80 002-1
Eliminat e Waste (Avoid OverProducti on)
A,B,C
417
Reuse existin g artefa cts
Standa rdise docum entatio n to facilitat e ease of com munica tion Eases capture of verifica tion results Autom ated test results can be automa tically logged Use risk analysi s from a similar device as a startin g point
Drob ka et al 2004
Cas e Stud y
Case Stud y
Refl ecti ve Anal ysis
Gary et al 2011
Refl ecti ve Anal ysis
ISO 1497 1:20 07 4.1 Note 1
Burt on, John PhD The sis 200 8
Refle ctive Anal ysis
Medi SPIC E Asses smen t1
Diffe renc es betw een devic es Decid e on level of concer n
Less rigour require d for "low" LOC
Build Quality In
418
Use establis hed risk analysi s techniq ues
ISO 1497 1:20 07 4.1 Note 1
Burt on, John PhD The sis 200 8
Case Stud y Choo sign the wron g LOC
Use establi shed risk analys is techni ques
Perfo rm syste matic evalu ation of differ ence s
Anal ysis tech niqu es need to be rigor ous
Have a clear proc ess for decid ing Use Failu re Mode and Effec ts Anal ysis (FME A)
Case Stud y
Feld man et al 2007
Cas e Stud y
Medi SPIC E Asses smen t1
Perfo rm a Syste m FME A only for high or mod erate LOCs Use Failu re Mode , Effec ts and Critic ality Anal ysis (FME CA) Use Fault Tree Anal ysis (FTA )
419
Case Stud y
Shoe make r& Van Shoo ende rwoe rt 2010
Feld man et al 2007
Ron kain en and Abr aha mss on 200 3
Perfor m Risk Revie w
Review and revise risk/ha zard analysi s Mitig ation s need to be fully trace able
Use Cross Functi onal Teams
Use predefine d templ ates
420
Assists by getting perspe ctives from multipl e stakeh olders Eases capture of risk data and assists in traceab ility
Creat e "neg ative " user stori es
Shoe make r& Van Shoo ende rwoe rt 2010 Shoe make r& Van Shoo ende rwoe rt 2010
Case Stud y
Refl ecti ve Anal ysis
Case Stud y
Refl ecti ve Anal ysis
Use a docu menta tion manag ement tool
ENG .5.5. SP1 1
Conduct risk evaluation and risk control activities
ISO 14971, IEC TR/80 002-1
Build Quality In
B,C
421
Use Cross Functi onal Teams
Used to secure docum ents and automa te workfl ow, includi ng remote collabo ration Cross functio nal teams assists in evaluat ing risk from various perspe ctives (ISO 14971 pg 40)
Refle ctive Anal ysis
Cas e Stud y
Case Stud y
Refl ecti ve Anal ysis
Mitig ation s need to be fully trace able
422
Add or chan ge relev ant docu ment ation, inclu ding the risk mana geme nt file. Reco rd who atten ds the meet ings
Shoe make r& Van Shoo ende rwoe rt 2010
Cas e Stud y
Appendix M – Analysis of the Literature Review Table M-41 The recurring themes and high level categories from the literature review Recurring Themes
Categor y(s)
Sources
Agile practices
People
Beck, 1999
Jeffries et al., 2001
Audit trail
Regulati ons
Vogel, 2010
Ambler & Kroll, 2007
Certificatio n
Lean
Chisholm, 2007
Vogel, 2010
Wils et al., 2006 Rottier & Rodrigue s, 2008 Hibbs, Jewett & Sullivan, 2009
Vogel, 2010
Kane et al, 2006
Communic ation
People People, Project Manage ment
Ambler & Kroll, 2007
Rottier & Rodrigue s, 2008
Continuou s integration
Software Practice
Cusumano, 2008
Tabaka & Martins, 2008 Poppendiec k& Poppendiec k, 2006
Vogel, 2010 Huffman Hayes, 2009
Paasivaara et al, 2009 Lin & Fan, 2009
Collaborati on
Customer access Document ation
People Software Practice
Pham et al, 2007
Sidky & Arthur, 2007
Lin & Fan, 2009
IEC, 2006
Roberts et al., 1998
Paasivaara et al, 2009
Raymond, 2001
Tabaka & Martins, 2008
Levy & Hazzan, 2009
Poppendi eck, 2003
Cordeiro et al, 2007 Hibbs, Jewett & Sullivan, 2009 Drobka et al, 2004
Spence, 2005 Vogel, 2010
Rasmuss en et al, 2009
Kane et al, 2006 Hibbs, Jewett & Sullivan, 2009
Poppend ieck, 2003 Hibbs, Jewett &
Drobka et al., 2004 Rasmussen et al, 2009
423
Gary et al., 2011
Grenning, 2002
Shoemaker & Van
Employee Expertise Employee Handoffs
Employee Motivation Employee Responsibi lty Employee Team configurati on Employees organised and motivated Face-toface communic ation
People Project Manage ment People, Organisa tion, Manage ment Organisa tion People, Manage ment People, Organisa tion, Manage ment
Drobka et al., 2004 Poppendiec k, 2003
Liker, 2003 Poppendiec k& Poppendiec k, 2006
Sullivan, 2009 Vogel, 2010
Hamel, 2006
Liker, 2003 Poppend ieck & Poppend ieck, 2006
Ambler & Kroll, 2007
Vogel, 2010
Ambler & Kroll, 2007
Hoegl, 2005
Poppend ieck, 2003
Ambler & Kroll, 2007
Poppendiec k& Poppendiec k, 2006
Ambler & Kroll, 2007 Poppendiec k& Poppendiec k, 2006
People
Vogel, 2010
Ambler & Kroll, 2007
Feedback loops
People
Vogel, 2010
Ambler & Kroll, 2007
Hardware
Project
Vogel, 2010
Ronkainen
Paasivaa ra et al, 2009 Poppend ieck, 2003
Schooender woert, 2010
Rottier & Rodrigues, 2008
Vogel, 2010
Levin, 2007
Hibbs, Jewett & Sullivan, 2009
Beck, 1999
IEC, 2006
Rasmussen et al, 2009
424
Drobka et al, 2004
Dependen cy
Leaders not managers Leadershi p Lean governanc e
Lean or agile Lean practices Legacy code Manageme nt ethos
Manage ment, Software Practice Project Manage ment, People Organisa tion Lean, Manage ment, Organisa tion Lean, Software Practice Lean, Software Practice
Manageme nt structure
Software Practice Manage ment Manage ment, Organisa tion
Measurem
Project
& Abrahamsso n, 2003
Ambler & Kroll, 2007
Poppendiec k& Poppendiec k, 2006
Ambler & Kroll, 2007
Kotter, 1995
Ambler & Kroll, 2007
Reinertsen, 2009
Ambler & Kroll, 2007
Konrad & Over, 2005
Robinson, 1997
Reinertsen, 2009
Kotter, 1995 Poppend ieck, 2003
Petersen & Wohlin, 2010 Poppend ieck, 2003 Rottier & Rodrigue s, 2008
ISO, 2003
Graaf et al, 2003 Cusumano, 2008
Ambler & Kroll, 2007
Hoegl, 2005
Hamel, 2006
Vogel, 2010
Poppendiec
Rasmuss
Vogel, 2010
IEC, 2006
Boehm & Turner, 2005
Smits & Rilliet, 2011
Abrahams son et al., 2002
Hibbs, Jewett & Sullivan, 2009
Thimbleby, 1988
Kniberg & Skarin, 2010
Anderson, 2010
Hiranabe, 2008
Poppendiec
Beck &
Jeffries et
Ambler &
425
Poppend ieck & Poppend ieck, 2006
IEC,
ents
Options thinking
Patterns
People Policies & Procedure s Prioritisati on
Process change Process control
Manage ment People, Software Practice Lean, Software Practice, Organisa tion
People Organisa tion, Project Manage ment Project Manage ment Organisa tion, People, Manage ment Project Manage ment, Software
Poppendiec k& Poppendiec k, 2006
Cordeiro et al, 2007 Poppendiec k& Poppendiec k, 2006
k& Poppendiec k, 2006
en et al, 2009
Walker, 2009
Highsmit h 2002
Kettunen & Laanti, 2005
Poppendiec k, 2003
Fowler, 2000
Kotter & Schlesing er, 1979
Small & Downey, 2001
Kotter, 1995
Cusuman o, 2008
Roberts et al., 1998
Drobka et al., 2004
Palmer & Felsing, 2002
Vogel, 2010
Boehm & Turner, 2005
Smits & Rilliet, 2011
al., 2001
Kroll, 2007
Mullins, 2004
Vogel, 2010
2006
Tsai et al., 2005 Srinivasan et al., 2009
Poppendiec k, 2003
Ambler & Kroll, 2007 Hibbs, Jewett & Sullivan, 2009
Kotter & Schlesinger, 1979
Small & Downey, 2001
Kotter, 1995
Bradley, 2007
Farrow & Greene, 2008
Rasmuss en et al, 2009
Vogel, 2010
k, 2003
Reinertsen, 2009
Thimbleby, 1988
426
Wils et al, 2006
Hiranabe, 2008
Highsmit h 2002
Highs mith & Cockb urn
Kettu nen & Laanti , 2005
Ambl er, 2006
Practice Project characteri stics
Organisa tion, Software Practice
Regulation s Regulation s Requireme nts traceabilit y Risk Manageme nt
Regulati ons Regulati ons Regulati ons, Software Practice Project Manage ment
Situational context Slack (build in or eliminate)
Organisa tion
Tool support Traceabilit y
Project Manage ment
Software Practice Regulati ons, Software Practice
2001 Shoemaker & Van Schooender woert, 2010
Drobka et al., 2004
ISO, 2003
IEC, 2006
Basel, 2004
SOX, 2002
Cockbur n, 2001 TheWorldBank, 2004 Ziegler, 2006
Vogel, 2010
Huffman Hayes, 2009
Rottier & Rodrigue s, 2008
Vogel, 2010 Ambler & Kroll, 2007 Poppendiec k& Poppendiec k, 2006
ISO, 2003 IEC, 2006
Boehm, 2002 Ambler & Kroll, 2007
Bush, 2007 ANSI/AAMI /IEC, 2006
Jordan, 2005
EU, 2007
Medi-SPICE, 2011
Boehm & Turner, 2003
Feldmann et al 2007
ISO 14971, 2007
Rottier & Rodrigues, 2008
Mueller & Borzuchow ski, 2002
Van Schooender woert & Morsicato, 2004
Middleton, 2001
Cusumano, 2008
Ambler & Kroll, 2007
Jeffries et al, 2001
Vogel, 2010
Huffman Hayes, 2009
Rottier & Rodrigue s, 2008
427
ISO, 2007
ISO, 2003
Verificatio n and validation Verificatio n and validation Verificatio n and validation
Lean, Software Practice Software Practice, Project Manage ment Software Practice
Rasmussen et al., 2009
Rasmussen et al., 2009 Van Schooender woert & Morsicato2 004
Fagan, 1976
Yoo et al 2005
Vogel, 2010
Jeffries et al., 2001 Huffman Hayes, 2009 Huffman Hayes, 2009
Cordeiro et al, 2007
Masticola & Gall, 2008
Mueller & Borzucho wski, 2002
Jeffries et al, 2001
Shoemaker & Van Schooender woert, 2010
Poppendi eck & Poppendi eck, 2006
Jeffries et al, 2001
Drobka et al, 2004
Vogel, 2010
428
Robinson, 1997
429
![[PdF] Download Lean Software Development - Google Sites](https://m.moam.info/img/260x300/pdf-download-lean-software-development-google-site_6478471c097c47a9708ca5f0.jpg)
