Members of'the Bootstrap Project E a m
BOOTSTRAP: EUROPE'S ASSESSMENT METHOD New views of mature ideas on software quality and productivity.
This isuei column was written by a univerdy-indumy team wwking under the auspiees of the European Strategic Proyamme fir Research in Injamation Technology. The authors adaptedad exfended the So$ware Enginewing Institute k &@are Procesr Assessment approach, then applied it to a numbw of'sojkvare organizations. The Boomap prvject shows an alterrutive path ofalolutionfor the SEI approach. -Dave Card
EVER SINCET H E SOETWAKEENGINEEKing Institute first published its Capability Maturity Model, process assessment and improvement have been of great concern. During the last three years, an ESPFUT project, Bootstrap, has made a serious attempt to apply the SEI model to the European softwareindustry. The project team- formed from a cornsomum of industrial companies and research institutes from Austria, Belgium, Finland, Germany, and Italy adopted and extended the o r i p a l SEI questionnaire with questions based on I S 0 9000 quality standards and the European Space Agency's process model. T h e res u 1tin g ass e s s men t methodology yields m o r e detailed capability profiles and maturity levels than the Capability Maturity Model and gives capability profiles of organizations and projects separately. For the most part the SEI's maturity levels are still recognizable, and the assessments are compatible.
I
PROJKT GOALS. Our main goal was to speed up the application of software-engineering technology in the European software industry. Starting from that, we developed a software-assessment methodology, "king the SEI's maturity model as a basic reference. We then applied our model to a number of European companies to define their maturity levels and capability profiles and to generate their improvement plans. The goals of our process assessment are to + Determine and present an analytlc quality profile for both a software-producing unit and its projects. An SPU can be the softw,are-engineering department or group of a company, a software house, or a part of the software house. T h e profile describes the capability features assessed to identify the strengths and weaknesses of the SPU and its projects, as well as to identify the differences hetween them. + On the basis of the identified strengths and weaknesses, form a plan of both short- and long-term actions for improvement. + Transform the action plan into a series of miniprojects to implement the recommended improvements. Thus, the Bootstrap assessment methodology describes the assessment process, determines where an organization stands (maturity level), identifies its strengths and weaknesses (capability), and offers improvement guidelines (action plans). To meet these goals, we looked at the SEI maturity model and decided that our assessment goals were similar enough to use it with a few extensions. We added IS0 9000 quality standards and the ESA process model (standard ESA-PSS 005) to complete the basis of our methodology. We adopted the I S 0 9000 quality standard for two reasons. First, it contained guidelines for an organization-wide quality system. The SEI model does not have these guidelines, probably because it is based mainly on the US Depamnent of Defense standard DoD 2 167A for developing defense-system software. Second, many European companies use IS0 9000 as a primary standard. We chose the ESA process-model standard be-
ASSESSMENTS ARE COMPA'BLE WITH THE SEI MODEL AND HAVE MORE DETAILED CAPABILITY PROFILES.
l
Editor: h i d Cord Computer StientesCorp. Centml Pork Two #700
4061 Powder Mill Rd. Coberton, MO 20705 Intanet
[email protected]
BACMGROUND. Bootstrap, which ended in February 1993, predated the European System and Software Initiative, which began in March 1903.ESSI is a technology-transfer program proposed for the Commission of European Countries to help software producers and users introduce new methods and tools for software development and maintenance. In Bootstrap, our mission was to invest in technology upgxades and generally lay the groundwork for ESSI. 7 b do that, we had to assess the capability levels of the organizations and their development and maintenance methods.
___ _____~_ _ ______ IEEE SOFTWARE
- - -~_ -.~--~___
93
phasizes continuous improvement through the involvement of all company personnel.
cause we considered it complete and general; it includes the features of previous intemational process-model standards, including those developed by the IEEE and DoD. It was not ideal but we could use it to model any company-specific process. We also wanted a process model sophsticated enough to senre as a solid background for developing our questionnaires.
QUESTIONNAIRES. il major part of the assessment is gathering data. I l k use two questiomaires, one to gather data about the SPU and the other to gather data on projects. Consultants fill in the questionnaires during converjations with representatives fiom the client organization and as they review the client’s quality handbooks, project documentation, and so on. T h e questions are divided into three qwlps: + Qi/c.viori.c.itbout tL7e oi~ailrzizatiou.Includes general data like internal structure and appli.. cation domains, data on the organization-wide quality system, and data on re-
~
1
I
rity levels, except that we wanted to express quamles inside those levels. Using quartiles let us express maturity more precisely. The algorithm also takes care of the three intensity levels of yes. It gives a certain maturity level for an SPU or a project when the score of answers from the questions within that level fits inside defined value limits. Like the SEI model, maturity levels start with level 1 and end with level 5. + Provide the Bootstrap method with a way to generate a capability profile that indicates the strengths and weaknesses of the SPU or project. The capability profile also shows the maturity levels of each capability factor. Through the capability profiles we can see which factors decrease the total maturity level and show the areas to be iinproved. Sometimes we can also see how the pure maturity level results do not tell the , whole truth about the “real” maturity because occasionally the prehious maturity , level was not fulfilled but the SPU or project had reached the next level.
ASSESSMENT STEPS. Our process-assessment and -improvement methodology includes + descriptions to guide the assessment, + questionnaires and an algorithm to de, termine maturity and capa1 bility, + guidelines for process improvement (standards for generating an action plan) + guidelines for establishing an assessor training + Viiestrons about the program and licensing polmethodology and engmeeang icy, and k?roil)-hm..Includes project + a database of Eurofunctions, processes independent of the software life MATURITY DETERMINATION. The questionpean companies that supports consultancy activities qcle (process and quality naires were designed to test a pamcular set of and the comparison of mamanagement), and life- criteria, which make up the SPU’s o r turity and capability levels qcle processes as defined project’s capability. There are a total of 2 1 of the assessed companies. by the-ESA process model. capability factors, and a set of questions is In most respects, our assessment and im+ Questioilr2sabout technologyand its wan$m devoted to each factor. We determine the provement process is quite similar to the , Includes technology introdudon, support of maturity level by applying the Bootstrap alSEI’S.There is one major exception. Assess- process functions, and support oftechnology gorithm to the set ofquestions. Organization ments are done only by companies external that is independent of and dependent on the and methodology are rated with the same I to 5 rating used by the SEI. Technology is to the client organization (consultant asses- software life cycle. Assessors scale answers to one of five val- rated A (low) or B (high). It also gives us a sors). We do not support or offer guidelines for self-assessment activities. ues: absent, weak, fair, extensive, and non- percentage score or absolute maturity level T h e Bootstrap methodology has three applicable. This rating system differs from for each capability factor in the strengthssteps: that of the SEI questionnaire, which requires and-weaknesses profile. The capability pro1. Preparation. W e present the back- you to ‘answeryes or no only. We believe that file provided to the SPU is first a single absoground of the Bootstrap methodology and by dividing yes into three intensity levels lute histogram and then a single r-elatiue the assessment process to the management (weak, fair, and intensive), we can more pre- histogram. The second histogram shows and senior technical staff of the client organi- cisely define the capability profiles and get how the assessed SPU performed in relation zation. 1 more realistic answers. The questions that to an overall mean score calculated for each 2. Planning. We meet with representa- don’t apply are left out as nonapplicable. capabilityusing data collected !?om the same tives of the client organization to select the I application area. This relative histogram target SPU and the projects to be assessed. ASSESSMENT ALGORITHM. Determining the generates a lot ofinterest because it lets those 3. On-site assessment. T h e assessors per- maturity level from the assessment results being assessed compare their good and bad i form the assessment according to Bootstrap ’ should he objective, and it should alwavs fol- practices in light ofwhat their potential comguidelines. This is not a one-shot event; fol- low the same rules. For these reasons, we petltors are d o n g low-up assessments are done to ensure that created an algorithm whose main goals are to the SPU continues to improve. M’e used the + Create a detailed approach for deterPRELIMINARY RESULTS. T h e assessments Plan-Do-Check Act cycle model developed mining the maturity level that guarantees a conducted over the last two and a half years 1 from Walter Shewhart’s qualiq-control good understanding of the SPU’s or project’s gave us the overall maturity levels of the principles and the Kaizen philosophy, a ho- inaturity and is compatible with the SEI ma- SPUs and projects assessed Figure 1 gves listic approach to problem solving that em- turity levels. \Ye wanted to use the SEI matu- the percentage dismbutlon across maturity
MATURITY LEVELS ARE
QUARTILES. RESPONSES ARE MORE THAN YES OR NO.
~
~~
1 ~1
I
’
~~
-
~~~~
I1
~ _ _ _ _ _ . _ _ _
MAY 1993
levels divided into quartiles for 49 projects 1 identify the correlation between the maturity search and service activities performed; + training Bootstrap assessors to guaranlevel and the effectiveness of the SPU or and 23 SPUs. T h e assessments also gave us the mean p;oject in terms of product quality and the tee that all assessrnentS will fulfill the samc standard of quality regardless of the countq values of key capabilities in the capability 1 risks and productivity of the process. + A method is needed to or organization of the assessor; and profiles of SPUs and pro+ managing the licensing policy. implement the action plans iects. However, because the T h e institute's ultimate p l is to adopt thc in the assessed companies. mean values of capability B o o m p aSSeSSment methodology for wideprofiles are confidential and CONCLUSIONS. W e fin- spread use within European organizations. 4 we count them in different ished the project in Februways for different purposes, ary 1993. Since then sev- ~ - ~ ~ _ _ _ we do not present them This a d & is limed 012 t e x t and delizei-able eral of the team members here. have formed a new insti- produced by the following memben of the Boot. T h e results are prelimitute, whose name is still strap project team Adnana Biceg-0 ofEtnotean nary and should be read being established, to man- SPA;Robnq Cacchia $Etnoteam SPA;Volkma only as samples of the reage t h e assessment Haase of the Technical University o f Graz sults. Also, the size of the method. T h e institute is in Giinther Koch o f 2 Infnmtics; Lech IG?tani/ sample is so small that we of CCC Cowrpunies;Pnsi Kuvqa of the Univeicharge of cannot base any conclu+ coniiwing research on software-pro- si9 of07dii; .Vla?-co jVfaiocchi of Etnoteam Spcj sions on it regarding Europe!an state oftheart Robert Laizcellotti $Etnoteam SPA;itifarc 1 cess assessmentand improvement; in s o h a r e production. + managing dathase service for collect- Maiocchi of Etnoteam SPA;Richard ~lilesmar As we suspected, the maturity level of the projects seems to be marpally better than ing the results of assessmen@and mppo'ting of the Zch?zicul Unnivennity of G a z ; Samu that of the SPUs (because new technology the Bootstmp consultancy by providing infor- Saukkmnz cf the Uninenity of Oulu; Wolfgan, matinn nn ~ ~ ~ r r ~ - n._. ~ ~ ~ m a h i n ~ a c c o.Srhunll r d i n of' g . Roheit Rosch CmbH., n d J%l~*l hitc nrnirctc firct) T h__.Y i s trpnrl i s what l r r l i i 9 to .r .__ '""y*",-'" ..__-__-.__I_.___.____ "-" 1 Simila of CCC Companies. I(Nv4u and Biceg use separate questionnaires for organization 1 to company size, business area, and so on; l + disseminating the results of the re- 1 did most ofthe work on this a7ticle and project levels in the first place.
I
I
PROJECT MEMBERS ARE CONTINUING ASSESSMENTS THROUGH A SEPARATE INSTITUTE.
' ~
'%,'U
~
_ I
_1
FEEDBACK. Of course, we med to gather as much feedback as possible to improve the assessment process. Because both the assessors and personnel from the client organimtiom participated in Bootstrap, we had an excellent opportunity to get commentS firsthand. We had considerable and valuable internal discussion that helped us forni development guidelines. Some of the positive comments about the assessmentswere + They are complete, consistent, and compatible with the SEI'SCapability Maturity Model and should be repeated every two years to constantly fouow organizational upgrades. + They motivate individuals to evaluate their own working methods and environments and stimulate new ideas on how to improve them. + They give a good SPU picture in a very short time, pointing to fundamental softwareengineering problems. + They lead to open discussions and a high degree of acceptance because there is an emphasis on confidentiality. Some of the criticisms were + The questionnaire should better sup-
IEEE SOFTWARE
1
1 i
0'
I 1
I
Q'
~-
~
04
1 1
'I
3
a3 A7
I
,
' ~
' i
I I
0'
$ I
I
0
,
5
I
10
.
,
15 20 Distribution [pertent)
25,
30
+ St
