Software(Development(( ... applied(in! software! development! companies?! RQ1! What!challenges!are! associated!with! ... Systema:c(experimenta:on(rare(.
So#ware(Development(( as(an(Experiment(System!
A!Qualita)ve!Survey!on!the!State!of!the!Prac)ce!
! Eveliina(Lindgren,!Jürgen!Münch! XP2015! May!26,!Helsinki!
Outline! • Mo)va)on!and!Related!Work! • Research!Ques)ons! • Methods!and!Data!Collec)on! • Findings! • Conclusions!
1!
Mo)va)on! Delivering Value(
Test!the!customer! value!of!an!idea! Learn!&!improve! Repeat!
2!
Related!Work! Stairway!to!Heaven:! Tradi)onal!!!Agile!&!Lean!!!Con)nuous!delivery!
So#ware(development(as(an(experiment(system( CompanyTspecific! approaches:!! MicrosoV,!Google,!etc.!
!HYPEX!
Different approaches
Con:nuous( Experimenta:on( 3!
Research!Ques)ons! RQ1! How(is(con:nuous( experimenta:on( applied(in! soVware! development! companies?!
RQ2! What!challenges!are! associated!with! con)nuous! experimenta)on?!
4!
Methods!and!Data!Collec)on! • Qualita:ve(survey( • SemiTstructured!individual!interviews! • Thema)c!coding!analysis!
• Ten(ICT(companies!opera)ng!in!Finland! • Small!(50%),!medium!(20%),!and!large!(30%)! • B2B!and!B2C!
• Thirteen(experienced(interviewees( 5!
RQ1:!Development!Prac)ces! • Agile(methods(and(prac:ces(prevailed! • Scrum,!Kanban,!Lean! • Con)nuous!integra)on!
• Pursuit(of(easy(deployments( • Release!cycle!
• Direct(customer(feedback(in(use! • Interviews,!surveys,!prototypes,!user!tes)ng…(
6!
RQ1:!Development!Prac)ces!(cont.)! • Product(usage(data!collected!by!55%! • OVen!coarseTgrained!and!difficult!to!analyze! • Most!wanted!to!improve!data!collec)on!
• Systema:c(experimenta:on(rare( • Con)nuous!experimenta)on!in!only!one!company! • Experimenta)on!considered!an!appropriate!approach!
Suppor:ng(prac:ces(exist,(but(they(need(to(be( integrated(into(a(con:nuous(experimenta:on( process(
7!
RQ2:!Key!Challenges! • Organiza)onal!Culture! • Product!Management! • Data!Management! ! • Resources!
8!
RQ2!Findings!(cont.)! Changing!culture! and!processes!
Reaching!the!right! customers! B2BT specific!
Organiza(onal+ challenges( Breaking!down! silos!(business,! BA,!PM,!R&D)!
Overcoming! reluctance!to! provide!feedback! B2BT ! specific!
9!
RQ2!Findings!(cont.)! Accelera)ng! release!cycle! speed!
Iden)fying!an! MVP!
Product+ management+ challenges( Iden)fying! hypotheses!&! metrics!
Priori)zing!the! backlog! 10!
RQ2!Findings!(cont.)! Quality!of!! data!analysis!
Availability!and! sharing!of!data!
Data(management( challenges(
Linking!data!and! decision!making!
Consent!to!collect! product!! usage!data! B2BT specific!
11!
Findings:!Validity!Considera)ons! • External!validity! • Limited!scope! " Variety!of!companies,!experienced!interviewees!
• Construct!validity! • Possible!misunderstandings! " Central!concept!shared!with!par)cipants,!! clarifying!ques)ons!during!and!aVer!interviews!
• Reliability! • Replica)on!of!a!qualita)ve!study! " Interview!guide!and!analy)c!codebook!
12!
Conclusions! State!of!the! Systema)c!experimenta)on!is!rare! prac)ce! Suppor)ng!prac)ces!exist,!! but!they!need!to!be!integrated!into!! a!con)nuous!experimenta)on!process! Prac))oners!interested!in!the!approach! Challenges!
Developing!organiza)onal!capabili)es! Achieving!faster!releases! Establishing!dataTdriven!decision!making!
13!
Con)nuous!Experimenta)on!! is!a!change!of!mindset.! ! !
{eveliina.lindgren,!juergen.muench}@cs.helsinki.fi! www.juergenmuench.com! !
References! Holmström!Olsson,!H.,!Alahyari,!H.,!Bosch,!J.:!Climbing!the!“Stairway!to!Heaven”:!A! Mul)pleTCase!Study!Exploring!Barriers!in!the!Transi)on!from!Agile!Development!towards! Con)nuous!Deployment!of!SoVware.!In:!38th!EUROMICRO!Conference!on!SoVware! Engineering!and!Advanced!Applica)ons!(SEAA),!pp.!392–399.!IEEE!Press!(2012)! ! Holmström!Olsson,!H.,!Bosch,!J.:!The!HYPEX!Model:!From!Opinions!to!DataTDriven! SoVware!Development.!In:!Con)nuous!SoVware!Engineering!Part!IV,!!pp!155T164.! Springer!(2014)! ! Kohavi,!R.,!Deng,!A.,!Frasca,!B.,!Walker,!T.,!Xu,!Y.,!Pohlmann,!N.:!Online!Controlled! Experiments!at!Large!Scale.!In:!19th!ACM!SIGKDD!Interna)onal!Conference!on!Knowledge! Discovery!and!Data!Mining,!pp.!1168–1176.!ACM,!New!York!(2013)! ! Steiber,!A.,!Alänge,!S.:!A!Corporate!System!for!Con)nuous!Innova)on:!The!Case!of!Google! Inc.!European!Journal!of!Innova)on!Management,!16,!2,!pp.!243–264!(2013)! ! Fagerholm,!F.,!Guinea,!A.S.,!Mäenpää,!H.,!Münch,!J.:!Building!Blocks!for!Con)nuous! Experimenta)on.!In:!1st!Interna)onal!Workshop!on!Rapid!Con)nuous!SoVware! Engineering,!pp.!26–35.!ACM,!New!York!(2014)! ! Please!see!conference!paper!for!further!references!!
15!
