Issues of Transparency, Testing and Validation in the ...

S NE O

V E R V I E W

N

O T E

Issues of Transparency, Testing and Validation in the Development and Application of Simulation Models David J. Murray-Smith University of Glasgow, School of Engineering, Rankine Building, Glasgow G12 8QQ, Scotland, United Kingdom; [email protected] 6LPXODWLRQ1RWHV(XURSH61(  '2,VQHRQ 5HFHLYHG-XQH,QYLWHG2YHUYLHZ1RWH  $FFHSWHG-XQH

Abstract. The importance of verification, validation and documentation of simulation models is widely recognised, at least in principle. However, in practice, inadequate model management procedures can lead to insufficient information being available to allow a model to be applied with confidence or for it to be re-used without difficulty and much additional effort. The ease with which a model can be understood by someone not involved in its development depends on the transparency of the model development process. This paper reviews ideas associated with transparency and model management. It also includes discussion of some related issues that are believed to be particularly important, such as identifiability and experimental design for model validation. Some recent developments in engineering applications and in physiological and health-care modelling are discussed, along with the responsibilities of the academic community in giving more emphasis to simulation model testing and transparency.

Introduction 7KH WHVWLQJ YDOLGDWLRQ DQG GHWDLOHG GRFXPHQWDWLRQ RI VLPXODWLRQPRGHOVDUHDOOLPSRUWDQWLVVXHVLQHYHU\ILHOG LQZKLFKPRGHOOLQJDQGVLPXODWLRQWHFKQLTXHVDUHXVHG ,WLVDOVRZHOOHVWDEOLVKHGWKDWWKHGHYHORSPHQWRIVLPX ODWLRQ PRGHOV LQYROYHV DQ LWHUDWLYH SURFHVV DQG WKDW PRGHO WHVWLQJ LV LQVHSDUDEOH IURP DOO RWKHU DVSHFWV RI PRGHOGHYHORSPHQW2QHYHU\LPSRUWDQWPLOHVWRQHZDV WKH SURGXFWLRQ LQ  RI WKH JURXQGEUHDNLQJ UHFRP PHQGDWLRQVRIWKH6&67HFKQLFDO&RPPLWWHHRQ0RGHO &UHGLELOLW\>@

7KLVSURYLGHGDXVHIXOVHWRIFRQYHQWLRQVDQGGHILQL WLRQV IRU XVH LQ GLVFXVVLQJ WKH GHYHORSPHQW RI PRGHOV DQG WKH PDLQ PHVVDJH DERXW WKH LPSRUWDQFH RI PRGHO WHVWLQJDQGWKHLWHUDWLYHQDWXUHRIPRGHOOLQJDQGVLPXOD WLRQSURFHVVHVKDVVLQFHEHHQHPSKDVLVHGUHSHDWHGO\E\ PDQ\RWKHUVVXFKDV6DUJHQWHJ>@ gUHQHJ>@  %DOFLHJ>@ DQG%UDGHHJ>@  7UDQVSDUHQF\ LQ VLPXODWLRQ PRGHO GHYHORSPHQW LV DQRWKHU FORVHO\UHODWHG LVVXH DQG LV FRQFHUQHG ZLWK WKH HDVHZLWKZKLFKDPRGHOLWVDVVRFLDWHGVLPXODWLRQVRIW ZDUH DQG WHVWLQJ SURFHGXUHV FDQ EH XQGHUVWRRG E\ VRPHRQHQRWLQYROYHGLQLWVGHYHORSPHQW7KLVUHODWHVWR TXHVWLRQVRIPRGHOPDQDJHPHQWLQJHQHUDODQGWRGRF XPHQWDWLRQLQSDUWLFXODU ,QWKHFRQWH[WRIHQJLQHHULQJDSSOLFDWLRQVDVLPXOD WLRQ PRGHO WKDW LV XVHG LQ WKH GHVLJQ RI D QHZ SURGXFW PD\ ZHOO KDYH D FRQWLQXLQJ DQG VLJQLILFDQW UROH WKURXJKRXWWKHFRPSOHWHOLIHF\FOHRIWKDWSURGXFW7KH FRQWLQXLQJ YDOXH RI D PRGHO ZLOO GHSHQG RQ KRZ WKDW PRGHOLVPDQDJHGIURPWKHRXWVHWLQFOXGLQJWKHWHVWLQJ VWUDWHJ\ YHULILFDWLRQ DQG YDOLGDWLRQ SURFHVVHV DQG WKH DYDLODELOLW\RIXVHIXOLQIRUPDWLRQDERXWDOODVSHFWVRIWKH PRGHODQGLWVGHYHORSPHQW>@ ,QDOODSSOLFDWLRQDUHDVWKHLQFUHDVLQJXVHRIREMHFW RULHQWHG VRIWZDUH WKH LQWHJUDWLRQ RI VLPXODWLRQ WRROV ZLWK RWKHU IRUPV RI VSHFLDOLVW VRIWZDUH WKH DYDLODELOLW\ RI OLEUDULHV RI VXEPRGHOV DQG WKH GHYHORSPHQW RI JH QHULFVLPXODWLRQPRGHOVLQWHQGHGIRUXVHLQDQXPEHURI GLIIHUHQWDSSOLFDWLRQVDUHW\SLFDORIZD\VLQZKLFKPRG HOOLQJ DQG VLPXODWLRQ PHWKRGV KDYH EHHQ FKDQJLQJ LQ UHFHQW \HDUV >@  7KHVH FKDQJHV RI DSSURDFK VKRXOG KDYHEHHQDFFRPSDQLHGE\GHWHUPLQHGHIIRUWVWRHQVXUH WUDQVSDUHQF\ ZLWKLQ WKH GHYHORSPHQW SURFHVV WKURXJK FDUHIXO DQG V\VWHPDWLF GRFXPHQWDWLRQ WRJHWKHU ZLWK PRUH ULJRURXV SURFHGXUHV IRU WHVWLQJ YHULILFDWLRQ DQG YDOLGDWLRQ SNE 26(2) – 6/2016 57

Murray-Smith

Transparency, Testing and Validation for Simulation Models

+RZHYHUWKHUHLVFOHDUHYLGHQFHWKDWWKHVHLVVXHVDUH VWLOOQRWEHLQJJLYHQDSSURSULDWHDWWHQWLRQE\VRPHPRG HO GHYHORSHUV XVHUV DQG DOVR WKRVH LQYROYHG LQ HGXFD WLRQ>@,VVXHVUHODWLQJWRPRGHOTXDOLW\VXFKDVWKH UDQJHRIDSSOLFDELOLW\RIDJLYHQPRGHOKRZWKHEHKDY LRXURIDPRGHOFDQEHFRPSDUHGZLWKWKHEHKDYLRXURI WKHFRUUHVSRQGLQJUHDOV\VWHPWKHSUHFLVHH[SHULPHQWDO FRQGLWLRQV XVHG LQ FRPSDULQJ VLPXODWLRQ UHXOWV ZLWK UHDOZRUOGGDWDDQGDYDLODELOLW\RIRIGRFXPHQWDWLRQDUH VHOGRPHPSKDVLVHGLQSXEOLVKHGPDWHULDO 7KLV VLWXDWLRQ LV PDGH ZRUVH E\ WKH IDFW WKDW PDQ\ SXEOLVKHGMRXUQDOSDSHUVDQGUHSRUWVGHDOLQJZLWKPRG HOOLQJDQGVLPXODWLRQDSSOLFDWLRQVODFNUHDOWUDQVSDUHQF\ LQ WHUPV RI WKH PRGHO DQG WKH VLPXODWLRQ PHWKRGRORJ\ EHLQJXVHGDQGDOVRLQFOXGHOLWWOHDERXWWHVWLQJDQGYDOL GDWLRQ UHVXOWV +RZ FDQ VWXGHQWV EH H[SHFWHG WR   JLYH PRUH DWWHQWLRQ WR WKHVH LVVXHV ZKHQ WKH\ VHH WKDW WKHLU OHFWXUHUVDQGSURIHVVRUVDUHVWLOOVXFFHVVIXOO\SXEOLVKLQJ UHVHDUFK SDSHUV LQ ZKLFK VFDQW DWWHQWLRQ DSSHDUV WR EH JLYHQWRWKHVHWRSLFV" $QRWKHUUHODWHGLVVXHLVWKDWLQVRPHRUJDQLVDWLRQV VLPXODWLRQPRGHOOLQJLVRIWHQWUHDWHGLQDGLIIHUHQWZD\ IURPRWKHUVRIWZDUHGHYHORSPHQWSURFHVVHV6LPXODWLRQ PRGHOV DUH RIWHQ GHYHORSHG ZLWKRXW WKH EHQHILWV RI YHUVLRQ FRQWURO SURFHGXUHV WKDW DUH D FHQWUDO IHDWXUH RI PRVW VRIWZDUH GHYHORSPHQW HQYLURQPHQWV 6KRXOG PHWKRGV RI DSSURDFK WR YHUVLRQ FRQWURO WKDW KDYH EHHQ XVHG VXFFHVVIXOO\ LQ WKH VRIWZDUH HQJLQHHULQJ ILHOG EH DSSOLHGWRWKHGHYHORSPHQWDQGPDQDJHPHQWRIVLPXOD WLRQPRGHOV" 7KHUHLVDOVRDFOHDUQHHGWRFRQVLGHUZK\ULJRURXV DQG V\VWHPDWLF WHVWLQJ WHFKQLTXHV WKDW DUH FXUUHQWO\ DYDLODEOH DUH QRW EHLQJ DSSOLHG PRUH ZLGHO\ DQG ZK\ WKHGRFXPHQWDWLRQRIPRGHOVLVWRRRIWHQVXSHUILFLDORU QRQH[LVWHQW 7KH FRQVHTXHQW SUREOHPV WKDW IROORZ RQ IURPWKHVHIDLOLQJVDOVRQHHGWREHFRQVLGHUHGWRJHWKHU ZLWK FRQVLGHUDWLRQ RI WKH DGGHG VKRUWWHUP FRVWV RI DGRSWLQJDPRUHULJRURXVDSSURDFKRIWKLVNLQG

1 The Need for Good Model Management $VPHQWLRQHGDERYHTXDOLW\DVVXUDQFHPHFKDQLVPVDQG PRGHOPDQDJHPHQWSURFHGXUHVDUHRIWHQQRWDEO\DEVHQW IURP WKH PRGHOOLQJ DQG VLPXODWLRQ SURFHVVHV ZLWKLQ PDQ\RUJDQLVDWLRQV 58 SNE 26(2) – 6/2016

ON

:KHWKHUPRGHOVDUHGHYHORSHGIURPILUVWSULQFLSOHV RU UHLQVWDWHG IURP SUHYLRXV SURMHFWV RU DFTXLUHG IURP RWKHURUJDQLVDWLRQVDSSURSULDWHVWUDWHJLHVQHHGWREHLQ SODFH WR PDNH VXUH WKDW PRGHO TXDOLW\ LVVXHV DUH DG GUHVVHG SURSHUO\ DQG WKDW DOO DVSHFWV RI WKH PRGHOOLQJ DQGVLPXODWLRQDFWLYLWLHVDUHPDQDJHGLQDQDSSURSULDWH IDVKLRQ >@ >@ 7KLV LV LPSRUWDQW LQ DOO VFLHQWLILF PHGLFDO DQG GHFLVLRQ PDNLQJ DSSOLFDWLRQ DUHDV DV ZHOO DVLQHQJLQHHULQJ 2QH LPSRUWDQW DVSHFW RI PRGHO PDQDJHPHQW LQ YROYHV WKH HVWDEOLVKPHQW RI D VXLWDEOH SODQ IRU PRGHO YHULILFDWLRQ YDOLGDWLRQ DQG DSSURYDO DQG WKLV QHHGV WR EHVWDUWHGDWWKHHDUOLHVWVWDJHVRIDPRGHOGHYHORSPHQW SURMHFW6XFKDSODQVKRXOGOD\GRZQFOHDUTXDQWLWDWLYH UHTXLUHPHQWV IRU WKHYDOLGDWLRQIRU HDFK HOHPHQW RI WKH PRGHO EXW ZLWK IOH[LELOLW\ LQ WHUPV RI PHWKRGV WR EH XVHG )RU H[DPSOH SUHOLPLQDU\ DVVHVVPHQW RI D PRGHO RQWKHEDVLVRIIDFHYDOLGDWLRQPD\EHIROORZHGE\WKH DSSOLFDWLRQ RI D PRUH TXDQWLWDWLYH DSSURDFK DV FRQIL GHQFH LQFUHDVHV :KHUH PRGHO OLEUDULHV DUH EHLQJ DS SOLHG YDOLGDWLRQ LQIRUPDWLRQ DERXW VXEPRGHOV VKRXOG EH EURXJKW WRJHWKHU XVLQJ WKH DYDLODEOH GRFXPHQWDWLRQ DQG UHYLHZHG LQ D FULWLFDO IDVKLRQ LQ WKH FRQWH[W RI WKH LQWHQGHG DSSOLFDWLRQ 7KH SODQ KDV WR SURYLGH JXLGDQFH IRU HYHU\ DVSHFW RI PRGHO WHVWLQJ DQG KDV WR EH UHFRJ QLVHGDVEHLQJTXLWHGLIIHUHQWDQGGLVWLQFWIURPWKHPRG HO VSHFLILFDWLRQ ,Q DGGLWLRQ WKH SODQ IRU YHULILFDWLRQ YDOLGDWLRQ DQG DSSURYDO VKRXOG RXWOLQH WKH PHWKRGV WR EH DGRSWHG DQG SURYLGH LQIRUPDWLRQ DERXW UHVRXUFHV DYDLODEOH IURP SUHYLRXV SURMHFWV VXFK DV IXOO\WHVWHG VLPXODWLRQPRGHOVRUVXEPRGHOV 6LQFHWKHPRGHOOLQJSURFHVVLVLWHUDWLYHWKHYHULILFD WLRQ YDOLGDWLRQ DQG DFFHSWDQFH SODQ PD\ LWVHOI KDYH WR EHPRGLILHGDQGILQHWXQHGDVWKHZRUNSURJUHVVHV)RU H[DPSOH DV H[SHULHQFH JURZV IURP DQDO\VLV RI WHVW UHVXOWV IURP WKH UHDO V\VWHP WUDGHRIIV PD\ EHFRPH QHFHVVDU\ EHWZHHQ WKH YDOLGDWLRQ UHTXLUHPHQWV DQG WKH TXDQWLW\ RI DGGLWLRQDO YDOLGDWLRQ GDWD QHHGHG (YHQ LI FROOHFWLRQ RI WHVW GDWD IURP WKH UHDO V\VWHP LV WKH UH VSRQVLELOLW\ RI RWKHUV WKRVH LQYROYHG LQ WKH PRGHO GH YHORSPHQW SURFHVV VKRXOG EH DEOH WR FRQWULEXWH WR WKH SODQQLQJRIWKHPRGHOWHVWLQJSURFHVVVLQFHHDUO\H[SH ULHQFHZLWKDVLPXODWLRQPRGHOPD\SURYLGHLQVLJKWWKDW LQIOXHQFHV WKH GHVLJQ RI H[SHULPHQWV $Q H[DPSOH RI WKLVLVWKHIDFWWKDWXQGHUVWDQGLQJRISDUDPHWHUVHQVLWLYL W\LVVXHVEXLOWXSGXULQJWKHPRGHOGHYHORSPHQWSURFHVV FDQ FRQWULEXWH VLJQLILFDQWO\ WR H[SHULPHQWDO GHVLJQ DQG WRTXHVWLRQVUHODWLQJWRPRGHOXQFHUWDLQWLHV

ON

Murray-Smith

Transparency, Testing and Validation for Simulation Models

2Q WKH RWKHU KDQG WKH PDQDJHPHQW V\VWHP VKRXOG HQVXUH WKDW ZKHUHYHU SRVVLEOH WKH DVVHVVPHQW RI WHVW UHVXOWVDQGPRGHOUHVXOWVVKRXOGLQYROYHEOLQGFRPSDUL VRQV ,QLWLDOO\ H[SHULPHQWDOLVWV VKRXOG SURYLGH WKH VLPXODWLRQ GHYHORSHUV RQO\ ZLWK PHDVXUHG LQSXW GDWD DQG RWKHU UHOHYDQW LQIRUPDWLRQ DERXW WHVW FRQGLWLRQV 7KH VLPXODWLRQ JURXS ZRXOG WKHQ XVH WKH LQSXW GDWD SURYLGH E\ WKH H[SHULPHQWDOLVWV WR PDNH SUHGLFWLRQV RI FRUUHVSRQGLQJRXWSXWYDULDEOHV([SHULPHQWDODQGVLPX ODWLRQ UHVXOWV FRXOG WKHQ EH FRPSDUHG E\ PHPEHUV RI ERWK JURXSV ZRUNLQJ WRJHWKHU DQG FRQFOXVLRQV UHDFKHG DERXWWKHDGHTXDF\RURWKHUZLVHRIWKHPRGHO 'RFXPHQWDWLRQ LV DQRWKHU DUHD ZKHUH VWULFW PDQ DJHPHQWSURFHGXUHVDUHLPSRUWDQW$ZDUHQHVVRIPRGHO OLPLWDWLRQV LQ WKH PLQGV RI XVHUV LQHYLWDEO\ IDGHV ZLWK WLPH DQG DFFXUDWH DQG HDVLO\ DFFHVVLEOH GRFXPHQWDWLRQ LV HVVHQWLDO 7KLV VKRXOG GHDO ZLWK DOO DVSHFWV RI WKH PRGHO LQFOXGLQJ LWV SXUSRVH DVVXPSWLRQV DQG VLPSOLIL FDWLRQV GHWDLOV RI YHULILFDWLRQ DQG YDOLGDWLRQ WHVWV DQG DQ DVVHVVPHQW RI WKH UDQJHRI FRQGLWLRQVIRU ZKLFK WKH ILQDO DFFHSWHG PRGHO FDQ EH XVHG 7KH GRFXPHQWDWLRQ VKRXOGDOVRLQFOXGHLQIRUPDWLRQDERXWWKHPRGHOYDULD WLRQV GHYHORSHG GXULQJ WKH SURMHFW WRJHWKHU ZLWK UHD VRQVIRUDFFHSWLQJRUUHMHFWLQJHDFKRIWKHVH>@ 1.1 Model management practices 5HOLDEOH DQG HDVLO\XVHG PHWKRGV IRU PRGHO YHUVLRQ FRQWURODUHHVVHQWLDOIRUODUJHDQGFRPSOH[PRGHOV7KLV LV SDUWLFXODUO\ LPSRUWDQW ZKHQ PRGHOV DUH GHYHORSHG DQGPDLQWDLQHGE\DWHDPUDWKHUWKDQE\DQLQGLYLGXDO ,QWHUDFWLRQVEHWZHHQPRGHOVRIGLIIHUHQWW\SHVFDQDOVR EH DQ LPSRUWDQW IHDWXUH RI D SURMHFW DQG LW LV LPSRUWDQW WRHQVXUHWKDWQRGDWDWUDQVIHUHUURUVFDQRFFXUEHWZHHQ PRGHOV,QDGGLWLRQLWLVHVVHQWLDOWRHQVXUHWKURXJKDQ DSSURSULDWHPDQDJHPHQWV\VWHPWKDWZKHQHYHUFKDQJHV DUH PDGH DOO RI WKH UHOHYDQW PRGHOV DUH XSGDWHG DW WKH VDPHWLPH ,W KDV EHHQ VXJJHVWHG WKDW PRGHO GRFXPHQWDWLRQ VKRXOGDOZD\VEHGLYLGHGLQWRWZRGLVWLQFWVHFWLRQVHJ >@ 7KHILUVWZRXOGLQYROYHQRQWHFKQLFDOGRFXPHQWD WLRQDQGZRXOGEHDFFHVVLEOHE\DOOKDYLQJVRPHLQWHUHVW LQ WKH PRGHO DQG LWV DSSOLFDWLRQV 7KLV VHFWLRQ RI WKH GRFXPHQWDWLRQZRXOGLQFOXGHDQRYHUYLHZRIWKHPRGHO LQWHUPVRILWVSXUSRVHLQWHQGHGDSSOLFDWLRQVYDULDEOHV HTXDWLRQV SDUDPHWHUV HWF WRJHWKHU ZLWK D VXPPDU\ RI WKH YHULILFDWLRQ DQG YDOLGDWLRQ SURFHGXUHV GHWDLOHG YHULILFDWLRQ DQG YDOLGDWLRQ UHVXOWV DQG UHVXOWLQJ UHFRP PHQGDWLRQV LQ WHUPV RI WKH UDQJH RI FRQGLWLRQV RYHU

ZKLFK WKH PRGHO FRXOG EH XVHG 7KH VHFRQG VHFWLRQ ZRXOGSURYLGHDOOWKHDGGLWLRQDOLQIRUPDWLRQWKDWZRXOG EH UHTXLUHG E\ VRPHRQH ZLVKLQJ WR PDNH XVH RI WKH PRGHORUWRUHFRQVWUXFWWKHPRGHODQGUHSURGXFHUHVXOWV WKDW KDG EHHQ REWDLQHG SUHYLRXVO\ 6SOLWWLQJ WKH GRFX PHQWDWLRQ LQ WKLV ZD\ PHDQV WKDW DVSHFWV RI D PRGHO UHJDUGHG DV EHLQJ LQ VRPH ZD\ FRQILGHQWLDO FRXOG EH KHOGEDFNZKLOHVWLOOSURYLGLQJLQWHUHVWHGSDUWLHVZLWKD EURDGRXOLQHRIWKHPRGHODQGLWVFDSDELOLWLHV 7KH FKRLFH RI VRIWZDUH WRROV WR DVVLVW LQ WKH PDQ DJHPHQWRIPRGHOVVLPXODWLRQSURJUDPVDQGWKHGRFX PHQWDWLRQGHSHQGVRQWKHFRPSXWLQJHQYLURQPHQWEHLQJ XVHG'HWDLOVRIWKHV\VWHPVIRUNHHSLQJWUDFNRIPRGHO YHUVLRQVVLPXODWLRQSURJUDPVSDUDPHWHUYDOXHVYDOLGD WLRQ GDWD VHWV DQG UHVXOWV DUH DOVR JRLQJ WR EH GLIIHUHQW IRU GLIIHUHQW W\SHV RI RUJDQLVDWLRQ )RU H[DPSOH LQ DFDGHPLF HQYLURQPHQWV ODUJH UHVHDUFK JURXSV PD\ EHQHILWIURPUHODWLYHO\ULJLGDQGFHQWUDOLVHGV\VWHPVIRU PRGHOYHUVLRQFRQWURO2QWKHRWKHUKDQGDQLQGLYLGXDO UHVHDUFKHU ZRUNLQJ ZLWK RQH RU WZR UHVHDUFK VWXGHQWV PD\ ILQG DGYDQWDJHV LQ D VLPSOHU DQG OHVV IRUPDO DS SURDFK ZKLFK MXVW LQYROYHV HVWDEOLVKLQJ D V\VWHPDWLF ZD\ RI NHHSLQJ WUDFN RI GLIIHUHQW PRGHO YHUVLRQV RI OLQNLQJWKHPWRWKHDSSURSULDWHGDWDVHWVDQGWRWKHFRU UHVSRQGLQJVLPXODWLRQSURJUDPV5HVXOWVRIYHULILFDWLRQ DQG YDOLGDWLRQ WHVWV PXVW DOVR EH UHDGLO\ DYDLODEOH DQG EHHDVLO\OLQNHGWRHDFKPRGHOYHUVLRQ,QODUJHRUJDQL VDWLRQV WKH ZD\ LQ ZKLFK WKLV FRQWURO FDQ EH DFKLHYHG DUH FOHDUO\ YHU\ GLIIHUHQW HVSHFLDOO\ ZKHQ WHDPV DUH JHRJUDSKLFDOO\GLVSHUVHGZKHQDPRUHIRUPDOV\VWHPRI PDQDJHPHQWEHFRPHVUHDOO\LPSRUWDQW 1.2 Benefits versus costs in model management 2QHHYHUSUHVHQWLVVXHWKDWKDVWREHFRQVLGHUHGLQDQ\ RUJDQLVDWLRQ ZKDWHYHU LWV VL]H FRQFHUQV WKH FRVWV RI HVWDEOLVKLQJ V\VWHPV RI PRGHO PDQDJHPHQW ZKLFK LQ YROYH FRPSUHKHQVLYH YHULILFDWLRQ DQG YDOLGDWLRQ SURFH GXUHV DQG ODUJH DPRXQWV RI GRFXPHQWDWLRQ 7KH UHFXU UHQWH[SHQVHVDVVRFLDWHGZLWKVXFKV\VWHPVFDQEHFRQ VLGHUDEOH +RZHYHU WKH FRVWV UHVXOWLQJ IURP IDLOXUH WR HVWDEOLVK DSSURSULDWH V\VWHPV IRU WKH PDQDJHPHQW RI PRGHOVFDQEHPXFKJUHDWHU,QDQHQJLQHHULQJDSSOLFD WLRQIRUH[DPSOHWKHXVHRIDQLQDSSURSULDWHPRGHOIRU GHVLJQ SXUSRVHV PD\ OHDG WR YHU\ ODUJH DPRXQWV RI XQSODQQHG H[SHQGLWXUH ZKHQ UHGHVLJQ EHFRPHV QHFHV VDU\ 7KH ODWHU LQ WKH GHVLJQ SURMHFW WKDW WKH SUREOHPV DUHGLVFRYHUHGWKHODUJHUWKHFRVWVRIUHFWLILFDWLRQ SNE 26(2) – 6/2016 59

Murray-Smith

Transparency, Testing and Validation for Simulation Models

3URWRW\SHV WKDW IDLO WR PHHW SHUIRUPDQFH VSHFLILFD WLRQV LQHYLWDEO\ OHDG WR WLPHFRQVXPLQJ DQG H[SHQVLYH FKDQJHV WR KDUGZDUH DQG VRIWZDUH 6LPLODUO\ LQ DUHDV VXFK DV VFLHQWLILF UHVHDUFK WKH XVH RI DQ LQDSSURSULDWH PRGHOFDQUHVXOWLQIDOVHFRQFOXVLRQVDQGSRVVLEO\LQFRU UHFW GHFLVLRQV LQ WHUPV RI VXEVHTXHQW GLUHFWLRQV RI UH VHDUFKRUSROLF\UHFRPPHQGDWLRQV 3URSHUPRGHOPDQDJHPHQWSURFHGXUHVPRGHOWUDQV SDUHQF\ DQG GRFXPHQWDWLRQ DUH DOVR YHU\ LPSRUWDQW EHFDXVHNQRZOHGJHDERXWPRGHOVZKLFKUHVLGHVRQO\LQ WKH KHDGV RI PRGHO GHYHORSHUV LV OLNHO\ WR EH ORVW DV VRRQ DV WKRVH LQGLYLGXDOV PRYH RQ WR QHZ DUHDV RI UH VSRQVLELOLW\RUWRDGLIIHUHQWRUJDQLVDWLRQ6XFKDORVVLV FOHDUO\ YHU\ ZDVWHIXO 7KH DFDGHPLF ZRUOG SDUWLFXODUO\ ZHDNLQWKLVUHVSHFWEHFDXVHZRUNFDUULHGRXWE\JUDGX DWHVWXGHQWVDW0DVWHU¶VGHJUHHDQG3K'OHYHOLVUHFRUG HGPDLQO\WKRXJKGLVVHUWDWLRQVDQGWKHVHVHOGRPLQFOXGH WKH OHYHO RI GHWDLO DERXW PRGHOV WKDW LV QHFHVVDU\ WR EXLOG XSRQ ZKDW KDV EHHQ DFKLHYHG :KHUH PRGHOOLQJ IRUPV DQ LPSRUWDQW DVSHFW RI WKH ZRUN EHLQJ UHSRUWHG VRPH VHSDUDWH DUFKLYLQJ V\VWHP VKRXOG EH UHTXLUHG WR VXSSOHPHQWDQGVXSSRUWWKHLQIRUPDWLRQSURYLGHGLQWKH SXEOLVKHG WKHVLV DQG WKLV VKRXOG EH DFFHVVLEOH E\ DOO ZKRKDYHDFFHVVWRWKHWKHVLV 2QHZD\RIDWWHPSWLQJWRFRQWUROWKHFRVWVRIPRGHO YDOLGDWLRQLVWKURXJKWKHHVWDEOLVKPHQWRIDOLQNEHWZHHQ WKHYHULILFDWLRQYDOLGDWLRQDQGDFFHSWDQFHSODQDQGWKH PRUH JHQHUDO UHTXLUHPHQWV DQDO\VLV GRFXPHQW WKDW GH WDLOVWKHSXUSRVHRIWKHPRGHOLWVDFFXUDF\UHTXLUHPHQWV DQGGHILQHVWKHEURDGVWUDWHJ\IRULWVGHYHORSPHQW7KDW UHTXLUHPHQWV DQDO\VLV GRFXPHQW FDQ WKHQ SURYLGH WKH EDVLVIRUDSURMHFWSODQZKLFKLQFOXGHVHVWLPDWHVRIWKH KXPDQHIIRUWDQGFDQLQGLFDWHKRZWDVNVLQYROYHGLQWKH PRGHO GHYHORSPHQW SURFHVV FDQ EH VSOLW EHWZHHQ LQGL YLGXDOV(VWDEOLVKPHQWRIDSODQRIWKLVNLQGWKDWSODFHV GXH HPSKDVLV RQ PRGHO YDOLGDWLRQ VKRXOG DOVR DOORZ FRQILGHQFH WR EH EXLOW XS DERXW WKH ILWQHVV RI D PRGHO IRU LWV LQWHQGHG DSSOLFDWLRQ ZKLOH DW WKH VDPH WLPH DOORZLQJWKHRYHUDOOFRVWWREHPRQLWRUHGFRQWLQXRXVO\ ,W LV DOZD\V GLIILFXOW WR REWDLQ LQIRUPDWLRQ DERXW SURFHGXUHV IRU PRGHO PDQDJHPHQW DGRSWHG HOVHZKHUH DQG WKLV LV HVSHFLDOO\ WUXH LQFRPPHUFLDO RUJDQLVDWLRQV +RZHYHU D QXPEHU RI VWXGLHV KDYH EHHQ FDUULHG RXW 2QHRIWKHVHLVDQLQYHVWLJDWLRQE\)RVVHWDO>@ZKLFK UHODWHV WR V\VWHP PRGHOOLQJ DQG VLPXODWLRQ DFWLYLWLHV ZLWKLQ WKH FKHPLFDO LQGXVWU\ 7KH PRGHOOLQJ SURFHVV UHFHLYHVFORVHDWWHQWLRQLQFOXGLQJLVVXHVRIYHULILFDWLRQ YDOLGDWLRQ DQG GRFXPHQWDWLRQ XVLQJ LQIRUPDWLRQ IURP 60 SNE 26(2) – 6/2016

ON

 H[SHULHQFHG PRGHOOHUV DQG VLPXODWLRQ VSHFLDOLVWV LQ RUJDQLVDWLRQV LQ VHYHUDO GLIIHUHQW FRXQWULHV 6XJJHVWHG GHYHORSPHQWV DV D UHVXOW RI WKDW LQYHVWLJDWLRQ UHODWH PDLQO\ WR LPSURYHPHQWV LQ PRGHOOLQJ WHFKQRORJ\ DQG WKH XVH RI DGYDQFHG PRGHOOLQJ WRROV )RVV HW DO DOVR SURYLGH VRPH XVHIXO LQVLJKWV DERXW KRZ PRGHOOLQJ DQG VLPXODWLRQ DFWLYLWLHV DUH FDUULHG RXW ZLWKLQ WKH LQGXVWU\ >@ $ VHFRQG LQYHVWLJDWLRQ IRFXVVHG LWV DWWHQWLRQ RQ WKH KHOLFRSWHU PDQXIDFWXULQJ LQGXVWU\ DQG LQYROYHG UHVSRQVHVWRDTXHVWLRQQDLUHZKLFKVRXJKWYLHZVRQWKH XVH RI V\VWHP LGHQWLILFDWLRQ DQG SDUDPHWHU HVWLPDWLRQ WHFKQLTXHVIRUWDVNVVXFKDVWKHYDOLGDWLRQRISK\VLFDOO\ EDVHG IOLJKW PHFKDQLFV PRGHOV >@  (LJKW FRPSDQLHV IURP 1RUWK $PHULFD DQG (XURSH UHVSRQGHG DQG WKH DQVZHUV WR WKH TXHVWLRQV SRVHG VKRZHG FRQVLGHUDEOH LQWHUHVWLQPRGHOYDOLGDWLRQDQGLQWKHXVHRIWKHVHVSH FLILFWHFKQLTXHVZKLOHHPSKDVLVLQJWKHQHHGIRUSK\VL FDOO\EDVHGLQWHUSUHWDWLRQVDWDOOWLPHV &RVWSUHGLFWLRQVLQWHUPVRIDPRGHOEDVHGDSSURDFK WRHQJLQHHULQJGHVLJQDOVRUHFHLYHVDWWHQWLRQLQDGRFX PHQWE\3DFH>@ZKLFKGLVFXVVHVODUJHSURMHFWVLQWKH DHURVSDFHDQGGHIHQFHVHFWRU$FDVHLVSXWIRUZDUGIRU PRUH VKDULQJ RI LQIRUPDWLRQ DERXW WKH FRVWV RI PRGHO OLQJ DQG VLPXODWLRQ DFWLYLWLHV LQ RUGHU WR DOORZ WKH GH YHORSPHQWRIPRUHUHOLDEOHFRVWLQJSURFHGXUHV$QRWKHU IDFWRULVWKDWPRGHOVFDQRQO\EHPDLQWDLQHGSURSHUO\LI WKH\DUHVHHQWREHLPSRUWDQWHLWKHULQHFRQRPLFWHUPV RULQWHUPVRIWKHLUIXWXUHSRWHQWLDO7KHGLIILFXOWLHVDQG FRVWV RI PDLQWDLQLQJ PRGHOV RYHU WKH FRPSOHWH OLIH F\FOH RI WKH V\VWHP RU SURGXFW KDYH WR EH FRQVLGHUHG H[SOLFLWO\DQGLWDSSHDUVWKDWVRPHHQJLQHHULQJRUJDQLVD WLRQVDQGFRPSDQLHVDUHHVWDEOLVKLQJWHFKQRORJ\JURXSV ZKLFK DUH WDVNHG ZLWK PDLQWDLQLQJ PRGHOV WKHLU VRIW ZDUH DQG GRFXPHQWDWLRQ DQG WUDQVODWLQJ WKHVH WR QHZ VRIWZDUHHQYLURQPHQWVDVQHFHVVDU\

2 The Testing of Simulation Models 7KHUHDUHWZRGLVWLQFWDQGVHSDUDWHDVSHFWVWRWKHWHVWLQJ RI VLPXODWLRQ PRGHOV 2QH RI WKHVH LV WHUPHG µYDOLGD WLRQ¶´ DQG WKLV LV FRQFHUQHG ZLWK WKH SURFHVV RI HVWDE OLVKLQJ KRZ ZHOO RU RWKHUZLVH  WKH PDWKHPDWLFDO DQG ORJLFDO GHVFULSWLRQ JLYHV EHKDYLRXU LQ WKH PRGHO WKDW DJUHHVZLWKWKHREVHUYHGEHKDYLRXURIWKHV\VWHPWKDWLW GHVFULEHV>@

ON

Murray-Smith

Transparency, Testing and Validation for Simulation Models

:KHQDUHDOV\VWHPLVDYDLODEOHIRUWHVWLQJDQGGLUHFW FRPSDULVRQV FDQ EH PDGH RI WKH WLPH KLVWRULHV RI NH\ YDULDEOHVWKHSURFHVVFDQEHFDUULHGRXWXVLQJDUDQJHRI GLIIHUHQW TXDQWLWDWLYH PHWKRGV DQG PHDVXUHV ([DPSOHV RI PHWKRGV WKDW KDYH EHHQ VXFFHVVIXOO\ DSSOLHG LQ WKH SDVWLQFOXGHJUDSKLFDOFRPSDULVRQVRIYDULRXVNLQGVWKH XVH RI PHWKRGV EDVHG RQ V\VWHP LGHQWLILFDWLRQ DQG SD UDPHWHU HVWLPDWLRQ SDUDPHWHU GLVWRUWLRQ PHWKRGV DQG HYROXWLRQDU\ FRPSXWLQJ PHWKRGV VXFK DV *HQHWLF 3UR JUDPPLQJVHHHJ>@>@IRURXWOLQHVRIWKHVHGLIIHUHQW DSSURDFKHV   µ)DFH¶ YDOLGDWLRQ PHWKRGV LQ ZKLFK WKH PRGHOEHKDYLRXULVDVVHVVHGE\VRPHRQHZKRKDVH[SHUW NQRZOHGJH RI WKH UHDO V\VWHP EXW ZDV QRW LQYROYHG LQ WKHGHYHORSPHQWRIWKHPRGHOSURYLGHDXVHIXODOWHUQD WLYHVHHHJ>@ $PRGHOFDQQHYHUEHµYDOLG¶IRUDOO DSSOLFDWLRQVDQGPXVWEHDVVHVVHGLQWHUPVRILWVVXLWD ELOLW\ IRU VRPH VSHFLILHG XVH 2IWHQ D FRPELQDWLRQ RI TXDQWLWDWLYHDQGIDFHYDOLGDWLRQDSSURDFKHVDUHXVHGDQG VXEMHFWLYHDQGREMHFWLYHHYLGHQFHKDVWREHFRPELQHGLQ VRPHZD\LQHVWDEOLVKLQJZKHWKHURUQRWWKHPRGHOLVILW IRUSXUSRVH$FFHSWDQFHRIDPRGHOIRULWVDSSOLFDWLRQ UHTXLUHV VWDWHPHQWV RI WKH UDQJH RI FRQGLWLRQV RYHU ZKLFK LW FDQ EH XVHG DQG WKH DVVRFLDWHG DFFXUDF\ RI PRGHOSUHGLFWLRQV 7KHVHFRQGDVSHFWRIWHVWLQJLVWHUPHGµYHULILFDWLRQ¶ DQGUHODWHVWRWKHSURFHVVRIHVWDEOLVKLQJWKDWDFRPSXWHU LPSOHPHQWDWLRQRIDPRGHOFRUUHVSRQGVWRWKHXQGHUO\ LQJ PDWKHPDWLFDO DQG ORJLFDO VWUXFWXUH IRU WKDW PRGHO >@7KLVLQYROYHVV\VWHPDWLFFKHFNVRIVLPXODWLRQFRGH WRHQVXUHWKDWQRHUURUVDUHSUHVHQWDQGDOVRDOJRULWKPLF FKHFNV WR HVWDEOLVK WKDW DSSURSULDWH FRPSXWDWLRQDO PHWKRGVKDYHEHHQDSSOLHG>@7KLVLVDVLPSOHUSURFHVV WKDQ YDOLGDWLRQ DQG LW KDV EHHQ VXJJHVWHG WKDW IRUPDO PHWKRGV DQG HVSHFLDOO\ µOLJKWZHLJKW¶ IRUPDO PHWKRGV  FRXOGEHXVHG>@ %RWKDVSHFWVRIWKHPRGHOWHVWLQJSURFHVVDUHYLWDOO\ LPSRUWDQW DQG PXVW EH FRQVLGHUHG LQ D V\VWHPDWLF ZD\ LQ DQ\ DSSOLFDWLRQ RI PRGHOOLQJ DQG VLPXODWLRQ WHFK QLTXHV ,Q HVVHQFH YDOLGDWLRQ LV FRQFHUQHG ZLWK WKH TXHVWLRQ µ,V WKLV WKH ULJKW PRGHO WR GHVFULEH WKH JLYHQ V\VWHP"¶ ZKLOH YHULILFDWLRQ GHDOV HVVHQWLDOO\ ZLWK WKH TXHVWLRQµ,VWKHVRIWZDUHLPSOHPHQWDWLRQRIWKHVLPXOD WLRQ FRUUHFW"¶ (DFK WLPH D PRGHO DQG WKH DVVRFLDWHG VLPXODWLRQ DUH FKDQJHG LQ DQ\ ZD\ WKH SURFHGXUHV RI YHULILFDWLRQ DQG YDOLGDWLRQ PXVW EH UHSHDWHG DQG WKH ZKROH SURFHGXUH RI PRGHO IRUPXODWLRQ PRGHO WHVWLQJ DQG PRGHO XSGDWLQJ PXVW EH UHJDUGHG DV LWHUDWLYH SUR FHVVZKLFKPD\KDYHWREHUHSHDWHGPDQ\WLPHVGXULQJ WKHOLIHRIDVLPXODWLRQPRGHO

$OWKRXJK DV PHQWLRQHG DERYH WKHUH DUH PDQ\ GLI IHUHQWDSSURDFKHVWRPRGHOYDOLGDWLRQWKHUHLVRQHRYHU ULGLQJDVSHFWRIWHVWLQJWKDWLVUHOHYDQWZKDWHYHUPHWKRG LV DGRSWHG 7KLV UHODWHV WR TXHVWLRQV RI H[SHULPHQWDO GHVLJQ ,Q PDQ\ HQJLQHHULQJ DSSOLFDWLRQV DQG DOVR LQ GHDOLQJZLWKPDQ\SK\VLRORJLFDODQGELRPHGLFDOPRGHO OLQJ DSSOLFDWLRQV  WHVWLQJ LQFOXGHV WKH XVH RI LQSXW VLJ QDOV WR SHUWXUE WKH V\VWHP 7KLV LV RI IXQGDPHQWDO LP SRUWDQFH EHFDXVH WKH FKRLFH RI WHVW LQSXW VLJQDO KDV D GLUHFW EHDULQJ RQ WKH DPRXQW RI LQIRUPDWLRQ DYDLODEOH DERXWWKHV\VWHPXQGHULQYHVWLJDWLRQ,WWKXVDOVRKDVDQ LPSRUWDQWLQIOXHQFHRQWKHHIIHFWLYHQHVVRIWHVWVFDUULHG RXWIRUWKHSXUSRVHVRIYDOLGDWLQJDPRGHO

3 Issues of Identifiability and Test Input Design '\QDPLFUHVSRQVHVPHDVXUHGIURPH[SHULPHQWVLQYROY LQJWKHDSSOLFDWLRQRIWHVWLQSXWVFDQSURYLGHDJUHDWGHDO RILQIRUPDWLRQWKDWLVQRWDYDLODEOHIURPWKHDQDO\VLVRI VWHDG\VWDWHFRQGLWLRQVRUIURPUHVSRQVHVUHVXOWLQJIURP VRPH LPSRVHG VHW RI LQLWLDO FRQGLWLRQV ([SHULPHQWV LQYROYLQJ WHVW LQSXW VLJQDOV DOVR SURYLGH D EDVLV IRU PDQ\ ZHOOHVWDEOLVKHG WHFKQLTXHV RI V\VWHP LGHQWLILFD WLRQ 7KLV LV DQ LQYHUVH PRGHOOLQJ SURFHGXUH LQ ZKLFK WKH VWUXFWXUH DQG SDUDPHWHUV RI D PRGHO DUH HVWLPDWHG IURP VHWV RI PHDVXUHG LQSXWRXWSXW GDWD IURP WKH UHDO V\VWHP6XFKWHFKQLTXHVPD\EHLPSRUWDQWLQWKHGHYHO RSPHQWRISK\VLFDOO\EDVHGVLPXODWLRQPRGHOVLIVLJQLI LFDQWXQFHUWDLQWLHVH[LVWLQWHUPVRIWKHPRGHOVWUXFWXUH RU SDUDPHWHU YDOXHV 1RWH WKDW WKLV µPRGHO LGHQWLILFD WLRQ¶ W\SH RI LQYHUVH SUREOHP KDV WR EH GLVWLQJXLVKHG IURPWKHµFDXVDWLRQ¶W\SHRILQYHUVHSUREOHPZKHUHIRU D JLYHQ PRGHO RQH VHHNV WR ILQG LQSXWV WKDW SURGXFH D VSHFLILHGUHVSRQVH>@7KLVFDXVDWLRQW\SHRIDSSURDFK DOVR KDV UHOHYDQFH IRU PRGHO YDOLGDWLRQ DV GLVFXVVHG HOVHZKHUHHJ>@  2QHYHU\LPSRUWDQWFRQFHSWWKDWLVFORVHO\OLQNHGWR SDUDPHWHUHVWLPDWLRQDQGPRGHOWHVWLQJLVPRGHOµLGHQWL ILDELOLW\¶7KHUHDUHWZRW\SHVRILGHQWLILDELOLW\SUREOHP DQGWKHVHFDQEHFODVVLILHGDV D µJOREDO¶µVWUXFWXUDO¶µGHWHUPLQLVWLF¶RUµDSULRUL¶ LGHQWLILDELOLW\ DQG E  µSDWKRORJLFDO¶ µQXPHULFDO¶ µSUDFWLFDO¶ RU µD SRVWHULRUL¶LGHQWLILDELOLW\ $V WKH QDPHV VXJJHVW WKH ILUVW W\SH RI LGHQWLILDELOLW\ SUREOHPDULVHVEHFDXVHRIWKHVWUXFWXUHRIWKHPRGHO SNE 26(2) – 6/2016 61

Murray-Smith

Transparency, Testing and Validation for Simulation Models

6XFK LVVXHV DULVH ZKHQ IRU H[DPSOH D PRGHO KDV WRR PDQ\ SDUDPHWHUV WR DOORZ DOO RI WKHP WR EH IRXQG LQGHSHQGHQWO\ IURP DQ\ LGHQWLILFDWLRQ H[SHULPHQW LQ YROYLQJDQ\FRPELQDWLRQRIWHVWLQSXWV,GHQWLILDELOLW\RI WKLVNLQGLVWKHPLQLPXPFRQGLWLRQQHFHVVDU\WRREWDLQ HVWLPDWHV RI DOO PRGHO SDUDPHWHUV 7KXV LI D PRGHO LV IRXQG WR EH VWUXFWXUDOO\ XQLGHQWLILDEOH FHUWDLQ SDUDPH WHUV FDQQRW EH HVWLPDWHG LQGHSHQGHQWO\ RI RWKHUV DQG WKLV KDV LPSRUWDQW LPSOLFDWLRQV IRU PRGHO YDOLGDWLRQ VLQFH VHWV RI LQGHSHQGHQW YDOXHV FDQQRW EH DVVLJQHG WR WKRVH SDUDPHWHUV 7KH VHFRQG W\SH RI LGHQWLILDELOLW\ SUREOHPLVHQFRXQWHUHGZKHQDVWUXFWXUDOO\LGHQWLILDEOH PRGHOLVEHLQJLQYHVWLJDWHGXVLQJGDWDVHWVWKDWDUHWRR VKRUW LQ UHODWLRQ WR WKH G\QDPLF FKDUDFWHULVWLFV RI WKH PRGHO RU ZKHUH PHDVXUHG UHVSRQVH GDWD VHWV DUH FRU UXSWHG E\ VLJQLILFDQW PHDVXUHPHQW QRLVH 0HDVXUHG UHVSRQVHGDWDVHWVWKXVKDYHWREHORQJHQRXJKWRFDS WXUHWKHHVVHQWLDOFKDUDFWHULVWLFVRIWKHV\VWHPLIWKH\DUH JRLQJWREHXVHIXOIRUPRGHOYDOLGDWLRQDQGDOVRKDYHWR EHUHODWLYHO\QRLVHIUHH ,QWKHFDVHRIOLQHDUPRGHOVPHWKRGVIRUWKHLQYHVWL JDWLRQ RI VWUXFWXUDO LGHQWLILDELOLW\ DUH ZHOO NQRZQ DQG PDQ\UHODWHWRDWUDQVIHUIXQFWLRQW\SHRIDSSURDFKSUR SRVHG E\ %HOOPDQ DQG cVWUऺP >@ WR PHWKRGV EDVHG RQ 7D\ORU VHULHV H[SDQVLRQV RU WR D 0DUNRY SDUDPHWHU PDWUL[ DSSURDFK >@ ,Q WKH FDVH RI QRQOLQHDU PRGHOV WZR DSSURDFKHV DUH FXUUHQWO\ DYDLODEOH 7KH ILUVW LQ YROYHVOLQHDULVDWLRQRIWKHPRGHODERXWVXLWDEOHRSHUDW LQJSRLQWVWRUHGXFHWKHQRQOLQHDUSUREOHPWRDVHULHVRI OLQHDUSUREOHPVIRUZKLFKRQHRIWKHOLQHDUDSSURDFKHV PD\ EH XVHG 7KH VHFRQG DSSURDFK LQYROYHV D 7D\ORU VHULHV H[SDQVLRQ RI REVHUYDWLRQV >@ DOWKRXJK LW LV UHFRJQLVHG WKDW WKLV LV GLIILFXOW WR DSSO\ LQ WKH FDVH RI FRPSOH[PRGHOV 7HVWLQSXWVWKDWDUHJRRGIRUWKHSXUSRVHVRIV\VWHP LGHQWLILFDWLRQ DUH WKRVH WKDW H[FLWH WKH GRPLQDQW PRGHV RIWKHV\VWHPDQGDOVRFRYHUDUDQJHRIDPSOLWXGHVWKDW DUH DSSURSULDWH IRU FKDUDFWHULVLQJ SRVVLEOH QRQOLQHDU EHKDYLRXU7HVWLQSXWVWKDWDUHMXGJHGWREHJRRGIRUWKH SXUSRVHVRIV\VWHPLGHQWLILFDWLRQDQGSDUDPHWHUHVWLPD WLRQDUHDOVRJHQHUDOO\JRRGWHVWLQSXWVIRUWKHSXUSRVHV RIPRGHOYDOLGDWLRQ7KLVIDFWLVZHOOHVWDEOLVKHGEXWLV RIWHQ RYHUORRNHG ZLWKLQ WKH PRGHOOLQJ DQG VLPXODWLRQ FRPPXQLW\ 6SHFLILF WHFKQLTXHV RI WHVW LQSXW GHVLJQ WKDW KDYH EHHQ IRXQG WR EH SDUWLFXODUO\ XVHIXO LQ WKH FRQWH[W RI PRGHO YDOLGDWLRQ LQFOXGH WKRVH EDVHG RQ WKH 'RSWLPDO FULWHULRQ ZKHUH HTXDO HPSKDVLV LV SODFHG RQ DOO WKH UHOHYDQW SDUDPHWHUV  DQG WKHWUXQFDWHG'RSWLPDO FULWH 62 SNE 26(2) – 6/2016

ON

ULRQ ZKHUH D VXEVHW RI WKHUHOHYDQW PRGHO SDUDPHWHUV LV HPSKDVLVHG  >@ 7KH XVH RI WKHVH WHVWLQSXW GHVLJQ WHFKQLTXHVZLWKLQVLPXODWLRQPRGHOYDOLGDWLRQWRJHWKHU ZLWK GLVFXVVLRQ RQ WKH XVH RI IUHTXHQF\GRPDLQ PHDVXUHV VXFK DV VSHFWUDO HQHUJ\ DQG FRKHUHQFH IXQF WLRQV LQFRPSDULQJWKHHIIHFWLYHQHVVRIGLIIHUHQWLQSXWV LVDWWUDFWLQJUHQHZHGLQWHUHVWDQGGHWDLOVRIWKHVHPHWK RGV PD\ EH IRXQG HOVHZKHUH DORQJ ZLWK UHOHYDQW FDVH VWXGLHVVHHHJ>@>@ 

4 Developments in Some Specific Application Areas 4.1 Engineeering developments ,W LV YHU\ FOHDU WKDW LQ VDIHW\ FULWLFDO DSSOLFDWLRQ DUHDV VXFKDVDLUFUDIWGHVLJQWKHDXWRPRWLYHLQGXVWU\UDLOZD\ V\VWHPV WKH QXFOHDU LQGXVWU\ DQG LQ PDQ\ DUHDV RI GH IHQFH PRGHOVDUHQRZ VXEMHFWHG WR D WKRURXJKSURFHVV RI GHYHORSPHQW WKDW LQYROYHV YHUVLRQ FRQWURO WHVWLQJ DQG GRFXPHQWDWLRQ &DUHIXO VHOHFWLRQ RI WHVW LQSXWV IRU WKHSXUSRVHVRIPRGHOYDOLGDWLRQLVDOVRW\SLFDORIZRUN LQ WKHVH DSSOLFDWLRQ DUHDV 7KLV LQ PDQ\ FDVHV LV GXH SDUWO\ WR UHTXLUHPHQWV LPSRVHG E\ H[WHUQDO UHJXODWRUV DQGVDIHW\DXWKRULWLHV ,Q UHFHQW \HDUV WKH LPSRUWDQFH RI XVLQJ PRGHOOLQJ WHFKQLTXHVLQODUJHSURMHFWVKDVEHFRPHDFHQWUDOSDUWRI WKHSKLORVRSK\RIWKH86'HIHQVH6FLHQFH%RDUG'6%  LQ WKH FRQWH[W RI WKH '6% 0RGHO'ULYHQ $UFKLWHFWXUH >@&ORVHO\DVVRFLDWHGZLWKWKHLGHDVRID0RGHO 'ULYHQ $UFKLWHFWXUH LV WKH FRQFHSW RI D µPRGHO DV D VSHFLILFDWLRQ¶7KLVZDVSURPRWHGYHU\DFWLYHO\E\7HUU\ (ULFVHQ DQG KLV FROOHDJXHV DW WKH 86 2IILFH RI 1DYDO 5HVHDUFKDVSDUWRIDGULYHIRUPDMRUHQKDQFHPHQWVLQ WKH XVH RI PRGHOOLQJ DQG VLPXODWLRQ WHFKQLTXHV LQ WKH FRQWH[WRIVKLSGHVLJQFRQVWUXFWLRQDQGRSHUDWLRQHJ >@  7KH DSSURDFKHV XVHG LQ RWKHU OHVV FRQWUROOHG DQG OHVV VDIHW\FULWLFDO DUHDV FDQ VRPHWLPHV EH HTXDOO\ ULJ RURXV+RZHYHUWKHUHLVVWLOOSOHQW\RIHYLGHQFHWKDWLQ PDQ\ FRQWH[WV VLPXODWLRQ PRGHOV DUH EHLQJ GHYHORSHG DQGXVHGLQZD\VWKDWODFNH[WHQVLYHWHVWLQJDQGLQYROYH GRFXPHQWDWLRQWKDWLVLQDGHTXDWHIRUDQ\RQHDWWHPSWLQJ WR UHXVH D PRGHO 7KH GRFXPHQWDWLRQ DOVR RIWHQ ODFNV WUDQVSDUHQF\DQGPD\EHLQDGHTXDWHIRUVXSSRUWLQJDQG PDLQWDLQLQJ WKH V\VWHP UHSUHVHQWHG E\ WKH PRGHO RYHU LWVFRPSOHWHOLIHF\FOH

ON

Murray-Smith

Transparency, Testing and Validation for Simulation Models

7KH MXVWLILFDWLRQ IRU WKLV UDWKHU KDSKD]DUG DSSURDFK WR PRGHOOLQJ DQG VLPXODWLRQ DFWLYLWLHV LV RIWHQ WKDW WKH PRUHULJRURXVDSSURDFKDGRSWHGLQVDIHW\FULWLFDODSSOL FDWLRQ DUHDV FDQQRW EH DIIRUGHG DQG WKDW WKH DSSURDFK DGRSWHGLVµZKDWKDVDOZD\VEHHQGRQH¶ 6XFK QHJDWLYH YLHZV RI WKH VLJQLILFDQFH RI PRGHO WHVWLQJDQGGRFXPHQWDWLRQIDLOWRWDNHSURSHUDFFRXQWRI WKH FRVWV WKDW FRXOG SRVVLEO\ EH VDYHG LQ WKH GHYHORS PHQWDQGPDQDJHPHQWRIQHZSURGXFWVWKURXJKDPRUH ULJRURXV DSSURDFK $OWKRXJK VXFK VDYLQJV DUH GLIILFXOW WRTXDQWLI\LWLVLQWHUHVWLQJWRQRWHWKDWLQVRPHDUHDVRI LQGXVWU\ DQG VFLHQFH QHZ DSSURDFKHV WR FRPSXWDWLRQDO PRGHOVDUHEHFRPLQJZHOOHVWDEOLVKHG ,Q WKH 8QNLWHG .LQJGRP RQH LPSRUWDQW H[DPSOH RI WKLV LV WKH GHFLVLRQ WKDW DOO PDMRU QHZ EXLOGLQJ DQG LQ IUDVWUXFWXUHSURMHFWVIXQGHGE\FHQWUDOJRYHUQPHQWIURP  PXVW DGKHUH WR %XLOGLQJ ,QIRUPDWLRQ 0RGHOOLQJ %,0  /HYHO  UHTXLUHPHQWV /HYHO  %,0 SURYLGHV D FRPPRQ VLQJOH DQG FRRUGLQDWHG VRXUFH RI VWUXFWXUHG LQIRUPDWLRQ IRU FRQVXOWDQWV FRQWUDFWRUV DQG DOO RWKHU SDUWLHV HQJDJHG LQ D ODUJH DQG FRPSOH[ SURMHFW >@ ,W LQYROYHVWKHXVHRIFRPSXWHUEDVHGPRGHOVDQGDVVRFL DWHG GDWDEDVHV IRU YLVXDOLVDWLRQ LQIRUPDWLRQ UHWULHYDO GRFXPHQWDWLRQ DQG OLIHWLPH PDLQWHQDQFH DQG VXSSRUW DQGSURPRWHVFRQVLVWHQF\DQGWUDQVSDUHQF\,WLVDOUHDG\ EHLQJXVHGIRUWKH&URVVUDLOSURMHFWLQ/RQGRQDQGLVDQ HVVHQWLDO IHDWXUH RI WKH SODQQHG +6 KLJKVSHHG UDLO FRQVWUXFWLRQ SURMHFWEHWZHHQ/RQGRQ %LUPLQJKDP DQG FLWLHV IXUWKHU QRUWK $OWKRXJK QRW FRQFHUQHG SULPDULO\ ZLWKG\QDPLFV\VWHPVLPXODWLRQ%,0FDQSURYLGHGDWD IRU G\QDPLF VLPXODWLRQV LQ WHUPV IRU H[DPSOH RI EXLOGLQJHQHUJ\VLPXODWLRQVDQGRSWLPLVDWLRQ )XWXUHGHYHORSPHQWVOLHZLWK/HYHO%,0DQGZLOO UHTXLUHIXOOFROODERUDWLRQEHWZHHQDOOSDUWLHVLQYROYHGLQ D SURMHFW WKURXJK WKH XVH RI D VLQJOH VKDUHG SURMHFW PRGHO KHOG LQ D FHQWUDO UHSRVLWRU\ (YHU\RQH LQYROYHG ZLOOWKHQEHDEOHWRDFFHVVDQGPRGLI\WKDWVDPHPRGHO DQGWKHEHQHILWLVWKDWLWHOLPLQDWHVULVNVDVVRFLDWHGZLWK FRQIOLFWLQJ LQIRUPDWLRQ  ,QGXVWU\ FRQFHUQV DERXW FRP PHUFLDO VHQVLWLYLW\ DQG FRS\ULJKW LVVXHV DUH EHLQJ UH VROYHG WKURXJK UREXVW GRFXPHQWDWLRQ DQG VRIWZDUH FRQWUROSURFHGXUHV 7KLV UHTXLUHPHQW IURP WKH 8. JRYHUQPHQW LV RQH DVSHFW RI D SODQ IRU UHGXFLQJ ZDVWH LQ WKH FRQVWUXFWLRQ LQGXVWU\E\,WLVEHOLHYHGWKDWGLVFUHSDQFLHVPLV WDNHVDQGLQHIILFLHQFLHVLQWKHLQIRUPDWLRQVXSSO\FKDLQ DUHPDMRUFRQWULEXWRUVWRWKLVZDVWHDQGWKDWFROODERUD WLYH ZRUNLQJ FDQ VLJQLILFDQWO\ UHGXFH LW )XUWKHU %,0 GHYHORSPHQWVDW/HYHOV''DQG' LQYROYHWKHXVH

RI%,0GDWDWRDQDO\VHWLPHIRUSXUSRVHVRIFRVWPDQ DJHPHQWDQGIRUIDFLOLWLHVPDQDJHPHQW ,WLVLQWHUHVWLQJWRQRWHWKDW%,0/HYHOKDVIHDWXUHV WKDWFDQEHOLQNHGWRWKHFRQFHSWVRIDµPRGHODVDVSHFL ILFDWLRQ¶ 7KH FHQWUDO DQG FORVHO\ PDQDJHG FRPSXWHU EDVHGPRGHORI%,0/HYHOVKRXOGSURYLGHDFRPPRQ UHIHUHQFHSRLQWIRUDOOHQJDJHGLQDSURMHFW 4.2 Some developments in the modelling of physiological and health-care systems -XVW DV LQ HQJLQHHULQJ DSSOLFDWLRQV WKH GHYHORSPHQW RI PRGHOV RI FRPSOH[ SK\VLRORJLFDO V\VWHPV QHHGV WR EH PDGH WUDQVSDUHQW WR XVHUV  $ V\VWHPDWLF DQG SURSHUO\ PDQDJHGDSSURDFKLVRIJUHDWLPSRUWDQFHIRUWKHVSHFL ILFDWLRQRIDPRGHOIRUGHYHORSPHQWRIPRGHOHTXDWLRQV LQFOXGLQJ WKH FKRLFH RI YDULDEOHV SDUDPHWHUV PRGHO ERXQGDULHVDVVXPSWLRQVDQGVLPSOLILFDWLRQV IRUYHULIL FDWLRQIRUYDOLGDWLRQDQGIRUGRFXPHQWDWLRQ2QHLQWHU HVWLQJ H[DPSOH IURP WKH ELRORJLFDO VFLHQFHV LV WKH +X PDQ 3K\VLRPH 3URMHFW RI WKH ,QWHUQDWLRQDO 8QLRQ RI 3K\VLRORJLFDO6FLHQFHV>@7KLVLVDQLQLWLDWLYHZKLFK LV FRQFHUQHG ZLWK HVWDEOLVKLQJ D FHQWUDO UHSRVLWRU\ RI GDWDEDVHV RI H[SHULPHQWDOO\GHULYHG LQIRUPDWLRQ DQG UHODWHG FRPSXWDWLRQDO PRGHOV 7KH WHUP µ3K\VLRPH¶ FRPHV IURP µSK\VLR¶ PHDQLQJ µOLIH¶ DQG µRPH¶ ZKLFK PHDQVµDVDZKROH¶7KHSURMHFWDLPVWREULQJWRJHWKHU ZLWKLQ RQH VHOIFRQVLVWHQW IUDPHZRUN DOO WKH H[SHUL PHQWDODQGPRGHOOLQJHOHPHQWVRIFXUUHQWSK\VLRORJLFDO UHVHDUFK &RQWULEXWRUV WR WKH +XPDQ 3K\VLRPH 3URMHFW DUHIURPDOOSDUWVRIWKHZRUOGDQGVLPXODWLRQSURJUDPV DFFHSWHG IRU SXEOLFDWLRQ WKURXJK WKLV SURMHFW PD\ EH GRZQORDGHGDQGXVHGE\RWKHUV,WLVLPSRUWDQWWRQRWH WKDWZLWKLQWKH3K\VLRPH3URMHFWWKHZRUGµPRGHO¶FDQ EHXVHGWRGHVFULEHDQ\WKLQJIURPDVFKHPDWLFGLDJUDP WKDWVXJJHVWVUHODWLRQVKLSVEHWZHHQHOHPHQWVRIDSK\VL RORJLFDO V\VWHP WR D IXOO\WHVWHG DQG GRFXPHQWHG FRP SXWHU VLPXODWLRQ PRGHO $Q\ PRGHO WKDW LV DFFHSWHG LV UHJDUGHG DV DµZRUNLQJK\SRWKHVLV¶ DQG KDV WR IRUP DQ LQWHUQDOO\ VHOIFRQVLVWHQW VWDWHPHQW RI WKH DYDLODEOH LQIRUPDWLRQ :KDW LV HVSHFLDOO\ LQWHUHVWLQJ LV WKDW WKURXJK WKH SURFHVVHV RI SXEOLFDWLRQ LQ WKH 3K\VLRPH 3URMHFWWKHPRGHOVDQGDVVRFLDWHGGDWDVHWVSURYLGHDQ LPSRUWDQWVWHSSLQJVWRQHWRQHZH[SHULPHQWVRQWKHUHDO V\VWHP DQG WKXV WR QHZ PRGHOV 8QGHUVWDQGLQJ RI D JLYHQV\VWHPVKRXOGEHHQKDQFHGLQDVWHSE\VWHSIDVK LRQ WKURXJK WKLV W\SH RI FROODERUDWLYH SURFHGXUH ZKLFK LQYROYHV DQ LWHUDWLYH SURFHVV RI PRGHOOLQJ VLPXODWLRQ WHVWLQJDQGFRPSDULVRQ SNE 26(2) – 6/2016 63

Murray-Smith

Transparency, Testing and Validation for Simulation Models

2QH LPSRUWDQW IHDWXUH RI WKH 3K\VLRPH 3URMHFW LV WKDW DOO PRGHOV KDYH WR SDVV WKURXJK D µFXUDWLRQ SLSH OLQH¶SULRUWREHLQJDFFHSWHGDQGPDGHDYDLODEOHSXEOLF O\ 7KLV SURFHGXUH LV VLPLODU WR DQ LQGHSHQGHQW UHYLHZ SURFHVVIRUDFRQYHQWLRQDOSXEOLFDWLRQDQGLVDIRUPRI µDFFUHGLWDWLRQ¶ :LWKLQ WKH FXUDWLRQ SURFHGXUH WKH VXE PLWWHG VLPXODWLRQ PRGHO LV WHVWHG WR HQVXUH WKDW LW LV VHPDQWLFDOO\ VRXQG DQG UHVXOWV REWDLQHG IURP LW DUH FRQVLVWHQW ZLWK WKH DYDLODEOH SXEOLVKHG LQIRUPDWLRQ DERXWWKHFRUUHVSRQGLQJUHDOV\VWHP ,Q WKH KHDOWK FDUH V\VWHPV PRGHOOLQJ ILHOG RQH VLJ QLILFDQW GHYHORSPHQW LQ  ZDV WKH SXEOLFDWLRQ RI VHYHQ SDSHUV SUHSDUHG E\ WKH *RRG 5HVHDUFK 3UDFWLFHV LQ0RGHOLQJ7DVN)RUFH7KLVJURXSZDVHVWDEOLVKHGLQ E\WKH,QWHUQDWLRQDO6RFLHW\IRU3KDUPDFRHFRQRP LFVDQG2XWFRPHV5HVHDUFK,6325 DQGWKH6RFLHW\IRU 0HGLFDO 'HFLVLRQ 0DNLQJ 60'0  7KH 7DVN )RUFH SURGXFHG D VHW RI UHFRPPHQGDWLRQV DQG VHYHQ DUWLFOHV ZHUHMRLQWO\SXEOLVKHGE\ERWKVRFLHWLHVLQWKHLUUHVSHF WLYH MRXUQDOV $ VXPPDU\ RI WKH DUWLFOHV ZDV SUHVHQWHG DVDSOHQDU\VHVVLRQDWWKH,6325WK$QQXDO,QWHUQD WLRQDO 0HHWLQJ LQ 0D\  DQG DJDLQ DW WKH 60'0 UG$QQXDO0HHWLQJODWHUWKDW\HDUDQG>@SURYLGHVDQ RYHUYLHZ2IWKHVL[RWKHUSDSHUVSXEOLVKHGE\WKH7DVN )RUFH PHPEHUV WKH WZR PRVW GLUHFWO\ UHOHYDQW WR WKH WKHPH RI WKLV SDSHU IRFXV RQ SDUDPHWHU HVWLPDWLRQ DQG XQFHUWDLQWLHV >@ DQG RQ µEHVW SUDFWLFHV¶ LQ WHUPV RI PRGHOWUDQVSDUHQF\DQGYDOLGDWLRQLVVXHV>@0DQ\RI WKHEHVWSUDFWLFHUHFRPPHQGDWLRQVDUHGLUHFWO\UHOHYDQW WRPRGHOOLQJDQGVLPXODWLRQLQJHQHUDODQGDUHQRWOLP LWHG WR DSSOLFDWLRQV LQYROYLQJ KHDOWK FDUH 0RVW FRUUH VSRQG WR JHQHUDO LVVXHV RI PRGHO PDQDJHPHQW WHVWLQJ DQGGRFXPHQWDWLRQGLVFXVVHGLQWKLVSDSHU

5 Discussion and Conclusions 0RGHO PDQDJHPHQW SURFHGXUHV VKRXOG HQVXUH WKDW WKH PRGHO GHYHORSPHQW SURFHVV LV UHOLDEOH DQG UREXVW *RRG GRFXPHQWDWLRQ VWDQGDUGV VKRXOG OHDG WR KLJK OHYHOVRIWUDQVSDUHQF\LQWKHPRGHOVFUHDWHGZLWKLQVXFK DPDQDJHPHQWV\VWHP +HOSIXOLQVLJKWLQWHUPVRIJRRGSUDFWLFHFDQEHRE WDLQHG IURP ZRUN RQ VDIHW\FULWLFDO DQG GHIHQFHUHODWHG SURMHFWV EXW LVVXHV RI QDWLRQDO RU FRPPHUFLDO VHFUHF\ OLPLWWUDQVIHURINQRZOHGJHIURPWKHVHILHOGVLQWRRWKHU DUHDVRIDSSOLFDWLRQ2QWKHRWKHUKDQGVRPHYHU\SRVL WLYH GHYHORSPHQWV KDYH DOVR WDNHQ SODFH UHFHQWO\ LQ D QXPEHURIDUHDVRIELRPHGLFDODQGKHDOWKFDUHPRGHOOLQJ 64 SNE 26(2) – 6/2016

ON

DQG LQWHUHVWLQJ UHFRPPHQGDWLRQV KDYH IROORZHG LQ WHUPVRIWUDQVSDUHQF\DQGYDOLGDWLRQ 5HFHQW GHYHORSPHQWV LQ DUHDV VXFK DV VKLS GHVLJQ DQG FLYLO HQJLQHHULQJ DOVR VXJJHVW WKDW ZLWK VRPH H[ WHUQDO SUHVVXUHV LQLWLDOO\ JRRG PRGHOOLQJ SUDFWLFH FDQ SURGXFH VLJQLILFDQW EHQHILWV DQG SRWHQWLDO FRVW VDYLQJV LQ ODUJH SURMHFWV &XUUHQW WUHQGV LQ WKRVH DUHDV QHHG WR EH FRQVLGHUHG WR HVWDEOLVK ZKHWKHU RU QRW WKH LGHDV EH KLQG VXFK GHYHORSPHQWV FRXOG EH WUDQVIHUUHG WR RWKHU ILHOGV (GXFDWLRQ DQG WUDLQLQJ XQGRXEWHGO\ KDV DQ LP SRUWDQWLQIOXHQFH0DQ\VWXGHQWVHQFRXQWHUWKHLGHDVRI PDWKHPDWLFDO PRGHOOLQJ DQG FRPSXWHU VLPXODWLRQ EXW UHODWLYHO\IHZJUDGXDWHVDSSHDUWREHIXOO\DZDUHRIWKH LPSRUWDQFH RI WHVWLQJ WKHLU PRGHOV 7KH HPSKDVLV LQ PDQ\FRXUVHVLVRQWKHGHYHORSPHQWRIPRGHOVIURPWKH XQGHUO\LQJODZVRISK\VLFVFKHPLVWU\DQGRWKHUDUHDVRI VFLHQFH DQG WKHQ RQ WKH QXPHULFDO PHWKRGV IRU ILQGLQJ VROXWLRQV$QH[DPLQDWLRQRIV\OODEXVLQIRUPDWLRQIURP D ZLGH UDQJH RI XQLYHUVLWLHV VXJJHVWV WKDW LQ PDQ\ FRXUVHV LQYROYLQJ VLPXODWLRQ WKH WRSLFV RI PRGHO WHVW LQJYDOLGDWLRQDQGGRFXPHQWDWLRQGRQRWUHFHLYHVXIIL FLHQWDWWHQWLRQ $QRWKHULQWHUHVWLQJSRLQWWKDWUHODWHVWRDQXPEHURI VFLHQWLILFGLVFLSOLQHVHJWKHELRORJLFDOVFLHQFHV LVWKDW WKHFRQYHQWLRQLQSXEOLFDWLRQRIH[SHULPHQWDORUFRPSX WDWLRQDOUHVXOWVLQDMRXUQDORUFRQIHUHQFHSURFHHGLQJVLV WKDW D µPHWKRGV¶ VHFWLRQ LV WUDGLWLRQDOO\ LQFOXGHG 7KH LQFOXVLRQRIWKH PHWKRGVVHFWLRQLVLQWHQGHGWRSURYLGH HQRXJKLQIRUPDWLRQWRDOORZUHDGHUVWRIXOO\XQGHUVWDQG WKHWHFKQLTXHVXVHGWRREWDLQWKHSXEOLVKHGUHVXOWVDQG LGHDOO\ WR DOORZ UHDGHUV WR IROORZ WKH SURFHVVHV XVHG DQGWKXVUHSHDWWKHUHVXOWVLQGHSHQGHQWO\DQGEHDEOHWR UHSURGXFHWKHSXEOLVKHGILQGLQJV8QIRUWXQDWHO\WKLVLV QRW DFRQYHQWLRQ WKDW LVZLGHO\ IROORZHGDW SUHVHQW LQ PRVW SXEOLVKHG ZRUN RQ PRGHOOLQJ DQG VLPXODWLRQ DS SOLFDWLRQVEXWKDVFOHDUEHQHILWVLQWHUPVRIHVWDEOLVKLQJ WUDQVSDUHQF\DQGUHSURGXFLELOLW\ &OHDUO\ WKHUH DUH VHYHUDO DUHDV LQ ZKLFK WKHUH LV VFRSH IRU LPSURYHPHQW LQ WKH SURFHVVHV RI GHYHORSLQJ LPSOHPHQWLQJ GRFXPHQWLQJ DQG DSSO\LQJ VLPXODWLRQ PRGHOVZLWKLQPDQ\RUJDQLVDWLRQV,WLVFOHDUWKDWUHFHQW GHYHORSPHQWV LQ VRPH VSHFLILF ILHOGV KDYHOHG WR LQWHU HVWLQJ DQG KHOSIXO LGHDV DQG WR UHFRPPHQGDWLRQV LQ WHUPVRIEHVWSUDFWLFHV0DQ\RIWKHVHLGHDVDUHWUDQVIHU UDEOHWRRWKHUDUHDVDQGDUHZRUWK\RIFDUHIXOFRQVLGHUD WLRQ E\ DOO HQJDJHG LQ PRGHOOLQJ DQG VLPXODWLRQ DQG HVSHFLDOO\E\WKRVHLQYROYHGLQHGXFDWLRQ

ON

Murray-Smith

Transparency, Testing and Validation for Simulation Models

References >@ 6FKOHVLQJHU67HUPLQRORJ\IRUPRGHOFUHGLELOLW\Simulation.   GRL >@ 6DUJHQW5*9DOLGDWLRQRIVLPXODWLRQPRGHOV,Q+LJK ODQG+-HGLWRU3URFHHGLQJVRIWKH:LQWHU6LPXOD WLRQ&RQIHUHQFH3LVFDWDZD\,(((3UHVV9RO  >@ gUHQ7,&RQFHSWVDQGFULWHULDWRDVVHVVDFFHSWDELOLW\RI VLPXODWLRQVWXGLHVDIUDPHRIUHIHUHQFH,Q$GDP1HGL WRUSimulation Modelling and Statistical Computing &RPPXQLFDWLRQVRIWKH$&0  >@ %DOFL23ULQFLSOHVRIVLPXODWLRQPRGHOYDOLGDWLRQ Trans. Soc. Comput. Simul. Int.   >@ %UDGH'(QKDQFHGPRGHOOLQJDQGVLPXODWLRQDFFUHGLWD WLRQE\VWUXFWXULQJYHULILFDWLRQDQGYDOLGDWLRQUHVXOWV,Q -RLQHV-$%DUWRQ5.DQJ.HWDOHGLWRUV3URFHHGLQJV RIWKH:LQWHU6LPXODWLRQ&RQIHUHQFH/D-ROOD&$ 86$6RF0RGDQG&RPSXW6LPXO,QW  >@ 0XUUD\6PLWK'-Modelling and Simulation of Integrated Systems: Issues of Methodology, Quality, Testing and Application. &DPEULGJH8.:RRGKHDG3XEOLVKLQJ SS >@ %UXHQHVH$3-7RS--%URHQLN-)HWDO/LEUDULHVRIUH XVDEOHPRGHOVWKHRU\DQGDSSOLFDWLRQSimulation   GRL >@ 0XUUD\6PLWK'-Testing and Validation of Computer Simulation Models, &KDP6ZLW]HUODQG6SULQJHU SS >@ &ORXG'-5DLQH\/%HGLWRUVApplied Modeling and Simulation: An Integrated Approach to Develomment and Operation1HZ@ 3DFH'.0RGHOOLQJDQGVLPXODWLRQYHULILFDWLRQDQGYDO LGDWLRQFKDOOHQJHVJohn Hopkins APL Tech Digest   >@ 7KH0LWUH&RUSRUDWLRQ9HULILFDWLRQDQGYDOLGDWLRQRI VLPXODWLRQPRGHOVMitre Systems Engineering Guide >2QOLQH@$YDLODEOH ZZZPLWUHRUJSXEOLFDWLRQVWHFKQLFDOSDSHUVWKHPLWUH V\VWHPVHQJLQHHULQJJXLGH>$FFHVVHG-XQH@ >@ )RVV%$/RKPDQQ%0DUTXKDUGW:$ILHOGVWXG\RI LQGXVWULDOPRGHOOLQJSURFHVVModel. Identif. Control.  GRLPLF >@ 3DGILHOG*'7KHUROHRILQGXVWU\,QRotorcraft System Indentification. $*$5'$GYLVRU\5HSRUW $*$5'$5 6HFWLRQ1HXLOO\VXUB6HLQH )UDQFH$*$5' >@ .XKQ'5&KDQGUDPRXOL5%XWOHU5:&RVWHIIHFWLYH XVHRIIRUPDOPHWKRGVLQYHULILFDWLRQDQGYDOLGDWLRQ,Q YLWHG3DSHUProceedings Foundations ’02 Workshop. 2FW/DXUHO0DU\ODQG86$86'HSWRI 'HIHQVH

>@ *URHWVFK&:Inverse Problems:DVKLQJWRQ'&86$ 7KH0DWKHPDWLFDO$VVRFLDWLRQRI$PHULFD SS >@ %HOOPDQ5cVWUऺP.-2QVWUXFWXUDOLGHQWLILDELOLW\ Math Biosci.GRL  ; >@ +YDOD1=HF06WUPþQLN61RQOLQHDUPRGHOSDUDPH WHUHVWLPDWLRQ±HVWLPDWLQJDIHDVLEOHSDUDPHWHUVHWZLWK UHVSHFWWRPRGHOXVHMathematical and Computer Modelling of Dynamical Systems.   GRL >@ +XQWHU:*+LOO:-+HQVRQ7/'HVLJQLQJH[SHULPHQWV IRUSUHFLVHHVWLPDWLRQRIDOORUVRPHRIWKHFRQVWDQWVLQD PHFKDQLVWLFPRGHOCanadian J Chem Eng.  GRLFMFH >@ 'HIHQVH6FLHQFH%RDUG'6% 'HIHQVH6FLHQFH%RDUG 0RGHO'ULYHQ$UFKLWHFWXUH0'$ DSB Report:DVK LQJWRQ'&86$'HIHQVH6FLHQFH%RDUG >@ +HPH])07KHP\WKRIVFLHQFHEDVHGSUHGLFWLYHPRG HOOLQJ,QProceedings Foundations ’04 Workshop.  2FW7HPSH$UL]RQD86$5HSRUW/$85 /RV$ODPRV1DWLRQDO/DERUDWRU\86$ >@ (ULFVHQ73K\VLFVEDVHGGHVLJQWKHIXWXUHRIPRGHOOLQJ DQGVLPXODWLRQActa Polytechnica  >@ (ULFVHQ7*XLGDQFHIRUWKHGHYHORSPHQWRIVKLSVPDUW V\VWHPGHVLJQ6' 215%$$$GYDQFHG,Q WHJUDWHG3RZHU6\VWHPV-XO\%RVWRQ86$ >@ )DPPH-%*DOODFKHU&5DLWFK73HUIRUPDQFHEDVHG GHVLJQIRUIOHHWDIIRUGDELOLW\Nav. Eng. J.   GRLM[ >@ %XLOGLQJ,QIRUPDWLRQ0RGHOOLQJ%,0 7DVN*URXS >2QOLQH@$YDLODEOHZZZELPWDVNURXSRUJ>$FFHVVHG -XQH@ >@ $QRQ\PRXVAbout the IUPS Physiome Project >2QOLQH@$YDLODEOH ZZZSK\VLRPHSURMHFWRUJDERXWWKHYLUWXDO SK\VLRORJLFDOKXPDQ>$FFHVVHG-XQH@ >@ &DUR--%ULJJV$+6LHEHUW8.XQW].00RGHOLQJ JRRGUHVHDUFKSUDFWLFHVRYHUYLHZDUHSRUWRIWKH ,632560'00RGHOLQJ*RRG5HVHDUFK3UDFWLFHV7DVN )RUFH Medical Decision Making.   GRL; >@ %ULJJV$+:HLQVWHLQ0&)HQZLFN($/.DUQRQ- 6FKXOSKHU0-3DOWLHO$'0RGHOSDUDPHWHUHVWLPDWLRQ DQGXQFHUWDLQW\DUHSRUWRIWKH,632560'00RGHOLQJ *RRG5HVHDUFK3UDFWLFHV7DVN)RUFH Medical Decision Making.   GRL; >@ (GG\'0+ROOLQJVZRUWK:&DUR--7VHYDW-0F'RQ DOG.0:RQJ-%0RGHOWUDQVSDUHQF\DQGYDOLGDWLRQD UHSRUWRIWKH,632560'00RGHOLQJ*RRG5HVHDUFK 3UDFWLFHV7DVN)RUFH Medical Decision Making.   GRL;

SNE 26(2) – 6/2016 65