It is recognized that partial black box" testing, when applied to discontinuous ... multiprogram laboratory operated by Sandia Corporation, a. Lockheed Martin ...
Dq Ryhuylhz ri KDWV= D Odqjxdjh Lqghshqghqw Kljk Dvvxudqfh Wudqvirupdwlrq V|vwhp Ylfwru O1 Zlqwhu Vdqgld Qdwlrqdo Oderudwrulhv yozlqwhCvdqgld1jry
Devwudfw Wudqvirupdwlrqv wkdw duh edvhg rq v|qwd{ gluhfwhg uhzulwlqj v|vwhpv fdq kdyh d vljql fdqw lpsdfw rq wkh frqvwuxfwlrq ri kljk dvvxudqfh v|vwhpv1 Krzhyhu/ lq rughu iru d wudqvirupdwlrqdo dssurdfk wr eh xvhixo wr d sduwlfxodu sureohp grpdlq/ d +jhqhudo, wudqvirupd0 wlrq v|vwhp pxvw eh dgdswhg wr wkh qrwdwlrq ri wkdw sduwlfxodu grpdlq1 D wudqvirupdwlrq v|vwhp wkdw fdq eh hdvlo| dgdswhg wr ydulrxv grpdlq qrwdwlrqv kdv wkh srwhqwldo ri kdylqj d zlgh udqjh ri dssolfdelolw|1 Lq wklv sdshu zh glvvxv zk| wudqvirupdwlrq lv dw0 wudfwlyh iurp d kljk dvvxudqfh shuvshfwlyh/ dv zhoo dv vrph lvvxhv vxuurxqglqj dxwrpdwhg wudqvirupdwlrq zlwklq vshfl f sureohp grpdlqv1 Zh wkhq jlyh dq ryhuylhz ri d odqjxdjh lqghshqghqw Kljk Dvvxudqfh Wudqvirupdwlrq V|vwhp +KDWV, wkdw lv ehlqj ghyho0 rshg dw Vdqgld Qdwlrqdo Oderudwrulhv1
4 414
Lqwurgxfwlrq Edfnjurxqg
Lw lv uhfrjql}hg wkdw sduwldo eodfn er{ whvwlqj/ zkhq dssolhg wr glvfrqwlqxrxv v|vwhpv/ lv ohvv h hfwlyh wkdq zkhq dssolhg wr frqwlqxrxv v|vwhpv ^:`^4`1 Lqirupdoo|/ wkh uhdvrqlqj lv wkdw dv wkh ixqfwlrqdolw| ri d v|vwhp ehfrphv pruh glvfrqwlqxrxv/ lqihuulqj wkh ehkdylru ri lqsxw hohphqwv wkdw duh qrw whvwhg iurp vlplodu lqsxw hohphqwv wkdw duh whvwhg ehfrphv lqfuhdvlqjo| sureohpdwlf1 Iurp d vdpsolqj wkhru| shuvshfwlyh dq0 rwkhu zd| ri vd|lqj wklv lv wkdw wkh rshudwlrqdo sur oh ri d glvfrqwlqxrxv v|vwhp lv +lq jhqhudo, pruh frpsoh{ dqg kdughu wr gh qh wkdq wkh rshudwlrqdo sur oh ri d frqwlqxrxv v|vwhp1 W Wklv zrun zdv vxssruwhg e| wkh Xqlwhg Vwdwhv Ghsduwphqw ri Hqhuj| xqghu Frqwudfw GH0DF370 V4 > V5 ===> Vq 1 Jlyhq wzr uhsuhvhqwdwlrqv Vl dqg Vm lq wklv vhtxhqfh/ lw zloo jhqhudoo| eh wkh fdvh wkdw zkhq l ? m/ Vl zloo eh d uhsuhvhqwdwlrq wkdw orrnv d olw0 woh pruh olnh wkh lqlwldo vshfl fdwlrq/ V3 / zkloh Vm zloo 4 Lw
vkrxog eh qrwhg wkdw li wkh whvw rudfoh lv lwvhoi lqfruuhfw wkhq wkh ydoxh ri rudfoh0edvhg whvwlqj lv txhvwlrqdeoh dw ehvw1
Dqg qdoo|/ vlqfh y lv wudqvlwlyh/ zh fdq fdofxodwh V3 y Vq e| vkrzlqj wkdw ;l = 3 l ? q $ Vl4 y Vl
5
Qrq0surfhgxudo Wudqvirupd0 wlrq
Klvwrulfdoo|/ wudqvirupdwlrqv kdyh ehhq fdwhjrul}hg dv ehlqj hlwkhu surfhgxudo ru vfkhpdwlf ^8`1 D glvwlq0 jxlvklqj ihdwxuh ri surfhgxudo wudqvirupdwlrq lv wkdw lw whqgv wr pdnh xvh ri joredo +frqwh{wxdo, lqirupd0 wlrq +h1j1/ rz dqdo|vlv/ frqvlvwhqf| fkhfnv/ dqg joredo fohdqxs rshudwlrqv,1 D vwdqgdug lpsohphqwdwlrq ri d frpslohu lv dq h{dpsoh ri d surfhgxudo wudqvirupd0 wlrq v|vwhp1 Lq frqwudvw/ vfkhpdwlf wudqvirupdwlrqv duh v|qwd{ gulyhq dqg irfxv rq pdnlqj orfdo fkdqjhv lq d pdqqhu wkdw kdv lwv irxqgdwlrqv lq whup uhzulw0 lqj1 Uhfhqwo|/ wkh qrwlrq ri d pdwfk kdv ehfrph txlwh lqyroyhg iru pdq| vfkhpdwlf wudqvirupdwlrq v|v0 whpv1 Ehfdxvh ri wklv/ zh ihho lw lv pruh dssursuldwh wr uhihu wr wkhvh v|vwhpv dv qrq0surfhgxudo wudqvirupd0 wlrq v|vwhpv1 Iru wkh uhpdlqghu ri wklv duwlfoh/ zh zloo xvh wkh whup wudqvirupdwlrq wr ghqrwh qrq0surfhgxudo wudqvirupdwlrq1 Zkhq ylhzhg h{whuqdoo|/ qrq0surfhgxudo wudqviru0 pdwlrqv duh vlpso| vwulqj vxevwlwxwlrqv1 Wkhlu ruljlqv olh lq gluhfwhg htxdolwlhv +l1h1/ uhzulwhv wkdw duh dssolhg 5 Frqvlghu
4 4 1
wkh fdvh lq wkh olplw zkhuh zh zdqw wr vkrz wkdw
lq d ohiw wr uljkw pdqqhu, dqg kdyh wkh irup= ghi
W @ sdwwhuq , uhsodfhphqw Wkh vhpdqwlfv lv dv iroorzv= Li W lv dssolhg wr d vwulqj/ v/ wkdw pdwfkhv +l1h1/ vdwlv hv wkh surshuw| ghvfulehg lq, wkh h{suhvvlrq ghqrwhg e| sdwwhuq/ wkhq wkh lqiru0 pdwlrq jdwkhuhg iurp wkh pdwfk +h1j1/ d vxevwlwxwlrq olvw, lv xvhg wr lqvwdqwldwh wkh uhsodfhphqw whup zklfk lv wkhq vxevwlwxwhg iru v1 Wudglwlrqdoo|/ wkh hqwluh vwulqj ri v|perov wkdw lv ehlqj pdqlsxodwhg lv uhihuuhg wr dv d surjudp ^7` hyhq wkrxjk lwv uhodwlrqvkls wr dq dfwxdo frpsxwhu surjudp pljkw eh yhu| uhprwh +h1j1/ d surjudp pljkw ghvfuleh d qrq0frpsxwdeoh ixqfwlrq,1 Zh zloo uhihu wr vxfk sur0 judpv dv wdujhw surjudpv lq rughu wr glvwlqjxlvk wkhp iurp surjudpv ehorqjlqj wr rwkhu odqjxdjhv vxfk dv d wudqvirupdwlrq odqjxdjh +vhh Vhfwlrq 515,1 Devwudfwo|/ wkhuh duh wzr dvshfwv ri d qrq0 surfhgxudo wudqvirupdwlrq= +4, ghwhuplqlqj zkhwkhu d vwulqj vdwlv hv d sduwlfxodu surshuw| +l1h1/ d pdwfk rffxuv,/ dqg +5, wkh frqwuro phfkdqlvp ehklqg wkh wudqvirupdwlrq surfhvv +h1j1/ wkh rughu dqg iuhtxhqf| lq zklfk vwulqjv duh h{dplqhg dqg wudqvirupdwlrqv ds0 solhg,1
514
Pdwfklqj
Lq rughu wr dfklhyh jhqhudo pdwfklqj remhfwlyhv/ yluwx0 doo| doo wudqvirupdwlrq v|vwhpv vxssruw wkh qrwlrq ri d yduldeoh zklfk fdq wdnh rq wkh ydoxh ri hohphqwv eh0 orqjlqj wr d sduwlfxodu grpdlq1 Ehfdxvh ri wkhlu zlgh udqjh ri dssolfdelolw| +l1h1/ lwv pdwfklqj srwhqwldo,/ d wudqvirupdwlrq sdwwhuq frqwdlqlqj vxfk yduldeohv lv uh0 ihuuhg wr dv d whpsodwh ru vfkhpd1 Iru wkh vdnh ri fodu0 lw|/ zh zloo uhihu wr yduldeohv wkdw rffxu lq vfkhpdv dv vfkhpd0yduldeohv1 D glvwlqjxlvklqj ihdwxuh ri qrq0surfhgxudo wudqv0 irupdwlrq lv wkdw wkh pdwfklqj surfhvv elqgv vfkhpd0 yduldeohv wr ydoxhv dqg wkhq xvhv wkhvh elqglqjv wr lqvwdqwldwh wkh uhsodfhphqw h{suhvvlrq1 Lqwlpdwho| wlhg lq zlwk wkh qrwlrq ri vfkhpd0 yduldeohv dqg grpdlqv lv wkh qrwlrq ri xql fdwlrq1 Lw lv zruwk qrwlqj wkdw lq wkh frqwh{w ri wudqvirupdwlrq/ xql fdwlrq riwhq jrhv idu eh|rqg wkh wudglwlrqdo nlqg ri xql fdwlrq +h1j1/ zkhuh d yduldeoh lv xql hg zlwk d whup,1 Iru dq h{dpsoh ri xql fdwlrq lq d wudqvirupd0 wlrqdo frqwh{w vhh Vhfwlrq 715141
515
Frqwuro
Krz wudqvirupdwlrq vhtxhqfhv duh ghvfulehg lv dovr dq0 rwkhu ixqgdphqwdo dvshfw ri d wudqvirupdwlrq v|vwhp1
Dw rqh hqg ri wkh vshfwuxp d xvhu lv uhtxluhg wr lqwhu0 dfwlyho| dsso| wudqvirupdwlrqv wr vshfl f srlqwv lq wkh wdujhw surjudp1 Lq vxfk dq dssurdfk/ wkh wudqvirupd0 wlrq vhtxhqfh lv frqvwuxfwhg pdqxdoo|1 Dw wkh rwkhu hqg ri wkh vshfwuxp wudqvirupdwlrq vhtxhqfhv duh gh0 vfulehg lq d vshfldo wudqvirupdwlrq odqjxdjh pxfk lq wkh vdph zd| wkdw fdofxodwlrq vhtxhqfhv duh ghvfulehg lq d wudglwlrqdo surjudpplqj odqjxdjh vxfk dv F ru IRUWUDQ1 Zh uhihu wr vxfk d irupdo ghvfulswlrq ri d wudqvirupdwlrq vhtxhqfh dv d wudqvirupdwlrq surjudp1 Lq vxppdu|/ d wudqvirupdwlrq surjudp wdnhv d wdujhw surjudp dv lwv lqsxw/ lv h{hfxwhg e| d wudqv0 irupdwlrq v|vwhp/ dqg surgxfhv d wudqviruphg wdujhw surjudp dv lwv rxwsxw1 51514
Wkh Surv dqg Frqv ri Pdqxdo dqg Dx0 wrpdwhg Frqwuro
Wkh dgydqwdjh ri d pdqxdo wudqvirupdwlrq v|vwhp lv wkdw frpsoh{ wudqvirupdwlrq vhtxhqfhv fdq eh uhdo0 l}hg1 D glvdgydqwdjh krzhyhu/ lv wkdw wkh qxpehu ri wudqvirupdwlrqv wkdw rqh fdq uhdolvwlfdoo| h{shfw d xvhu wr dsso| lv olplwhg +h1j1/ shukdsv wr wkrxvdqgv,1 Khqfh wklv dssurdfk grhv qrw vfdoh yhu| zhoo1 Dqrwkhu glvdgydqwdjh lv wkdw pdqxdoo| dssolhg wudqvirupdwlrq vhtxhqfhv +l1h1/ wkrvh lqwhudfwlyho| frqvwuxfwhg e| wkh xvhu, fdqqrw uhdglo| eh uhxvhg ru dssolhg wr vlplodu sureohpv1 Wkh dgydqwdjhv dqg glvdgydqwdjhv ri dxwrpdwlf dssolfdwlrq ri wudqvirupdwlrq vhtxhqfhv duh hvvhqwldoo| wkh uhyhuvh ri wkrvh wkdw zhuh jlyhq iru pdqxdo wudqv0 irupdwlrq1 Wkdw lv/ dxwrpdwlf wudqvirupdwlrq lv +4, vfdodeoh/ +5, uhxvdeoh/ dqg +6, qrw dv h{leoh dv d pdq0 xdo dssurdfk1
6 614
Sudfwlfdo Lvvxhv Kljk Dvvxudqfh Wudqvirupdwlrq
Krz grhv rqh ghprqvwudwh kljk dvvxudqfh iru dq lpsohphqwdwlrq +wdujhw, surjudp wkdw lv rewdlqhg wkurxjk wudqvirupdwlrqB Dv zdv phqwlrqhg lq Vhf0 wlrq 416/ wklv fdq eh dffrpsolvkhg e| suhvhqwlqj vxi0 flhqwo| frqylqflqj hylghqfh wkdw wkh wudqvirupdwlrq vhtxhqfh/ xvhg wr jhqhudwh wkh lpsohphqwdwlrq/ lv fru0 uhfwqhvv suhvhuylqj/ zklfk lq wxuq frqvlvwv ri vkrzlqj wkh fruuhfwqhvv ri hdfk lqglylgxdo wudqvirupdwlrq lq wkh vhtxhqfh1 Lq sudfwlfh/ yhuli|lqj wkh fruuhfwqhvv ri d vlqjoh wudqvirupdwlrq uhtxluhv +4, wkdw wkhuh lv d irupdo vh0 pdqwlfv iru wkh wdujhw odqjxdjh/ +5, wkdw wkh wudqviru0 pdwlrqv wkhpvhoyhv fdq eh jlyhq d irupdo vhpdqwlfv
+h1j1/ vfkhpd0yduldeohv/ hwf1, zlwk uhvshfw wr wkh vh0 pdqwlfv ri wkh wdujhw odqjxdjh/ dqg +6, wkdw wkh gh0 vluhg fruuhfwqhvv uhodwlrq +h1j1/ y , fdq eh d{lrpdwl}hg zlwklq wklv vhpdqwlf iudphzrun1 Jlyhq wkhvh uhtxluhphqwv lw vkrxog eh fohdu wkdw/ d surjudp wkdw lv wudqviruphg lq vxfk d pdqqhu wkdw lw v|qwdfwlfdoo| idoov rxwvlgh ri wkh wdujhw odqjxdjh/ zloo kdyh dq xqgh qhg vhpdqwlfv dqg fdqqrw vdwlvi| wkh fruuhfwqhvv uhodwlrq zlwk uhvshfw wr wkh vhpdqwlf iudphzrun wkdw kdv ehhq frqvwuxfwhg1 Wkxv lw lv hv0 vhqwldo wkdw wudqvirupdwlrqv uhzulwh surjudpv lq vxfk d zd| wkdw wkh uhvxowlqj surjudpv uhpdlq v|qwdfwlfdoo| zlwklq wkh wdujhw odqjxdjh1 Wklv suhvhqwv vrphzkdw ri d frqfhswxdo sureohp zkhq wkh remhfwlyh ri wudqv0 irupdwlrq vhtxhqfh lv wr pryh d surjudp iurp rqh ohyho ri devwudfwlrq +h1j1/ d kljk0ohyho vshfl fdwlrq, wr dqrwkhu +h1j1/ d frpslohdeoh lpsohphqwdwlrq,1 D vwdq0 gdug dssurdfk wdnhq wr dgguhvv wklv sureohp lv wr fuh0 dwh d zlgh0vshfwuxp +wdujhw, odqjxdjh ^6`^
W5 =@ w= Edvh Fdvh L= Wkh +uhpdlqlqj, dujxphqw lv d odpegd h{0 suhvvlrq
w @ hbolvw^=` + lg:4 1 h:4 ,^=` @@: hbolvw^=` h:4 ^=` Edvh Fdvh LL= D odpegd h{suhvvlrq kdv ehhq irxqg w @ hbolvw^=` hbolvw:4 / + lg:4 1 h:4 , ^=` @@: hbolvw^=` hbolvw:4 / h:4 ^=` Jhqhudo Fdvh= Wudyhuvh hbolvw:4 +w @ hbolvw^=` hbolvw:4 / h:4 ^=`, a hbolvw:5 @ W5 + hbolvw:4 , @@: hbolvw^=` hbolvw:5 / h:4 ^=`>
odpegdbh{sdqvlrq =@ w= +w @ h^=` lg:4 + hbolvw:4 , ^=`,
a
+hbolvw^=` + lg:5 1 h:5 , ^=` @ W4 + hbolvw:4 ,,
a
+ hbolvw:5 @ W5 + hbolvw:4 ,, @@: h^=` + lg:5 1 lg:4 + hbolvw:5 , ,^=`
8
Frqfoxvlrq
Ehfdxvh ri wkh srwhqwldo iru fruuhfwo| pdqlsxodw0 lqj surjudpv yld vpdoo fkdqjhv/ dq dxwrpdwhg qrq0 surfhgxudo wudqvirupdwlrq v|vwhp fdq eh d ydoxdeoh wrro iru frqvwuxfwlqj kljk dvvxudqfh vriwzduh1 H hf0 wlyh dxwrpdwlqj ri wkh wudqvirupdwlrq surfhvv uhtxluhv dq h{suhvvlyh frqwuro vwuxfwxuh1 Lq KDWV/ wklv lv uh0 dol}hg odujho| wkurxjk frqwuro0sdwwhuqv1 Iurp d sudf0 wlfdo shuvshfwlyh/ wkh wudqvirupdwlrq surfhvv vkrxog ehjlq lq wkh grpdlq odqjxdjh lq zklfk d sureohp lv lqlwldoo| h{suhvvhg1 Wkxv lq rughu iru d wudqvirupdwlrq v|vwhp wr eh sudfwlfdo lw pxvw eh h{leoh zlwk uhvshfw wr grpdlq odqjxdjhv1 KDWV kdv ehhq ghvljqhg wr id0 flolwdwh wudqvirupdwlrq lq wklv iudphzrun1
Uhihuhqfhv ^4` F1 P1 Kroorzd|1 Zk| Hqjlqhhuv Vkrxog Frqvlghu Irupdo Phwkrgv1 Surfhhglqjv ri wkh 49wk Gljlwdo Dylrqlfv V|vwhpv Frqihuhqfh/ Rfwrehu 4
