Implementation of a Real-Time Optical Flow Algorithm ...

Lpsohphqwdwlrq ri d Uhdo0Wlph Rswlfdo Iorz Dojrulwkp rq d Slsholqh Lpdjh Surfhvvru Pljxho Y1 Fruuhld W_ pfruuhldCih1xs1sw

Dxuìolr F1 Fdpslokr fdpslokrCih1xs1sw

Devwudfw

h

wdwlrq lq uhdo0wlph rq wkh Pd{Ylghr slsholqh lpdjh surfhvvru1 Uhvxowv duh glvfxvvhg lq vhfwlrq 81

Wklv sdshu suhvhqwv wkh lpsohphqwdwlrq ri dq rswlfdo rz dojrulwkp rq d slsholqh lpdjh surfhvvru1 Lw fdq rshudwh dw vwdqgdug ylghr iudph0udwh dqg uhvroxwlrqv/ 5 Wkh Slsholqh Lpdjh Surfhvv0 zklfk lv fxuuhqwo| xqdfklhydeoh zlwk frqyhqwlrqdo sur0 lqj V|vwhp fhvvruv iru lpdjh vhtxhqfhv ri hyhq prghudwh uhvrox0 wlrq1 Wkh ryhudoo rswlfdo rz frpsxwdwlrq phwkrg lv Pd{Ylghr lv dq lpdjh surfhvvlqj v|vwhp surgxfhg e| suhvhqwhg dqg vrph lvvxhv uhodwhg wr wkh lpsohphqwd0 Gdwdfxeh/ Lqf1 wkdw rshudwhv dv d gdwd slsholqh1 Lw lv d wlrq rq slsholqh lpdjh surfhvvruv duh glvfxvvhg1 prgxodu v|vwhp frpsrvhg ri 68 glhuhqw w|shv ri gh0 ylfhv/ vxfk dv= dqdorj0wr0gljlwdo dqg gljlwdo0wr0dqdorj frqyhuwhuv/ owhuv/ wlphuv/ pxowlsoh{huv/ dffxpxodwruv/ 4 Lqwurgxfwlrq phprulhv/ gdwd exvhv dqg jdwhzd|v ^7`1 Hdfk ghylfh Rswlfdo rz dojrulwkpv jhqhudoo| ghpdqg iru kljk shuirupv d vlqjoh dqg yhu| vshflf ixqfwlrq ru w|sh frpsxwdwlrqdo srzhu dqg kxjh vwrudjh fdsdflwlhv1 Rq ri ixqfwlrq vxfk wkdw lw grhv qrw vshqg wlph ordglqj wkh rwkhu kdqg/ dqg ghvslwh uhfhqw ghyhorsphqwv rq lqvwuxfwlrqv dv lq d jhqhudo sxusrvh surfhvvru1 Vlqfh doo ghylfhv rshudwh dw wkh vdph forfn iuh0 plfursurfhvvru whfkqrorj|/ wkhvh vwloo gr qrw dfklhyh txhqf|/ uhvxowv duh rewdlqhg dw wkh rxwsxw dw wkh vdph wkh wkurxjksxw uhtxluhg wr surfhvv d frqwlqxrxv udwh wkh lqsxw gdwd lv ihhg wr wkh slsholqh1 Wkhuh lv/ vwuhdp ri ylghr vhtxhqfhv ri hyhq prghudwh uhvroxwlrq1 krzhyhu d vpdoo whpsrudo ghod|/ xvxdoo| fdoohg wkh od0 Wkhuh duh pdq| lpsohphqwdwlrqv ri rswlfdo rz do0 whqf| ri wkh slsholqh wkdw frqvlvwv rq wkh {hg qxpehu jrulwkpv uhsruwhg e| vhyhudo dxwkruv/ exw wkh| duh ri forfn f|fohv qhhghg iru wkh gdwd wr furvv doo ghylfhv prvwo| h{shulphqwdo rqhv zlwk qr lqwhqw wr dsso| lw lqvhuwhg rq wkh slsholqh1 Lq wklv zd|/ wkh h{hfxwlrq lq dq| sudfwlfdo vlwxdwlrq uhtxlulqj rxwsxwv dw vwdq0 wlph ri dq rshudwlrq lv shuihfwo| ghwhuplqlvwlf1 Lw fdq gdug ylghr iudph0udwh1 Iru wklv sxusrvh/ rswlfdo rz eh fdofxodwhg e| pxowlso|lqj wkh forfn shulrg e| wkh dojrulwkpv duh jrrg fdqglgdwhv iru lpsohphqwdwlrq rq qxpehu ri sl{hov lq wkh lpdjh dqg dgglqj wkh odwhqf| ghglfdwhg kdugzduh vxfk dv wkh slsholqh lpdjh surfhv0 wlph1 vru xvhg lq wklv zrun1 Fxuuhqw Pd{Ylghr v|vwhpv rshudwh dw d forfn iuh0 Wkh remhfwlyhv ri wklv zrun duh wr ghyhors dq dojr0 txhqf| ri 53 PK}/ vr wr surfhvv dq lpdjh ri  e  ulwkp iru rswlfdo rz frpsxwdwlrq wkdw doorzv wr fkdu0 sl{hov d wrwdo ri 46 ploolvhfrqgv soxv odwhqf| wlph lv uh0 dfwhul}h dssduhqwh prwlrq lq lpdjh vhtxhqfhv iurp wkh txluhg/ xqohvv gdwd kdv wr furvv wkh vdph ghylfh pruh srlqw ri ylhz ri ylvxdo shufhswlrq1 Wklv dojrulwkp wkdq rqfh1 pxvw rshudwh dw vwdqgdug ylghr iudph0udwh/ wkhuhiruh/ Iljxuh 4 vkrzv d eorfn gldjudp ri Pd{Ylghr5331 dyrlglqj wkh uhtxluhphqw wr vwruh kxjh dprxqwv ri Doo ghylfhv duh frqqhfwhg wr rqh dqrwkhu wkurxjk d gdwd1  e  e  elw furvv srlqw vzlwfk frqwuroodeoh e| vriw0 Lq wkh qh{w vhfwlrq zh eulh | ghvfuleh wkh dufklwhf0 zduh1 wxuh ri wkh slsholqh lpdjh surfhvvru dqg lwv vriwzduh Wr surjudp wklv v|vwhp rqh vhohfwv wkh ghylfhv/ vxssruw1 Vhfwlrq 6 glvfxvvhv sudfwlfdo dvshfwv ri rs0 vshflhv wkhlu sdudphwhuv dqg hvwdeolvkhv wkh sdwk iru wlfdo rz frpsxwdwlrq dv zhoo dv lpsohphqwdwlrq uh0 gdwd wr furvv/ lq rughu wr shuirup wkh lqwhqghg wdvn1 odwhg lvvxhv1 Lq vhfwlrq 7 zh sursrvh dqg suhvhqw wkh Wklv lv dffrpsolvkhg e| wkh xvh ri d F odqjxdjh ol0 phwkrg dqg lpsohphqwdwlrq iru rswlfdo rz frpsx0 eudu| ri fdoodeoh urxwlqhv ghvljqdwhg LpdjhIorz/ sur0 W Vxss ruwhg e| I1F1W1 xqghu judqw EG265832 eh dovr eh frqvlghuhg glhuhqw irupxodwlrqv uhvxowlqj  dqdo|vlv ri vhyhudo lpdjh vhtxhqfhv rq dssolfd0 lq vshfldo fdvhv ri vsdwlr0whpsrudo owhulqj1 Vlprq0 wlrqv iru wkh vwxg| ri ylvxdo shufhswlrq lv d yhu| fhool dovr vxjjhvwv d frpprq lpsohphqwdwlrq iudph0 whglrxv wdvn> zrun zlwk glhuhqw ghjuhhv ri frpsoh{lw|1 Wkh hydoxdwlrq pdgh e| Eduurq hw do1 ohg xv wr  xvh ri wkhvh phwkrgv rq uhdo0wlph dssolfdwlrqv lv dgrsw wkh phwkrg ri Oxfdv ^;` wr frpsxwh rswlfdo rz yluwxdoo| lpsrvvleoh1 e| wkh iroorzlqj uhdvrqv= d orz frpsxwdwlrqdo frp0 \hw/ dqrwkhu sureohp dulvhv zkhq xvlqj vwdqgdug soh{lw| dqg udwkhu jrrg uhvxowv/ frqvlghulqj wkh dowhu0 ylghr htxlsphqw wr uhfrug uhdo lpdjh vhtxhqfhv uh0 qdwlyhv +vhh ^5` iru d ixoo glvfxvvlrq,1 Dovr/ Iohhw*v jduglqj wkh whpsrudo vdpsolqj udwh1 Lq prvw fdvhv/ uhfxuvlyh lpsohphqwdwlrq ghvfulehg lq ^9` lv d jrrg 58 ru 63 lpdjhv shu vhfrqg dqg iudph lqwhuodflqj/ do0 dfklhyhphqw lq whupv ri vlpsolflw| dqg frpsxwdwlrqdo wkrxjk txlwh vdwlvidfwru| iru ylhzlqj rq vfuhhq/ duh frvw1 Ilqdoo|/ Vlprqfhool sursrvhv wkh xvh ri Ndopdq qrw dw doo dghtxdwh iru lpdjh vdpsolqj dqg gljlwdo owhulqj lq d pxowl0vfdoh/ pxowl0uhvroxwlrq s|udplg wr surfhvvlqj wrzdugv prwlrq fkdudfwhul}dwlrq1 Lw lv/ lpsuryh hvwlpdwhv/ lq d frxuvh0wr qh dssurdfk/ dqg wkhuhiruh/ qhfhvvdu| wr dgrsw pxowl0uhvroxwlrq vfkhphv surylgh d phdvxuh ri frqghqfh lq wkh uhvxowv1 Wkh wkurxjk zklfk rqh fdq frpshqvdwh dftxlvlwlrq gh0 lpsohphqwdwlrq zh ghvfuleh khuhdiwhu iru wkh frpsx0 flhqflhv1 Pxowl0uhvroxwlrq klhudufklfdo vwuxfwxuhv duh wdwlrq ri rswlfdo rz lv edvhg rq wkhvh wkuhh zrunv1 Wkh phwkrg lv judglhqw0edvhg/ zklfk phdqv wkdw yh0 frpprqo| xvhg wr lpsuryh hvwlpdwhv rq vhyhudo lpdjh orflw| Y [ W lv frpsxwhg iurp uvw0rughu ghulydwlyhv surfhvvlqj dojrulwkpv dqg/ lq wklv sduwlfxodu fdvh/ sur0 ri lpdjh euljkwqhvv/ xvlqj wkh zhoo nqrzq prwlrq frq0 ylgh wkh phdqv wr uhgxfh wkh hhfwv ri whpsrudo dqg vwudlqw htxdwlrq= vsdwldo doldvlqj1 Wkh pdmru gl!fxowlhv wkdw dulvh lq wkh ghwhuplqd0 wlrq ri rswlfdo rz fdq eh vxppdul}hg dv iroorzv= U, [ W d Y [ W  ,W [ W  +4,  odujh txdqwlw| ri gdwd uhvxowlqj iurp wkh qhhg wr zkhuh U, [ W ghqrwhv wkh vsdwldo judglhqw ri lp0 surfhvv orqj lpdjh vhtxhqfhv> djh lqwhqvlw| , [ W dqg ,W [ W lwv sduwldo whpsrudo  doldvlqj lq vsdfh dqg wlph gxh wr wkh orz vdpsolqj ghulydwlyh1 Lq rughu wr rewdlq erwk frpsrqhqwv ri yh0 udwh ri prvw ylghr htxlsphqw> orflw| zh lpsrvh d frqvwdqw sdudphwulf prgho rq wkh yhorflw| hog dqg dvvxph vsdwldo dqg whpsrudo frkhu0  odujh dprxqw ri fdofxodwlrqv wr shuirup lq rughu hqfh orfdoo| zlwk vxssruw Z [ W / dv lq ^;`1 wr rewdlq d ghqvh rz hog1 Iru wkh lpsohphqwdwlrq ri wkh sduwldo glhuhqwldo o0 whuv/ zh frqvlghu d vhsdudeoh vprrwklqj owhu lq vsdfh0 Wkh vroxwlrqv irxqg vr idu e| wkh vflhqwlf frppx0 wlph/ vr wkdw wkh vsdwldo dqg whpsrudo frpsrqhqwv qlw| iru wkhvh sureohpv duh= fdq eh frpsxwhg lqghshqghqwo|1 Wr frpsxwh vsdwldo  wkh xvh ri ghglfdwhg kdugzduh iru lpdjh surfhvv0 ghulydwlyhv zh xvh d Jdxvvldq ILU owhu * [ zlwk lqj> d vxssruw ri  e  sl{hov iroorzhg e| qxphulfdo gli0 ihuhqwldwlrq zlwk d Vreho rshudwru1 Wkh whpsrudo o0  wkh dgrswlrq ri pxowl0uhvroxwlrq klhudufklfdo whu zdv ghvljqhg dffruglqj wr wkh zrun ri Iohhw e| d vwuxfwxuhv> fdvfdgh ri wzr wuxqfdwhg LLU h{srqhqwldov dv vkrz lq jxuh 51 Whpsrudo vprrwklqj dqg glhuhqwldwlrq duh  wkh ghvljq ri vlpsoh/ uhfxuvlyh dojrulwkpv1 erwk rewdlqhg vlpxowdqhrxvo| zlwk wklv owhu +vhh ^9` Slsholqh lpdjh surfhvvruv duh d vljqlfdqw frqwul0 iru surri,1 Wkh q sdudphwhu frqwurov wkh dprxqw ri exwlrq/ vlqfh doo surfhvvlqj dqg rxwsxw duh shuiruphg vprrwklqj iru wkh vdph whpsrudo vxssruw1 Wrwdo whp0 dw wkh udwh ri wkh lqsxw gdwd1 Qhyhuwkhohvv/ wkhuh duh srudo ghod| lv lq wkh rughu ri 6 iudphv/ odujho| uhgxflqj kdugzduh olplwdwlrqv erwk lq wklv udwh dqg lq wkh qxp0 vwrudjh uhtxluhphqwv1 ehu ri fdofxodwlrqv wkdw fdq eh shuiruphg1 Diwhu doo Iljxuh 6 suhvhqwv wkh ryhudoo vprrwklqj dqg gli0 wkhuh lv rqo| d olplwhg vhw ri ghylfhv rq d vlqjoh sur0 ihuhqwldwlrq rshudwlrqv dv wkhlu duh lpsohphqwhg iru fhvvru dqg shuirupdqfh kdv lwv frvwv1 wkh slsholqh lpdjh surfhvvru1 Hdfk eorfn uhsuhvhqwv 6

+

I(t)

q2

+

+

z-1

Î(t)

z-1 2q2

-2r

+

q2

+

-r

q

+ E1(t) z-1 -r2

q2

+

-

+

T

It(t)

E2(t)

It(x,t)

Iljxuh 5= Fdxvdo uhfxuvlyh whpsrudo owhu iru vprrwk0 lqj dqg glihuhqwldwlrq1 Wkh idfwruv kdyh wkh ydoxhv T q q   / U q r  q   +

+

Ix(x,t)

Iy(x,t) x

It(x,t)

x

x

(.)2

G(X)

G(X)

G(X)

G(X)

G(X)

E2(t)

E2(t)

E2(t)

E2(t)

E2(t)

E1(t)

E1(t)

E1(t)

E1(t)

E1(t)

b0

b1

A00

A10=A01

A11

(.)2

I(x,t)

G(x)

E2(t)

E1(t)

d(.) dx

d(.) dy

Ix(x,t) Iy(x,t)

Iljxuh 6= Ryhudoo vprrwklqj dqg glhuhqwldwlrq rshu0 dwlrqv1 d YHLO rshudwru wkdw pdsv wr wkh Pd{Ylghr kdug0 zduh1 Wkh yhorflw| rq hdfk sl{ho Y [ W lv rewdlqhg e| frpsxwlqj= e 7 vr 7 $ :$ Y [ W $ :E +5, e 7 v zkhuh $ : $ lv d  e  pdwul{ dqg $7 : E lv d  e  pdwul{ jlyhq e|/ uhvshfwlyho|= … 3  3 { Z, Z, [ ,\ 7 [ 3 3  $ :$ +6, Z,\ ,[ Z,\ … 3 { 3 Z,[ ,W $7 : E +7, Z,\ ,W

x x

+ +

+ -

-

+

(.)2

x

zkhuh vxppdwlrq lv wdnhq ryhu d vsdwlr0whpsrudo qhljkerukrrg } zlwk wkh zhljkwlqj idfwru Z shu0 iruphg e| dgglwlrqdo vprrwklqj rshudwlrqv/ dv vkrz lq jxuh 71 Wkh yhorflw| uhvxowv iurp vlpsoh dulwk0 phwlf lpsohphqwhg dv vkrz rq wkh uljkwprvw sduw ri wkh gldjudp1 Gxh wr kdugzduh olplwdwlrqv/ wkh dulwk0 phwlf rshudwlrqv duh olplwhg wr 49 elw lqwhjhu uhvrox0 wlrq dqg/ vlqfh wkhuh lv rqo| rqh dulwkphwlf xqlw uhfhly0 lqj d pd{lpxp ri irxu ; elw lqsxwv dqg surgxflqj d vlqjoh rxwsxw uhvxow dw d wlph/ wklv lv wkh prvw fulwlfdo sduw wr lpsohphqw1

8

x x

+

+

/

/

vx

vy

Iljxuh 7= Gldjudp ri yhorflw| fdofxodwlrq1

Uhvxowv

Wklv rswlfdo rz frpsxwdwlrq phwkrg zdv suhylrxvo| lpsohphqwhg lq vwdqgdug F odqjxdjh rq d Vxq zrun0 vwdwlrq dqg dssolhg rq h{shulphqwdo sv|fkrorj| wr 7

wr ghwhuplqh hjrprwlrq dqg prwlrq ri rwkhu remhfwv lq wkh vfhqh1 Wklv zloo doorz wr rewdlq d pruh frpsdfw dqg uholdeoh fkdudfwhul}dwlrq ri ylvxdo prwlrq1 Wkh qdo jrdo lv wr surylgh d uhdo0wlph/ sudfwlfdo wrro iru wkh hydoxdwlrq ri dssduhqw prwlrq lq vwxglhv ri yl0 vxdo prwlrq shufhswlrq/ xowlpdwho|/ zlwk dssolfdwlrqv frqfhuqlqj wkh ehkdylrxu ri gulyhuv lq urdg wud!f1 Lw pd| dovr eh dssolfdeoh wr dxwrqrprxv qdyljdwlrq dw wkh hduo| vwdjh ri prwlrq vhqvlqj1

Uhihuhqfhv ^4` S1 Dqdqgdq1 D frpsxwdwlrqdo iudphzrun dqg dq dojrulwkp iru wkh phdvxuhphqw ri ylvxdo pr0 wlrq1 Lqwhuqdwlrqdo Mrxuqdo ri Frpsxwhu Ylvlrq/ 5+6,=5;6643/ Mdqxdu| 4