Long Short-Term Memory in Recurrent Neural Networks

Long Short-Term Memory in Recurrent Neural Networks

` THESE N 2366 (2001) ´ ´ AU DEPARTEMENT ´ PRESENT EE D’INFORMATIQUE

´ ´ ERALE ´ ECOLE POLYTECHNIQUE FED DE LAUSANNE ` SCIENCES POUR L’OBTENTION DU GRADE DE DOCTEUR ES

PAR

FELIX GERS ¨ Hannover, Deutschland Diplom in Physik, Universitat de nationalite´ allemand

soumise a` l’approbation du jury: ´ Prof. R. Hersch, president ` Prof. Wulfram Gerstner, directeur de these Dr. habil. Jurgen ¨ Schmidhuber, corapporteur Prof. Paolo Frasconi, corapporteur Dr. MER Martin Rajman, corapporteur

Lausanne, EPFL 2001

!"#$% % &%(')$ &*+,'-./ &021

3 4(5(6"78596):

; =@?BABCEDGFIHJ?BKLCM= OQPRO ST>UWVYXZXZT>[]\_^T>V`Xbadce^TW\gfihdXZjklmS^^konpPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP OQPts uT>[`TWXbadceUvhJ[`kZwyxYTWXbaQ\ZwzhJ[`k>P]PqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP OQPtsPRO |}XZhd~czT>GhJ[YT>[]\ZwRadcExYT>UWahddXoadx`wzT>[]\wR[YLhdXb aQ\ZwzhJ[ePPqPP&PqPPqPqPPqP&PqP OQPtsPts hJcRVY\ZwzhJ[E}ihJ[k\Zad[\TWXZXZhdXUWaQXZhJVkT>cRkWPPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP OQPt |}XZTWwRhJV`kad[`xST>cRaQ\T>xhdXZjPqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP OQPtPRO S^^kP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP OQPtPts S^^kdTWXbkbV`k\ZYTWXT>]V`T>[ÙvT&|XbhUvT>kZkbwR[Y`XZhJadUb`T>k PqPP&PqPPqPqPPqP&PqP OQP£¢ V`\ZcRwR[YT PqPqPP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP ¥ ¦iA¨§`DGKy?¨K©C@=I§Mª«¬e¦% sPRO ®YhdXbfiaQXbx¯|GadkZk°P&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP sPts ±IT>aQXb[`wR[Y²PqPqPP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP sPt ³ádkjkhJczdT>xµfwz\Z³IXoadx`wz\ZwzhJ[àdcE±G³¶ PP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP · «i¸§À¨=´Km=I¹º?Cº»CA¨¹¸]?J¼½%C@=@?BKm=EFI§ªi¾qA¨¸D´KLHd?BKLC@=¿2Ky?¨ÀÁ«¬E¦ PRO ÂÃ[\Xbhx`VUv\ZwzhJ[PP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP PROQPRO ±wR wz\Zkhd´\Xbadx`wR\ZwzhJ[àdcE±Ý³¶ PqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP Pts hJcyVY\ZwzhJ[EÄ®YhdXZdTW\uaQ\T>kPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP PtsPRO ®`hdXZfiaQXox|GadkZk_hdGÄ\T>[`xYT>x±´³¶Åfwz\Z®`hdXZdTW\uaQ\T>kÆPqPP&PqPPqPqPPqP&PqP PtsPts ÇadUbj]faQXbx²|ádkbkhdÄÄ\T>[`xYT>x±G³¶Èfwz\Z®YhdXZdTW\quaQ\T>kÉPP&PqPPqPqPPqP&PqP PtsPt hJczTvYwz\gPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP Pt TWXbwR T>[\ZkPP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP PtPRO hJ[\ZwR[VàdcE&~T>x`xYT>xSTW~TWXuXbad aQX|XZhd~MczT> PPqPqPqPP&PqPPqPqPPqP&PqP PtPts ^TW\gfihdXZj³hdhJczhdd2ad[x|GaQXbadTW\TWXbk"PPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP PtPt }S_uËS_T>kbV`cz\ZkÌPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP PtP£¢ [àdczkZwRkhd´\ZYT}S_uËS_T>kbV`cz\Zk P&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP PtPtr hJ[\ZwR[VàdcE^hJwRkµ³IT> @hdXoadcIXbxYTWX|}XZhd~czT>ÍPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP P£¢ ihJ[UWcRV`kZwzhJ[ PP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP Î Ï2AQK©?BÀÐÑ¸]?BKLH²ÒÓ}¸]A¨§Y?BKLCM=ÔCM=Õ½%C@=@?BKm=EFI§ªi=Ó´FI?µ¬?AB¸§ÐÑÔ ¢YPRO ÂÃ[\Xbhx`VUv\ZwzhJ[PP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP ¢YPts TWXbwR T>[\ZkPP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP ¢YPtsPRO ^TW\gfihdXZj³hdhJczhdd2ad[x|GaQXbadTW\TWXbk"PPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP ¢YPtsPts ST>kZV`cz\Zk PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP ¢YPt ihJ[UWcRV`kZwzhJ[ PP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP wywRw

N r {

¡ ¤

;M; O>s O> Ov¢ ;N O>r O>r O>{ O>{ O sQÊ sO sO sd s¨¢ sdr sdr sd¡ ¥Ö sd¤ sd¤ QÊ QÊ O

wz N

×_ØÙÚGÛ}ÙÚÜ

i« ¸§À¨=´Km=I¹Ñ¾qA¨¸]HKLÔ>¸¦KmÐÑKm=¹¿ÝK©?BÀÁ¾¸¸]ÓGÀIC@ªL¸²«¬e¦% ·M· rPRO ÂÃ[\Xbhx`VUv\ZwzhJ[PP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP d rPts \T>[`xwR[Y±´³¶ÉfwR\ZÞ|TWTWMYhJczTihJ[[YT>Uv\ZwzhJ[`kZßàPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP ¨¢ rPt ®YhdXbfiaQXbx¯|GadkZk°P&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP dr rP£¢ uXoadx`wzT>[]\ÃágÇiadkZT>xÇadUZjfiaQXox¯|GadkZkàPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP d{ rPtr TWXbwR T>[\ZkPP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP J rPtrPRO ^TW\gfihdXZj³hdhJczhdd2ad[xÄTWXbwRT>[]\ZadcE|GaQXbadTW\TWXbk"PqPqPqPP&PqPPqPqPPqP&PqP d¡ rPtrPts ¶T>adkZVYXowR[YÝwRjdTâT>cRaklã¶²Yân¯P&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP d¤ rPtrPt uT>[YTWXbaQ\ZwR[Y ³wRT>xwRjdT>k&läu³`noP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqPp¢ rPtrP£¢ |´TWXbwRhx`wyU®V`[Ùv\ZwzhJ[¯uT>[YTWXbaQ\ZwzhJ[ålm|}®uqnPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqPp¢d¢ rPtrPtr uT>[YTWXbadc~MkTWXZQaQ\ZwzhJ[E^TW\gfihdXZjwR[`wz\ZwyadcRwzæ>aQ\ZwzhJ[çP&PqPPqPqPqPP&PqPPqPqPPqP&PqP rO rPt{ ihJ[UWcRV`kZwzhJ[ PP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP rO è ¬eKmÐºÓGªL¸½%C@=@?¸]éE? »IA¨¸]¸²§=DÕ½%C@=@?¸]éE?µ¬¸=IÔBKy?¨Kyê@¸¯«i§=¹MF§`¹¸Ô N· {PRO ÂÃ[\Xbhx`VUv\ZwzhJ[PP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP rd {Pts TWXbwR T>[\ZkPP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP r¨¢ {PtsPRO ³XbadwR[wR[Yad[`x³T>k\ZwR[YÑPqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP rdr {PtsPts ^TW\gfihdXZj³hdhJczhdd2ad[xÄTWXbwRT>[]\ZadcE|GaQXbadTW\TWXbk"PqPqPqPP&PqPPqPqPPqP&PqP rdr {PtsPt |}XZTWwzhJV`kXZT>kZVcz\Zk_PqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP rd{ {PtsP£¢ ±G³¶ëS_T>kZVcz\ZkìPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP rJ {PtsPtr [àdczkZwRkPqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP rd¡ {Pt ihJ[UWcRV`kZwzhJ[ PP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP {ds í ¦KmÐº¸¬¸A¨KL¸Ô ¾qA¨¸D´KLHd?§î´ªL¸ï¦%ÀABC@FI¹@À"¦KmÐº¸ðZñÈKL=DĆ]¿ÅÏ2Ó´ÓA¨C§H¨À¸]Ô èM· ]PRO ÂÃ[\Xbhx`VUv\ZwzhJ[PP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP {d ]Pts TWXbwR T>[\ZadcITW\ZV` PqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP {¨¢ ]PtsPRO ^TW\gfihdXZj³hdhJczhdd!P&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP {dr ]Pt ¶¯adUZjdTW]áòucRadkZkqaQhd\ZwRU³_wRT%TWXbwzT>kóPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP {dr ]PtPRO |}XZTWwzhJV`khdXZjpPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP {d{ ]PtPts ST>kZV`cz\Zk PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP {d{ ]PtPt [àdczkZwRkPqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP {J ]P£¢ ±adkTWXâaQ\Za PP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP ¨Ê ]P£¢YPRO |}XZTWwzhJV`khdXZjpPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP O ]P£¢YPts ST>kZV`cz\Zk PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP Qs ]P£¢YPt [àdczkZwRkPqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP Qs ]Ptr ihJ[UWcRV`kZwzhJ[ PP&PqPqPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP Q íN ô ½CM=HªmFIÔKLC@= ¡PRO ¶¯adwR[¯ihJ[]\Xbwz~MVY\ZwzhJ[`kiPPqPqPP&PqPqPPqPqPP&PqPPqPqPqPP&PqPPqPqPqPP&PqPPqPqPPqP&PqP Qr ¡Pts ®`V`\ZVYXZTfihdXZjad[`xhJkZkZwz~MczTaQcRwRUWaQ\ZwzhJ[khdÄ±G³¶ºP8PP&PqPPqPqPqPP&PqPPqPqPPqP&PqP Q{ íMí Ï õÐïî¸D´D¸Dö2¸]î}¸]A÷2A¨§ÐåÐº§À¬@?§`?BKLÔW?BKLHJÔ ø ¾¸¸]ÓGÀIC@ªL¸¯«¬E¦ ¿ÝK©?BÀÁ»CA¨¹¸]?÷2§`?¸]ÔÝKL="¾qÔ>¸FDC`ðbHJCED¸ íÖ ö2¸]ùú¸]A¨¸=IHJ¸Ô ôM· ¾¸]A¨Ô>C@=I§ªiö2¸]HCA¨D ÖMû

×_ØÙÚGÛÙÚ}Ü Ï2H¨üe=IC¿ÝªL¸D¹@Ðº¸=@?Ô

ÖÎ

w

×_ØÙÚGÛ}ÙÚÜ

_× ØÙÚGÛÙÚ}Ü O ý)þÿþÿ/ÿ Eþàÿbÿ! "Eÿ$# ½À§`Ó?¸A2¥ uTWXb&k %i®P%'P %YUb` wyx`]V`~@TW(X %!)`'P %"*ÉV wR[`k %G®iPçlãsQÊdÊdÊnoP ±eT>aQXb[`wy[Y¯\hLhdXZdTW\W¯ihJ[]\ZwR[Vàdc `XZT>x`wRUv\ZwRhJ[Ýfwz\Z±´³¶ P +-,/.1032546!7589:.;@?A75BC%!D5EÄlÃOWÊn %Es¨¢rO Fs¨¢]OQP uTWXb&k %Ä®iP%'P %Ub wRx`VY~TWX %G)Y'P %*,V` wR[`&k %I®iPºlÃO>¤d¤d¤¨~@noP ±eT>aQXb[`wy[Y\hLhdXZdTW\WÝihJ[]\ZwR[Vàdc `XZT>x`wRUv\ZwRhJ[fwz\Zå±´³¶ PÂÃI [ H0J7(K(LNM56PO++RQ STSVUMBW;=LX6!75BY&LZ75B[O0/;@? \K?@2>4!+-,/.103254]+-,&;@^_750`(a cl bÄhJcãPMd s %èP¡drQÊ¨áú¡drdrJnoPx`wR[]~VYXZJe %Ùvhd\ZcRad[xEÄÂòf %`±IhJ[`xYhJ[EP ½À§`Ó?¸AÝ· uTWXb&k %´®P&'P %]* Ub wRx`VY~TWX %P)YPlãsQÊdÊdÊdUnoP/^T>VYXbadcÄ`XbhUvT>kZkbwR[YµhdUvhJczTvUvhJ[\Zwy[]Vàdc}wR[YVY\ k\XZT>ad kWP&Âòg [ H0J7KhLiMje6P++RQ E1klklkhU]MB:;=Lj175?cBW;m6!75BY&Ln75Bg+o,&.10J254e+o,;@^_750`(a/LihJ hdXZ czT>aQXb[kUvhJ[]\Tv\LXZTWTad[`xÝUvhJ[]\Tv\ kT>[`kZwz\ZwRdTcyad[YJVàQdT>kWPqÂÃR [ M56PO++yxOzEdklk>D{6!75BY,/0,/B:K|,LµlmadUWUvTW`\T>xMn ½À§`Ó?¸AÝè uTWXb&k %J®iPQ&'P %JUZ j %â'P %T*-Uo` wRx]VY~TWX %5)YPElãsQÊdÊYOnoP`czwR[Y±G³¶Æ\h\ZwRTikTWXbwzT>k`XZT>xwRUv\ZaQ~czT \ZYXZhJVYJ2\ZwRTváäfwR[`xYhBf)aQ`XZhJadUb`T>kWPÂòg [ H0J7(K(LfM56PO++zEdklklDVU]MBW;=L6!75B(Y/Ln75BrO0/;@? \"K&?A254:+o,&.10J254 +o,;@^_750`ha/LobwzT>[`[`1 a %`V`k\XbwyaÄÂòf %`±ehJ[xYhJ[EP lmkZVY~M wz\\T>xMn Ò?BÀ¸]AÝÓ´FîGªLKLHJ§`?BKLCM=Ôd¼ lLÂòâqÂÃ \T>Ub`[`wyUWadceXZTWhdXZ\ZkaQXZT%[Yhd\cRwRkZ\T>ex %YkTWTqf_f_f&P wRx`kZwRaP UoEP n uTWXb&k %Y®iP%'P %Ub wRx`VY~TWX %})`'P %1*/V` wR[`&k %J®iPlÃO>¤d¤d¤QanoPihJ[]\ZwR[Vàdc``XbT>x`wRUv\ZwzhJ[ÝV`kZwR[Y±´³¶ fwz\Z2©hdXbdTW\JaQ\T>kWPÂò[¶ºP`¶aQXowR[àQXZ~h *ËSqPY³áQJcRwRaQLTWXZXbwGlm}x`k>P n %:+o,&.d03254:+-,&;caUnM3f+P?=,&;@0|?3SlSVU Hq037K,,5?cB}5aDlD>;cg750`(ad79I75Br+-,/.10J2>4 +o,;ca lLePeO>dBábO>d¡Jno P bwzTW\XowEkZV`cE¶aQXb>T %`Âú\ZadcRMXbwR[YdTWX bÄTWXbcRaQ< %ÇiTWXbcRwy[EP uTWXb&k %J®iPd&'P %T*çUo` wRx]VY~TWX %5)YPelãsQÊdÊdÊdanoP±G³¶ czT>aQXo[`kGUvhJ[\Tv\Ä©XbTWTicRad[`JVàQdT>kWPÂÃ[ CB 75^/?c0J Edklklk6!75B(Y&,&03,&B:K,(L

V` wy[`&k %]®i'P %@uTWXb&k %®i'P %:* Uo` wRx]VY~TWX %1)YPlÃO>¤d¤d¤JnoP±Iad[YJVaQdT&wRxYT>[]\Z'w UWaQ\ZwzhJ[µLXZhJ `XbhJkhx` fwz\ZYhJVY\TvcRwRUWwR\iLT>aQ\ZVYXZT>k>PqÂò[Hq037K,,5?cB}5a7JYtPoqdH_tt6PRQ STSºlcbÄhJcãPIO5%èPJOFJ ¢]noP

s

×_ØÙÚGÛ}ÙÚÜ Cpe $

Y® hdXaczhJ[Y&\ZwyT>%XZT>UWV`XZXZT>[]\[YT>VYXoadc[YTW\úfhdXZjklmS^^k nGfiTWXZT\Z`hJVYJ\i\h&~T\ZYTWhdXZTW\ZwRUWadcRcRLadkZUWwy[àQ\ZwR[YYP [`cRwzjdTk\Zad[`xaQXbxï©TWT>xYáä©hdXZfaQXbx [`TW\gfihdXZjk&S^^kqUWad[ºxYT>adcÄfwz\ZïaQXZ~Mwz\XbaQXZ²wR[YVY\kT>VYT>[ÙvT>kqwy[`k\T>adx hd´k\ZaQ\ZwRUwR[YV`\xàQ\ZahJ[`cRdP³_`wRkiUvhJ&~MwR[YT>x2fwz\ZÝ\ZYTqaQ~wRcywz\g\h T> hdXbwzæWTXZT>czTWQad[\TWdT>[]\ZkhdTWX\ZwRT aQjdT>kXZT>UWVYXbXZT>[\i[YTW\gfihdXZjkwy[`XbwR[UWwzadcYhdXZT@hBfTWXbLV`c\Zàd[ kZ\Zad[`xàQXbx ©TWT>xáäLhdXZfaQXbx [`TW\gfihdXZjk>P}³_YT kTW\}hdMhd\T>[]\ZwRadcàQ`cywRUWaQ\ZwzhJ[`kÄwRkGT>[`hdXbhJV`kWGad[]%\Zadkj\ZàQ\}XZT>]VwzXZT>kG\h%cRT>aQXb[Yhf"\h&V`kT_ T>hdXZwRka hd\T>[]\ZwRadc\ZadkZj ©hdXiXZT>UWVYXbXZT>[\i[YTW\gfihdXZjkWP}|´hd\T>[]\ZwRadcaQ`cRwRUWaQ\ZwRhJ[ aQXZT>adkwR[ÙWcyV`xYT\ZwRTkTWXbwRT>k}`XbT>x`wRUv\ZwzhJe[ % hd\hdXUvhJ[\XZhJcwR[[YhJ[áú¶¯aQXZjdhBwRad[T>[wzXZhJ[T>[\Zkad[`xXb]]\ZóxYTW\T>Uv\ZwzhJ[Ñlmwy[%VkZwRUad[`xkTWT>UoMnoP |}XZTWwzhJV`kÄkZVÙWUvT>kbkT>k}wR[XbT>adcYfhdXocRxaQ`cRwyUWaQ\ZwzhJ[`&k %dfwR\ZXZT>UWVYXZXZT>[]\[YTW\úfhdXZjkfiTWXZTcywR wz\T>ex %dYhBfTWdTW(X % x`V`Tq\hÝXbadUv\ZwRUWadc`XZhd~czT> k_fYT>[¯czhJ[YÝ\ZwRT%cyaQJk~TW\gfiTWT>[²XZT>czTWQad[\TWdT>[\Zk aQjdTczT>aQXo[`wR[Y x'w UWV`cz\WP ®YhdX\Z`T>kTqaQ`cRwyUWaQ\ZwzhJ[`kiUvhJ[dT>[]\ZwzhJ[àdcEdXoadx`wzT>[]\Ãáä~adkT>xµXZT>UWVYXbXZT>[\[YTW\úfhdXZjµadczdhdXbwR\Z` ki©hdX_cRT>aQXb[`wR[Y%\h k\hdXbTqwR[YLhdXb aQ\ZwzhJ[ÝhdTWX ,l;,&B:T,\ZwRTwR[]\TWXZQadcRki\ZaQjdTq\hh czhJ[YYP³_YT adwy[ÝXbT>adkhJ[LhdX\Z`wRkiLadwycRVYXZTwRk \ZYTXbaQMwRxµxYT>UWa>2hd~MadUZj]áä`XZhdaQJaQ\T>xTWXZXZhdX>P³_`TµÞ±IhJ[YÝYhdXb\³TWXb¶T>hdXZßlm±Ý³¶²nadczdhdXowz\Z` hBdTWXbUvhJT>k\Z`wykad[x¯XbT>cRaQ\T>xï`XZhd~cRT> k~T>[YLhdXbUWwR[` K75BBW;ÄTWXZXZhdmX `hBfqP kZwR[Y2dXbadx`wzT>[]\xYT>kZUvT>[]&\ % ±G³¶ÈTvcRwyUWwz\Zcz czT>aQXo[`kfYT>[\hÝk\hdXZT&wR[`©hdXb aQ\ZwzhJ[ad[`xµfYT>[\h adUWUvT>kZkwz\WP ÂÃ[Ý\Z`wRki\ZYT>kZwRkfiTTv\T>[èx %àd[adcz]æW>T %Yad[`xµaQ`Mcz\ZYT±´³¶,adczdhdXbwz\Z`P´ÂÃ[2aQXZ\ZwRUWVcRaQX %fTwRxYT>[]\ZwzL \úfhfiT>aQj[YT>kbkT>k´hd±´³N ¶ %B5h @TWXÄkhJcRV`\ZwzhJ[`kIad[x%hx`wzL\ZYTadczdhdXbwz\Z`àadUWUvhdXbx`wy[YJcz@lÃOnETXZT>UvhdJ[wzæWT afiT>aQj[YT>kZkhd`±´³¶Æ[YTW\úfhdXbjk`XZhUvT>kZkZwR[`_UvhJ[\ZwR[Vàdc]wR[YVY\Ek\XbT>ad kI\ZaQ\aQXZTi[Yhd\ 29:0|?A750?]kTWJT>[]\T>x wR[]\hkZV`~kT>VYT>[ÙvT>kqfwz\Z TvcRwRUWwR\Zcz¯ aQXbjdT>xºT>[`xk&aQ\f`wRUo\ZYTÝ[`TW\gfihdXZ:j kwR[]\TWXb[àdck\ZaQ\TµUvhJV`cRx ~T XZT>kZTW\WP´)wz\ZYhJV`\´XbT>kTW\Z&k %\Z`T_k\ZaQ\T a%dXZhBfwR[`x`&T [`wz\T>czqad[xTWdT>[\ZVàdcycz%UWadV`kZT\ZYT_[YTW\gfihdXZj\h~`XZT>aQj xYhBf[EPVYXXbT>T>xYwRkaq[YhBdT>Ac %]adxaQ`\ZwzdT ÞÃLhdXZdTW\}JaQ\T>ß%\ZàQ\T>[aQ~czT>kad[Ý±´³¶ UvT>cRcY\h%czT>aQXb[\hqXbT>kTW\ wz\ZkZT>czaQ\µaQ``Xbhd`XbwRaQ\T\ZwyT>&k %\ZV`k XbT>czT>adkZwR[YÑwR[\TWXo[àdc_XZT>kZhJVYXbUvT>kWP°lãsJn TïwRxYT>[]\Zwz© aÑfT>aQj[YT>kZk2wR[ ±G³{ ¶ kUvhJ[`[`T>Uv\ZwzhJ[µkZUbYT> >T %àd[`x2Tv\T>[`xµwz\i~ wR[]\XZhxVÙWwR[YÞÃTWTW`hJczTUvhJ[`[`T>Uv\ZwzhJ[`kZßLXZhJ ±´³{ ¶ k ÞbihJ[k\Zad[\¯XZXZhdX¯aQXZhJV`kT>cRß \h\ZYT %Vcz\ZwzcRwyUWaQ\ZwzdT¯JaQ\T>k¯`XZhd\T>Uv\Zwy[YÁ\ZYT>P³YT>kT UvhJ[`[YT>Uv\ZwRhJ[`k `XbhwRxYT\ZYT JaQ\T>kfwR\ZTvMcRwRUWwz\qwR[`©hdXb aQ\ZwzhJ[aQ~hJVY\&\ZYTÝk\ZaQ\TÝ\h¯f`wRUo\ZYTWUvhJ[]\XZhJcadUWUvT>kbkWPT kZ`hf-\ZàQ\@TWTWMYhJczTUvhJ[[YT>Uv\ZwzhJ[`kaQXZT[YT>UvT>kZkZaQXb LhdXi[V`TWXZhJVk}\Zadkjkad[`x2xYh[Yhd\ikbwzJ[`'w UWad[]\ZczVa T>Uv\ ±G³{ ¶ k@TWXb©hdXb ad[ÙvThJ[`XZTWwzhJV`kbcz khJczdT>x\ZadkjkWP TaQ`cR2\Z`T&Tv\T>[`xYT>x²±´³¶ëfwR\ZLhdXZdTW\JaQ\T>kad[xTWTWYhJczT%UvhJ[`[YT>Uv\ZwzhJ[`k\h2\Zadkjk\ZàQ\[Yh hd\ZYTWX S^^8adczdhdXbwR\Z` UWad[ÁkhJczdTºlmwR[UWcRV`x`wR[`\Xbadx`wz\ZwRhJ[àdc±´³¶ïno}uXbad aQX\Zadkjk ad[`x \T>hdXbadc hdXbx`TWXi\ZadkZjk_wy[dhJczwR[YUvhJ[]\ZwR[]Vadc@wy[YVY\k\XZT>ad k %aQXowz\Z`TW\ZwRUhdTWXbaQ\ZwzhJ[2hJ[UvhJ[]\ZwR[VàdcwR[YV`\ik\XbT>ad &k % \Zadkjk\ZàQ\XZT>V`wzXZT`XZT>UWwyk>T %EUvhJ[\ZwR[VàdcG\ZwR wR[Y[YTWXbaQ\ZwRhJ[ ad[`xïUvhJ[\Tv\qLXZTWTad[`x UvhJ[]\Tv\kT>[`kZwR\ZwzdT&cRad[YJVaQdTq\ZadkjkWPi®GwR[adcRcz fiT&T>kZ\ZaQ~cRwRkZcywR wz\ZkhdÄ±Ý³¶ÉhJ[\ZwRT&kZTWXbwzT>k`XbT>x`wRUv\ZwzhJ[ `Xbhd~czT> kkhJczQaQ~czT~2\ZwRTfwR[xYhfÕaQ`XZhJadUbYT>k>P

×_ØÙÚGÛÙÚ}Ü eÿ |´TWX hJcz\h\T> @hczT2XZTW\Zw[YT>V`XZhJ[àdcRwXbwRUvhdXZXbT>[\ZwkhJ[Yh kZ\ZaQ\TUvhJ[`kZwRxYTWXoaQ\T\TWhdXbwRUWadT>[]\TaVEadkZUWwR[àd[]\ZwãP ±IT XZTW\ZwXowRUvhdXZXZT>[]\ZwÄhJkZkhJ[Yh\XbaQ\\ZaQXZTµ[àQ\ZVYXbadcR T>[\T kZT>]VYT>[`æWT x`wxàQ\ZwwR[]dT>UvTÝx`wÄhd\TWX%XbwRUvTWdTWXZT2khJczh xàQ\Zw}k\ZaQ\ZwRUWwwR[wy[YVY\WP |´hJkZkhJ[`hwRaQXoaQXZT aT>hdXowzæWæ>aQXZT JcRwGTWdT>[]\Zw}wyhdXZ\Zad[\ZwäPy¡VYT>k\T UWaQMadUWwz\Ca ¢ czTXbT>[`xYhJ[YhwR[çcRwy[YT>ax`w`Xbwy[ÙWwzwzhÑ5w V-¢ hd\T>[\Zw%xYT>cRczTïXZTW\Zwq©TWT>xáäLhdXZfaQXbxEP±aÁUWcRadkZkTÑx`whd\T>[Yæ>wRadcRw aQ`McRwRUWaQæ>wzhJ[èw T£ adMwRaT>kbkZa%UvhJ[]\ZwzT>[YThdJ[`w@`XZhd~MczT> aqUoYTXowRUb`wRT>xYTAc Vkh&x`w@T>hdXbwya&wR[]\TWXb[àPÄcRUWV`[`w T>kT> w]kZhJ[YhcRaXZTWwykZwzhJ[YT}x`T>cRczTkZTWXbwzTk\hdXbwRUoYTqlL\ZwRTkTWXbwzT>k´`XZT>xwRUv\ZwzhJ[Mn %QwRcUvhJ[\XZhJcyczhx`T>c hd\hlm hd\hdX UvhJ[]\XZhJc©nwR[²ad&~wRT>[\Zw´[YhJ[¶aQXbjdhwRad[`w´TwycIXbwyUvhJ[YhJkZUWwRT>[]\hµxYT>cIXowz\Zh²lLTWXT>kT>MwzhÝ[YT>cycRa V`kZwRUWaÝh [YT>cycRacRwR[YJVà%aQXbcyaQ\ZanoP â adcz\Xoa%MaQXZ\>T %`kZwR[YhdXoa%czTXZTW\ZwXbwyUvhdXZXZT>[]\ZwMad[`[`haV`\h@hUvhkZVÙWUvT>kZkh [`T>cRAc aQ`cywRUWaQæ>wzhJ[YTa`Xbhd~czT> w XZT>adcywUWaQXbaQ\\TWXowzæWæ>aQ\Zwqxà wR[]\TWXZQadcRcRw_\T> @hdXoadcRwcyV`[YJ`w\XbaåTWdT>[\ZwwR @hdXb\Zad[\ZwxYT>cRAc wy[YVY\WP/ucRwadczdhQá XbwR\Z wUvhJ[dT>[Yæ>wRhJ[àdcRwEx`wEaQ`XZT>[`x`wR T>[\[]\>T %`àd[`[`h~MwRkhdJ[Yh x`w@\XZhd@h\T>hTWX wR aQXbaQXZTa T>hdXbwzæWæ>aQXZT²xYT>cRczTwy[Y©hdXo aQæ>wzhJ[`wUvhJ[-wR[\TWXbädcRcyw\T> @hdXoadcRw_cRV`[`J`wãPÁ±IaºXbaQJwzhJ[YT`XbwR[á UWwzMadcz[ T T £ cRaåXbaQMwRxàåxYT>UvXZT>kZUWwz\Zax`T>cRAc £TWXZXZhdXbTXZTW\XZhdXZhdaQJaQ\h lL~adUbjáäXZhdaQJaQ\ZwzhJ[ TWXZXZhdX noP°±ITXZTW\Zwx`w \ZwzhczhJ[YQágkbYhdXZ\\TWXb T> hdXZ)lm±G³¶ïn%5h @XbhJ[YhºV`[àïkhJcyVYæ>wzhJ[YTa VYT>k\h`Xbhd~czT> 1a %`XZhdhJ[YT>[`xYh V`P[ aQXbUb`wR\TW\\ZVYXba²xYhdTwR"c VkZkhïxYT>cRAc £TWXZXbhdXZT Xbwy ad[YTµUvhJk\Zad[\TdP kZad[`xYhAc wR[UWcRwR[àQæ>wzhJ[`T xYT>cidXbadx`wzT>[]\T lLdXbadxwzT>[\xYT>kZUvT>[]\on %@czT&XZTW\ZwI±G³¶ëhJkZkhJ[Yh wyaQXbaQXZTqVàd[`x`h Vè[ wR[`©hdXb aQæ>wzhJ[YTxYTWdTT>kZkTWXZT& T>hQá XbwRæWæ>aQ\ZaTq]Vad[`xYhQa kZVÙWUvT>kZkbwzäd T>[\T&VkZaQ\ZaP ¡V`T>k\Za\T>kZwJad[àdcywzæWæ>1 a %BT>kZ\T>[`xYTÄT>x&aQcRwRUWaAc adczdhdXowz\Zh±´³¶ PBwdwyxYT>[\Z¤w UWad[Yhx`VYTx`wR©TW\\ZwdxYT>cyAc adczdhdXbwz\Z h `XbTWT>kZwRk\T>[]\¥ T T¢ kZwXZhdhJ[YdhJ[Yhx`VYTT>k\T>[kZwzhJ[`w`XowR[ÙWwzadcywxYT>cyAc adczdhdXbwz\Z hïUoYTXbwykhJczdhJ[Yh w`Xbhd~czT> w XbwykZUvhJ[\XoaQ\ZwãPÂò[ÁaQXZ\ZwRUvhJcRaQXbTdÑlÃOnwzT>[YTwRx`T>[\Z'w MUWaQ\h V`[Áx`wzLTW\\h xYT>cR=c adczdhdXbwz\Zh±G³¶!UbYTadUWUWadx`T Vàd[`xYh2Ac wR[YMVYw\ T¢ UvhJ[\ZwRJVYVYT>[YæWT UvhJ[wR[wzæ>we-T [`w´x`wRk\Zwy[\ZwãP ÂÃ[]V`T>k\h UWadkwRadczTµT¯w_ädczhdXowwR[]\TWXb[`whJkZkhJ[`hºUvXZT>kZUvTWXZTïwRcRcywR wz\ZaQ\ZadT>[]\TUWadV`kZad[xYhºV`[à aQXbadcRwRkbwxYT>ckZwyk\T> aP ÂÃcÄXbwR T>x`wzhµ`XZhdhJk\hkZw~adkZakZVÑV`[à[VYhQaV[`wzh\ a£ hJcz\ZwRcRwRUWaQ\ZwzQaïlLJaQ\T2V`[wz\onadxàQ\\ZaQ~wRcRT Ub`wyad aQ\h ÞÃLhdXZdTW\JaQ\T>ßP}}kbkZa&TWXb TW\\Tadx2V`[à%UvT>cRcRaxYT>cRcRaXbTW\T±G³¶8x`w@wRaQXoaQXZTa%Xbwz\hdXb[àQXbTadxµV`[Yhk\ZaQ\h `XbT>UvT>xYT>[\T%wR[hJT>[]\ZwEhd`hdXZ\ZV`[Aw %cywz~TWXbad[`xYhUvhJk £wEXbwykhdXbkTqwR[]\TWXb[YTdP lãsJnwGwRx`T>[\Z'w MUWaÝV`[ïx`wR©TW\\h[YT>cRcRh2kZUoYT> axYT>cyczTUvhJ[[YT>kZkZwzhJ[wxYT>cyczT%XbTW\ZwÄ±´³¶ÌT czhkZw´XbwRkhJczdTwR[]\XZhQá x`VUvT>[`xYh¯UvhJ[[YT>kZkZwzhJ[wUo`wRad aQ\TÁÞÃTWTWYhJcRT2UvhJ[`[`T>Uv\ZwzhJ[`kZßPºkZkTUvhJcRczTWJad[`hï=c V`[`wzh\ a¯£ UvT>[\XbadcRTºlvÞUvhJ[á k\Zad[]\TWXZXZhdX_UWaQXZhJVkT>cRßnixYT>cRcRTUvT>cRczTadcRcRTV`[wzh\ a%£ hJcz\ZwzcRwyUWaQ\ZwzdTUoYTqczTk\Zad[`[YhaQ\\hdXb[YhYPÂò[µVYT>k\hhxYh dT>[YdhJ[`hÝLhdXb[`wz\T%adcRczT&V`[wzC\ a ¢ wR[YLhdXb aQæ>wRhJ[`wET>kcywRUWwz\TqkZV`cycRa UvhJ[`x`wRæ>wzhJ[YT&xYT>cyAc £hdddTW\\h2xwIUWV`wUvhJ[\XZhJcycRad[Yh Ac adUWUvT>kbkhYP&w hJk\Xba2wR[YhJcz\XZTUbYTczT%TWTWYhJczTUvhJ[`[YT>Uv\ZwzhJ[`kkhJ[`hµ[`T>UvT>kZkZaQXbwzTTWX[V`TWXbhJkZwe`Xbhd~czT> w TUbYT[YhJ[¯XbwRx`VUvhJ[Yh kZwzJ['w UWaQ\ZwzQadT>[]\T%cRa @TWXb©hdXb ad[ÙvT%xYT>cyczTqXZTW\Zw´±´³¶ kZVXZhd~czT> we`XZT>UvT>xYT>[]\Tvá T>[]\T&Va XZhJ[]\ZaQ\ZwãP ±a\T>kZwEaQ`cywRUWaAc adczdhdXbwz\Z h±G³¶ÅT>k\T>kh UvhJ[©hdXZdTW\JaQ\T>k_TTWTWYhJczTUvhJ[`[`T>Uv\ZwzhJ[`k_a`Xbhd~czT> w UoYT[YT>kZkbV`[ adcz\XZh%adczdhdXbwz\Zh&@TWXXZTW\ZwYXbwyUvhdXZXZT>[]\ZwY¦V hq£ XowRkhJczdTWXZT&lmUvhJ `XZT>khczT_XZTW\Zw`±G³¶ \Xbadx`wRæ>wzhJ[àdcRw©no `Xbhd~czT> w`xw`dXbad aQ\ZwRUW1a §`XZhd~czT> wx`whdXbx`wR[àd T>[\Zw\T>hdXbadcRwUoYTUvhJwR[dhJcRdhJ[Yhwy[YVY\}UvhJ[]\ZwR[VÀw §hdYá TWXbaQæ>wRhJ[`wGaQXbwz\ZTW\ZwRUoYTkZVwR[YMVY\UvhJ[]\ZwR[VY wÚbYTXbwyUb`wzT>x`hJ[Yh2V[àÝUvhJ[]\ZwR[VàµT`XZT>UWwRkba2 wRkZVYXba xYT>c´\T>[YTWXoaQæ>wzhJ[YT x`wLV[Yæ>wzhJ[`wTWXbwzhx`wRUb`TqTXbwRUvhJ[YhJkZUWwyT>[\hx`w´dXbad aQ\ZwRUb`T UvhJ[\Tv\Ãáä©XbTWT TUvhJ[\Tv\ÃágkT>[`kbwz\ZwzdTdPÂÃ[\Z¤w UWad[YhÝx`T>wcRwy wz\ZwexYT>cyAc adczdhdXbwz\Z hÝ±Ý³¶T>k\T>khXZT>cyaQ\Zwzwa2`XZhd~Yá czT> w´xwG`XZTWwRkZwzhJ[YTx`wÄkZTWXbwzT k\hdXbwyUbYT UoYT khJ[YhXbwRkhJczwz~wycRwexàdcRcya2UWcyadkZkT x`wÄTW\hx`wÄ~MadkZaQ\ZwkZ V [`T>k\XZT \T>hdXbadcywãP

¢

×_ØÙÚGÛ}ÙÚÜ

¨

²

©«ªX¬®~¯Z°

596¥³)4µ´ ¶

±

Ë6¹¸4(5 ·

³_`TdhJadc`hd@\Z`wRkÄ|}eP â%P\ZYT>kZwRkwykG\hqTv\T>[xe%]ad[àdcR]æWT>%ad[`x aQ`cR%aXZT>UvT>[]\&%[YhdT>cA%]`XZhJ wykZwR[YdXbadx`wzT>[]\ czT>aQXo[`wR[YadczdhdXbwR\Z`LhdXXZT>UWV`XZXZT>[]\[YT>VYXbadc[YTW\gfihdXZjklmS^^konoP_³_`T&adczdhdXbwz\ZwRkUWadcyczT>» x ºc¼G75B}rxµ~]Ã ÄhUoYXZT>wz\TWXad[`x¯Uo` wRx`VY~TWXlÃO>¤d¤JdnoP ÅÆÅ

Ç

_ee_ È®e fgÈ®]1XieÉËÊ Ç

È®È®PÌ

³_`T%S^^kfiTUvhJ[kZwRxYTWXYTWXZTUvhJ[`kZwRk\hd}V`[`wR\ZkwR[]\TWXbadUv\ZwR[YµwR[¯x`wRkZUvXZTW\T%\ZwRTwya x`wzXbT>Uv\T>xe%MfiT>wzJ]\T>x UvhJ[`[`T>Uv\ZwzhJ[`kÄfwR\Z%fiT>wzJ]\Zk ÍfÎÐÏ lL©XbhJ V`[`wz]\ Ñó\hqV`[wz!\ ÒãnoPÄ}dTWXZV[`wz\Äàdkad[ adUv\ZwRäQ\ZwzhJ~[ Ócl ÔZn´VYExàQ\T>x aQ\TWdTWXZ\ZwyoT ÔÕÆ5O Öbds Ö&×&×&×BP³_YTadUv\ZwzQaQ\ZwzhJ[`khd\ZYTV`[`wR\Zk_LTWT>x`wR[YµwR[]\hÝhd\ZYTWXV`[wz\Zk_LhdXb8\ZYTk\ZaQ\Î T hd´\ZYTq[YTW\úfhdXZj@P[adUv\ZwzQaQ\ZwzhJr[ Ó Î hdGV`[`wz\ ÒGwRkVYExàQ\T>xµ~µUvhJVY\ZwR[`wR\Zk_[YTW\gfihdXZjwR[YMVY\ikbV`¾ ØGÙ&Ô % Î ØGÙÔ cl Ô n ÕÛÚ ÍfÎÜÏÝÓ Ï cl Ô5Þ On Ö Ï

ad[`xÞkb]Vàdkb`wR[YJßwz\fwz\Za x`wßTWXZT>[]\ZwRaQ~czTmV`[Ùv\ZwzhJ[rà$%àdUWUvhdXbxwR[YY Î Ó cl ÔZ n Õ¹àÄcl ØGÙÔ Î cl ÔZná n × ÂÃ[YVY\ad[`xµhJV`\VY\\h \ZYT%[YTW\gfihdXZjaQXZTq\ZwR TváääQXZwR[Y kZTWXbwzT>khdGdT>Uv\hdXáäaQ\\TWXb[kUWadcRcRT>xkT>VYT>[ÙvT>kWP T_x`&T [YTczT>aQXb[`wR[`wR[S^^kÄadkÄhd`\ZwR wRæ>wR[Yax`'w @TWXZT>[]\ZwRaQ~cRT}hd1~ âT>Uv\ZwzdTmV`[Ùv\ZwzhJ[ ã~%]kZV` T>xhBdTWX adcRc´\ZwR TkZ\TWkhdiadcyc´kZT>]VYT>[UvT>kWP%~¯adxaQ`\ZwR[Yµ\ZYT UvhJ[`[`T>Uv\ZwzhJ[ïfT>wRJ\ZkWP ã wyk~MadkT>xïhJ[ kZVYTWXZwRkT>x \ZaQXZdTW\Zk ÔJäl%`f`TWXZsT åwR[`x`TvT>k\ZYT&hJVY\VY\_V`[wz\ZkhdG\ZYT&[YTW\úfhdXZjµfwR\ZadUv\ZwzQaQ\ZwzhJ[`fk Ó1äJP[TvadMczTqwRk \ZYTqkb]VàQXbT>xTWXZXZhdX_hdUv\ZwzdT&LV`[Uv\ZwzhJ[E O Ú Ù lcÔZnçæ®§iÙ lcÔZnièÕ¥Ô ä lcÔZn]ÞéÓ ä lcÔZn Ö ãÝlcÔZnÕ ä s ä ä f`TWXZ_T Ù ä xYT>[Yhd\T>k\ZYTTv\TWXb[àdcyczwyV[ âÃT>Uv\T>x&TWXZXZhdX §>ã2cl ÔnEXZTW`XZT>kT>[]\Zke\Z`TTWXZXZhdXáQ\I\ZwR!T ÔLhdXIhJ[YTikT>VYT>[ÙvT UvhJhJ[YT>[]\UWadcRczT>xïaQ\\TWXo[EP%®YhdX%aµ\gwRUWadcÄxàQ\ZakTW\qUvhJ[kZwRk\ZwR[`ÝhdikT>]V`T>[ÙvT>khd}MaQ\\TWXb[`&k %Gã wRk\ZYT kZV hd] ã2cl ÔnhdTWXadcycEaQ\\TWXb[`k_hdGadcycEkT>]V`T>[ÙvT>k_wR[µ\ZYT%kTW\WP r

{

×]êëìÚÄÛ_í»î>ïðòÙÚ]íqØqñÃò×Ú$ðWØÙ Out

Out

Out

Out

In

In

In

In

G® wzJVYXbTOQPROQ}±ITW©\W®`TWT>xáä©hdXbfiaQXbx[`T>VYXbadc[YTW\gfihdXZj@P¶¯wRx`x`czTdG±IadTWXZT>x[YTW\úfhdXbj fwz\Z2ad[wR[YMVY\cRadTWX%à mV`cRcz XZT>UWVYXZXZT>[]\_`wRxxYT>[µcRa>dTWXad[`xad[hJVY\MVY\_cRa>dTWXP}SwzJ]\W}®`VcRczÝUvhJ[[YT>Uv\T>xXZT>UWV`XZXZT>[]\_[YTW\gfihdXZj@P °dXbadx`wzT>[]\xYT>kZUvT>[]\qczT>aQXb[`wR[`ÝadczdhdXbwR\Z`8©hdXqS^^k&%ekZVÙoïadkq±´³¶N%UvhJVY\T>k\Z`TdXoadx`wzT>[]\hd °fwz\ZµXZT>kT>Uv\_\h T>adUbfiT>wzJ]\fÍfÎÜÏ \h xYTW\TWXb wR[YT\ZYTfiT>wzJ]\Ubàd[`dT>kóÍfÎÐÏ& ãÝcl ÔZn óÍfÎÜÏlcÔnôÕ ÞiõÝö Ö ã

ö

ÍfÎÐÏ

f`TWXZTsõåwRk_UWadcRczT>x\ZYT&czT>aQXb[wR[YXbaQ\TdP®`hdXad[TvYUvT>cRczT>[]\wR[]\XZhx`VÙv\ZwzhJ[µ\hÝdXbadx`wzT>[]\_czT>aQXb[wR[Y wR[S^^k kTWT&-wRcycRwRad kiad[`R x ÷wRkTWX&lÃO>¤d¤dsJnoP ³YTUvhJ[`[YT>Uv\ZwzhJ[ kbUbYT>Thda[YTW\úfhdXZjwRkÄUWadcRcRT>x\ZYT[YTW\gfihdXZjaQXbUbwz\T>Uv\ZVYXZThdX\hdhJczhdddPÄXbUbwz\T>U á \ZVYXbT>kfwR\ZYhJVY\czhhdkaQXZTUWadcRczT>xLTWT>xáäLhdXZfaQXbxï[YT>VYXbadc[YTW\gfihdXZjklm®GwzJV`XZT2OQPR5O %EczTWL\onoPS^^ \hd@hJcRhdJwzT>k Xbad[`dT©XZhJaQXZ\ZczÝXZT>UWVYXZXZT>[]&\ %Y\h mV`cRcz XZT>UWVYXZXZT>[]\_[YTW\úfhdXZjkWPi[µTvYadczThdÄaaQXb\ZczÝXbT>UWVYXZXZT>[]\_[YTW\Ãá fihdXZj wRkiacRa>dTWXbT>x[YTW\gfihdXZj fwz\ZÝxwRk\ZwR[Ùv\iwR[YV`\}ad[`x2hJVY\VY\cRadTWXb&k %fYTWXZT\ZYTXZT>UWVYXbXZT>[ÙvTwRkcywR wz\T>x \h\ZYT`wyx`xYT>[²cRa>dTWXBlmk n %adkqkZ`hf[ïwR[²\ZYT wyx`x`czT%hd®GwzJVYXZTOQPROQPÂÃ[ïLV`cyczXZT>UWVYXZXbT>[\[YTW\úfhdXZjkT>adUo [YhxYTdTW\ZkwR[YVY\iLXZhJadcRchd\ZYTWX[YhxYT>kqlm®GwzJVYXZT OQPR5O %XbwzJ]\onoP ÿ Eÿ S^^kUvhJ[k\Zwz\ZVY\TaqdTWXZ @hBfTWXbLV`cMUWcRadkZkhdeUvhJVY\ZaQ\ZwRhJ[àdcM hxYT>cyk&%]UWaQaQ~MczThdIwy[`k\Zad[]\ZwRaQ\ZwR[Y&adcyhJk\ aQXZ~Mwz\XbaQXZ2xY[ad wRUWkqläwzTWdT>cy ad[`r[ *°hJ[]\ZaQ¤d¤OnoP kZVàdcyczÝ\úfhµ~adkbwRUq\ú]T>khd}S^^/aQXZTx`wRk\ZwR[`JV`wRkZYT>xe}adVY\hJ[YhJhJVkS^^kfwz\ZUvhJ[]dTWXZJwR[`2xY]á [àd wRUWk_f`TWXZT&\ZYTwR[YV`\wRfk `T>ex %M©hdXTvYadcz-T ÄhdC T>cRx[YTW\úfhdXbjk@l ÄhdC `T>cRex %Ó>¤d¡dsJnhdXÇihJcz\æ> ad[`[ adUo`wR[YT>k_l@ÄwR[]\hJ[e%çT â[YhBfkjAw %}*çUZjczTW> %@O>¤d¡¨¢]n %dTWXokZV`k[YhJ[YágadVY\hJ[YhJhJV`k}S^^k}fwz\Z\ZwRTváäQaQXZwR[Y wR[`VY\ZkWP³YTÝS^^kUvhJ[`kZwRxYTWXbT>xºwR[Ñ\Z`wRk%\ZYT>kZwRkaQXZT2hd_\ZYTµcRaQ\\TWXUWcRadkZkWPï³_YTW UWad[åTWXZLhdXbëdXbadx`w á T>[]\&xYT>kZUvT>[]\&wR[adTWXZïdT>[YTWXbadc}kZadUvT hdhd\T>[\ZwRadcycz¯[YhJwykTváäXZT>kZwRkZ\Zad[\qadczdhdXowz\Z` kqVkZwR[Yx`wRkZ\Xbwz~VY\T>Px % UvhJ[]\ZwR[VYhJV`kÃáäQadcRVYT>xÝwR[\TWXo[àdc`k\ZaQ\T>kWP³_YTaQ~MwRcRwz\ú%\h aQ2XZT>adc áäQadcRVYT>x wy[YVY\kT>VYT>[ÙvT>k\hXZT>adc áäQadcRVYT>x hJVY\MVY\qkT>VYT>[ÙvT>&k %´ aQjwy[YV`kT hdi\ZYT>wzX%wR[\TWXo[àdcÄk\ZaQ\T \h²wR[ÙvhdXZhdXbaQ\TMadk\&UvhJ[]\Tv&\ %Ä aQjdT>k&\ZYT> aïXZT> aQXZjäQ~MczºdT>[YTWXoadc_kT>VYT>[ÙvT`XbhUvT>kZkbwR[YxYTWwRUvT>klmÇiT>[YJwz¤d¤dJnoP S^^kaQXZTÝT>kZ@T>UWwyadcRczï`XZhJ wykZwR[YLhdX\Zadkjk\ZaQ\%XZT>V`wzXbT \hïczT>aQXb[åYhf \hV`kTµT>hdXZdPï|´hd\T>[\Zwyadc aQ`McRwRUWaQ\ZwzhJ[`k&aQXZTd \ZwRT2kTWXbwzT>k%`XZT>xwRUv\ZwzhJ["lLTdP£Y'P %}hdf [àd[ÙWwyadc}kTWXbwRT>kon %G\ZwRTµkTWXbwzT>k&`XZhx`VÙv\ZwzhJ[-lLTdP£Y'P % hd\hdXUvhJ[]\XZhJcwy[ [YhJ[áú¶¯aQXZjdhwRad[ÑT>[wzXZhJ[T>[\Zk nad[`xº\ZwRT kTWXowzT>k&UWcRadkbkZ'w UWaQ\ZwzhJ[ hdXcRaQ~T>cRwy[Y lLTdP£Y'P % Xb]\Z`óxYTW\T>Uv\ZwzhJ[¯wR[%VkZwRUad[`xkTWT>UoMnoP ÅÆJø

ù

_e fR! e " Æ

Û >Ø ÄÜ Ù

Û ÚGÛñÑØqí

î>ïûúTïüì_í sý¦ð ò ë ñþí ÿGë

"!#!%$&')(#+*,'.-/021 '(3& !%$4& !305" 67'+$& "!8

_³ `T%Tv\T>[\&\hf`wRUo¯\ZwRk@hd\T>[]\ZwRadcGUWad[~TTvcRhJwz\T>xe%ewRkYhBfTWdTWX&cywR wz\T>x¯~]¯\ZYTT/@T>Uv\ZwRdT>[YT>kZkqhd \ZYTÝ\XbadwR[`wy[Y`XZhUvT>x`VYXZTÝaQ`cRwzT>xePuXoadx`wzT>[]\&~adkT>xÑTW\ZYhx`kµlmkTWTÝkZV`XZdTWMlm|´T>aQXbcRVY\\TWX%_O>¤d¤drJnn9 ÞÇadUZj]ág|XbhdaQJaQ\ZwzhJ[)³_YXZhJV`Jå³wRT>ßÁlL)wRcRcRwRad k* ÷wzkTWX(%O>¤d¤dsd§TWXZ~hJk&%O>¤d¡d¡JnhdXºÞS_T>adc ág³wRT ST>UWVYXZXZT>[]\e±eT>aQXb[wR[YJßqlmS_hd~MwR[`khJs[ *Á®àdcRcRkZwyxY>T %O>¤d¡J} §J)wRcRcRwyad ek *÷wzkZTWX %O>¤d¤dsJnIad[`x\ZYT>wzXUvhJ&~wy[àQ\ZwzhJ[ läUo` wRx]VY~TWX %O>¤d¤dsQan 9-kbàQXZT ad[çwRhdXZ\Zad[]\ÝcywR wz\ZaQ\ZwzhJ[ePà³_`T¯\T>hdXbadcTWdhJcRVY\ZwRhJ["hd%\ZYT²aQ\Z wR[]\TWdXbadcÄhBdTWXadcycÄTWXZXZhdX&kbwzJ[àdcRkJÞ `hBfwR[Y~adUZjwR[\ZwyT>ßµTvhJ[YT>[]\ZwRadcRcR¯xYTWT>[`x`khJ[º\ZYT aQJ[wz\ZV`xYT hd\ZYT fiT>wzJ]\Zk2@l ÄhUbYXZT>wR\TWX %O>¤d¤OnoP³_`wRkqwycRwzT>k\ZàQ\%\ZYT ~MadUZj]áä`XZhdaQJaQ\T>xåTWXZXZhdX]VwRUZjcz²T>wz\ZYTWX Qad[`wRkZ`T>k hdXµ~czhBfk2VY @l ÄhUbYXbT>wz\TWrX * YUb` wyx`]V`~@TW(X %O>¤d¤J} §ÇiT>[YJwz¤d¤¨¤d¤ds¨~no P ÄT>[ÙvT²k\Zad[`xàQXoxÁS^^k Ladwyc\hÑczT>aQXo[ÁwR[\ZYT`XZT>kT>[ÙvThdqczhJ[Y\ZwRT¯cRaQJk ~TW\gfiTWT>[çXZT>cRTWäd[]\wR[YV`\Ýad[`x)\ZaQXZdTW\TWdT>[\ZkWP³ádkjkfwz\Z-\ZwR TïcyaQJkdXZT>aQ\TWX\Zad[Õ(r FMOWÊÁadczXZT>adxY ~T>UvhJTix`'w UWV`cR\e©hdX´\ZYT>[%\hcRT>aQXb[&fwz\Z`wR[XbT>adkhJ[àQ~czT\ZwRT@l ÄhUbYXbT>wz\TWX *-Uo` wRx]VY~TWX %]O>¤d¤JdnoP}³_YT Qad[`wRkZwR[YTWXZXZhdX`XZhd~MczT> UWadkZ\Zkx`hJVY~`\hJ[²f`TW\ZYTWXkZ\Zad[`xàQXbx²S^^kUWad[ïwR[`x`TWT>xTvY`wz~wz\kZwzJ[`'w MUWad[\ `XoadUv\ZwRUWadcGadxYäd[]\ZaQdT>kqhBdTWX&\ZwR TváäfwR[`xYhBfáä~adkT>x¯LTWT>xáä©hdXbfiaQXbx[`TW\gfihdXZjkÝ@l ÄhUbYXbT>wz\TWX %}O>¤d¤5O §GÇiT>[YJwzh TW\adcä'P %eO>¤d¤¨¢]noP

: ;Uv\ZwzhJ[fiT fwRcRc´JwzdTÝa~`XbwzTWhdTWXZwzTWf hJ[ºadcz\TWXb[aQ\ZwzdT aQ``XbhJadUbYT>k&©hdXq\ZwyT kTWXbwzT>kq`XZhUvT>kZkÃá wR[` ad[`xÁ`XbhwRxYThJwR[]\TWXbk \hÑ\ZYThdXbwzJwy[àdcfihdXZjk>P-³_`XZhJVYJYhJVY\ \ZYT\ZYT>kZwyk fiTfwRcycixwRkZUWV`kZk \ZYT>wzX aQ`McRwRUWaQ~wRcywz\gLhdX_\ZYTQaQXbwzhJV`k_\ZadkjkfTqwy[dT>k\ZwRJaQ\TdP ¦KmÐº¸Ñ¿ÝKm=DC¿§`Ó´ÓÁBCM§`H¨À¸EÔ S [k\ZaQ\ZwyUaQ\\TWXb[Ñ aQ\ZUb`wy[YxYTWwRUvT¯lLTdP£Y'P %GLTWT>xáä©hdXbfiaQXbxÑ[YTW\Ãá fihdXZj`nfwz\ZÁ{ a YT>x\ZwRTfwR[`xYhBfhdXZT>UvT>[\2wR[YV`\ZkUWad["kTWXbdT¯adk \T> @hdXoadckT>VYT>[ÙvTXZhUvT>kbkZwR[Y kk\T>P³`wRkiaQ``XZhJadUoàdkkTWdTWXbadcEkbwzJ[`'w UWad[]\xYXba>f~adUZjkWlÃOnÂò\wRkx`'w UWV`cR\}\h xYTW\TWXb wR[YT\ZYThdYá \ZwR adc\ZwR Tfwy[`xYhf)kZwzæW>T %Ywz\i\ZYTWXZTwRkiad[]dPlãsJni®YhdX\Zadkjkfwz\Z2czhJ[YQáä\TWXbóx`TW@T>[xYT>[ÙWwzT>kacyaQXZdTwR[YVY\ fwy[`xYhfËwRk[YT>UvT>kbkZaQXZdP°khJcyVY\ZwzhJ[ï wzJ]\~@T\hV`kZTaUvhJ%~wR[aQ\ZwzhJ[ïhdikZTWdTWXbadcG\ZwRTfwy[`xYhfkWP&ÇVY\ \Z`wykwRkhJ[`czaQcRwRUWaQ~cRT&fYT>[¯\ZYTTvadUv\qcRhJ[YQáä\TWXb,xYTWT>[`xYT>[ÙWwRT>khd\Z`T\ZadkZj¯aQXZTj[`hfe[ %fwRUb²wRk V`kbVàdcRcz[Yhd\´\ZYTUWadkTdP_lãJn´®Gw T>x%\ZwRTifwR[xYhfkáQXZTwR[àdx`T>]VàQ\TifYT>[a\Zadkj%adkÚoàd[YJwR[`czhJ[`Qáä\TWXb xYTWT>[`x`T>[ÙWwzT>kWP [)S^^ëaQ`XZhJadUbP %hJ[)\ZYThd\Z`TWXàd[èx %UWad[ça>dhJwRxç\Z`T>kT`XZhd~cRT> &k %_~T>UWadV`kTS^^kxYhwR[ `XowR[ÙWwzcRT [Yhd\[YTWT>x adUWUvT>kZk\hï\ZYTµadk\WPº³YTWºUWad[åhd\T>[\ZwRadcyczºczT>aQXb[å\hïTv\XbadUv\ ad[`xåXZTW`XZT>kZT>[\a ¶¯aQXZjdhkZ\ZaQ\TdP

T UWVXVXB

õªmÐº§M=/=I¸]?o¿CA¨üEÔ§=D ö@YFYÔï¿2Ky?¨À/HJCM=@?B¸éE?ºF´=K©?ÔS ÂÃ[ÑcR ad[å[YTW\gfihdXZjÑ\ZYT2UvhJ[]\T>[]\hd_\ZYT `wyx`xYT>[V`[`wz\ZkGwRkÄUvhdwRT>xwR[\hkhqUWadcyczT>xUvhJ[\Tv\V[`wz\Zk&%Qf`wyUbLTWT>x~adUZjwR[]\h\ZYT_`wRx`x`T>[%cRadTWXlmcR ad[e%

¡

×]êëìÚÄÛ_í»î>ïðòÙÚ]íqØqñÃò×Ú$ðWØÙ

>O ¤d¤QÊnoPlm³`wRk\hd@hJcRhdd wRkT>V`wzQadczT>[]\\ha[YTW\gfihdXZj fwz\Z a`wRx`x`T>[ cRa>dTWX(%fYTWXZTT>adUoV`[`wz\}©TWT>x`kwR[]\h TWdTWXZ2hd\ZYTWXhJ[YTwya\ZwRTqxYT>cRadT>xUvhJ[`[YT>Uv\ZwRhJ[`kfwz\Zx`T>cRa>2hJ[YTdP ncR ad[[YTW\Zk_aQXbTq\XbadwR[YT>xµ~2~adUZj]á `XbhdaQJaQ\ZwzhJ[)lmSV`T>cyàQXZ\&%!Äwy[\hJ[e%* -wycRcRwRad k%iO>¤d¡d{Jn§i\Z]Vk&\ZYTWÑxYh²[`hd\&TWdT>[ `XZhdMaQJaQ\T2TWXbXZhdXbk ~adUbj\ZYhJVYJ \ZwRTdPGÂò[ adcz\TWXb[àQ\ZwzdTaQ``XZhJadUoYT>kfwz\Z UvhJ[]\Tv\V`[`wR\ZkÄ\ZYT`wyx`xYT>[V`[`wR\ZkÄ©TWT>x²lLTdP£Y'P %LV`cycz UvhJ[`[`T>Uv\T>xMnÄwR[]\hq\ZYTUvhJ[\Tv\iV[`wz\Zk_lL\ZYT>wzXi[]V&~TWX a~@T_xßw @TWXbT>[\LXZhJ/\Z`T[]V&~TWXhd\ZYTwRx`xYT>[ V`[wz\ZkonoP kbVàdcRcz2Çi|³_³çhdXSG³_S±"lmad[`x\ZYT>wzX_\XbV[ÙWaQ\T>xdTWXbkZwRhJ[`koniaQXZT&VkT>xµ©hdX\XbadwR[`wy[YYP ¦KmÐº¸D´¸ª©[§ Z=I¸FIA¨§ªM=I¸]?o¿CA¨üE]Ô \Z_¦ ^`YFY¯bÔ aDS ³_wR TvágâT>cya>^T>VYXoadcJ^TW\úfhdXbjklm³_â^^konÄ@l ÄVa E[YTWX * adwz~T>Ac %EO>¤d¤dsJniadcRczhBfÕadUWUvT>kZk\hadk\TWdT>[]\Zk_wRaUWadkZUWadxYT>xwR[]\TWXb[àdc@xYT>cRaÝcRwy[YT>kWPÄ³_YTwR[]\TWXZädc@\ZYTW UWad[)adUWUvT>kZkx`TW@T>[x`k hJ[)\Z`T²[YTW\úfhdXbj"\hdhJczhdddP°³]V`kÝ\ZYTW-kZ1V TWX2LXZhJ(\ZYTkZadTï`Xbhd~czT> k2adk LTWT>xáäLhdXZfiaQXox[YTW\gfihdXZjkV`kbwR[Ya\ZwRTqfwR[`xYhBfqP Y¯CM=´ªmKL=¸§À_§FI?CA¨¸¹A¨¸]Ô>ÔBKyê@¸%ÐÑCED¸ªLÔi¿ÝK©?BÀµ¸éeCM¹¸=IC@FIÔKL=ÓGFe?cÔ \dY²Ï2Aö e4a}=I¸]? ¿CMABüEEÔ S ^8S f [YTW\úfhdXbjklm±wRe[ %lÄhdXb[Y>T %³>w [Yg k %Ò>¤d¤d{Jn %adcRcRhfLhdX}kTWdTWXbadc`xwRk\ZwR[Ùv\GwR[YMVY\I\ZwR TváäfwR[`xYhBfklLhJkZkZw á ~cRhdekbwzæWThJ[`TBnÄfwR\Z x`ßw TWXZT>[]\}\T>hdXbadcM5h EkTW\ZkWP}³_`TW UWad[ hd\T>[]\ZwRadcRcz khJcRdT\Zadkjkifwz\ZÝk\ZaQ\ZwzhJ[àQXZ czhJ[` \ZwRTcRaQJ&k §Ewz\XZT> adwR[kaÝ`XZhd~czT> \hxYTW\TWXb wR[`T&\ZYTXbwzJ]\fwy[`xYhfkWiP ÄhfiTWdTWX %efYT>[\ZYTczhJ[Y \TWXb xYTWT>[`xYT>[ÙWwzT>kaQXZT%[YhJ[ágk\ZaQ\ZwzhJ[aQXZµ\ZYTqaQ``XZhJadUomadwRcRkWP »CEHFÔ>¸]Dïî´§`H¨ü@ðbÓA¨CÓ´§`¹§`?BKLCM3= S ³hxYT>adc]fwz\Z%czhJ[Y\ZwRT}cRaQJk %d¶hdæWTWXlÃO>¤d¡d¤JnV`kT>k\ZwRTiUvhJ[`k\Zad[]\Zk fwRUb%wR[UvT}adUv\ZwzQaQ\ZwzhJ[Ubàd[YdT>k>]P ÄhfiTWdTWX %JLhdXÄczhJ[Y\ZwRT}JaQMk\Z`T\ZwR TUvhJ[`k\Zad[]\ZkÄ[YTWT>x%Tv\TWXo[àdc [`T \ZV`[`wR[`Ñlã¶hdæWTW( X %_O>¤d¤dsJnoP YV`[TW\adcã'P k%adcR\TWXb[àQ\ZwzdTµaQ``XZhJadUo-lÃO>¤d¤dJnV`xaQ\T>kq\ZYT2adUv\ZwzäQ\ZwRhJ[Ñhd aåXZT>UWVYXbXZT>[\V`[wz\ ~"adx`xwR[Yº\Z`T²hJcyx-adUv\ZwRäQ\ZwzhJ[ ad[`x-\ZYTÁlmkZUWadczT>xMnÝUWVYXZXbT>[\[YTW\wR[YMVY\WP/³_YT[YTW\ wR[`VY&\ %YhBfTWdTW(X %E\T>[`x`k\hÝTWXZ\ZVYXb~\ZYTk\hdXZT>x²wR[YLhdXb aQ\ZwzhJP[ %f`wyUbaQJadwy[² aQjdT>kczhJ[Y \TWXo k\hdXoaQdT wR `XbadUv\ZwRUWaQ~cRTdP ½CM=@?BKm=EF§>ª h%i¯KL¸AB§ABHQÀKLHJ§M ª hÄ=HJA¨¸ÐÑ¸=@?B§ªÄ«i¸]§ÀQ=´KL=¹§=j D ^¸]êM¸ªLCÓGÐº¸=@k? \ _½ i¤d¤¨¢]nÄ`XZhdhJkT>x\ZYTq ÄÂò±â)TW\ZYhx©hdX~XbwRxYJwR[`czhJ[Y\ZwyTcRaQJkWPG-`T>[YTWdTWXaV`[wz\´wy[`wRkG[YTW\gfihdXZj XZT>UvT>wRdT>kÚvhJC[ wRUv\ZwR[YTWXZXZhdXGkZwzJ[àdcy&k %dYTadx`xka`wzJ`TWXIhdXbxYTWXGV`[`wR\wR[ÙWwR[Y_aQ`XZhd`XbwyaQ\TUvhJ[`[YT>Uv\ZwRhJ[`kWP cz\ZYhJVYJwRkGaQ``XZhJadUo UWad[ khJ TW\ZwRT>kÄ~TTv\XZT>T>czLadkZ&\ %\hq~`XowRxYdTa\ZwRT_cRaQqwy[dhJczwR[YOWÊdÊqkZ\TWk a XZT>V`wzXbT\ZYTadx`x`wR\ZwzhJ[ hdiOWÊdÊV`[`wz\Zk>PG³YT[`TW\gfihdXZj2UWad[[Yhd\dT>[YTWXbadcRwzæWT\hkT>VYT>[ÙvT>kfwR\Z2V[`kTWT>[ cRaQ xVYXbaQ\ZwzhJ[`k>P ½ÀEF=ü¸A"lÔ ZeÔW?¸ÐºEÔ S ]V`[`jdTWXµkZk\T> k läUb` wRx`VY~TWX %qO>¤d¤ds¨P~ §¶hdæWTWX %O>¤d¤dsJnµxYhàdT\ZYT aQ~wycRwz\gÝ\h ~`XowRxYdTqaQXZ~wR\XbaQXZ2\ZwRT&cRaQJk %`~VY\_hJ[cz2wR\ZYTwR[YV`\kZT>]VYT>[UvT&TvY`wz~wz\ZkcRhUWadcRcR `XZT>xwRUv\ZaQ~czT XZTWJVcRaQXbwz\ZwzT>k>P «¬e¦%m S ±G³¶ÍxYhT>k[Yhd\kb1V @TWXLXZhJÅ\Z`TÝ`Xbhd~czT> k%aQ~@hBdTdPïÂú\kTWT> k%\hï~@TÝ\ZYTµk\ZaQ\Tµhd_\ZYT aQXZ\TW\ZYhx LhdX&XZT>UWV`XZXZT>[]\%[YTW\úfhdXZjk%LadUvT>xÑfwz\ZXbT>adcRwRk\ZwR5U %GczhJ[Y\ZwRT cRaQJk%~TW\gfiTWT>[ºhUWUWVYXZXbT>[ÙvT>k&hd XZT>cRTWäd[]\TWdT>[\Zk>P

T : UWVXVXB@n"G 6Bb=3Bpo`$Eq3G r;!#1vuW33 6'*6q#+B ^KLÔ>HJA¨¸]?¸ºÔwIZ Ðïî}CMªmKLH ¹A¨§ÐåÐº§À¯ª©¸§ÀQ=Km=¹Á§ªL¹CMA¨K©?BÀ´ÐÑÔx\ ¬÷µ«iÏ2ÔbayS ù±k lm±ITWT }O>¤d¤d{ ÄYaQjäQj]á >%

d§

wz~MaQXba1%eO>¤d¤Jdn_ a>µLadkZ\TWXczT>aQXb[dXbad aQ\ZwRUWadck\XbVÙv\ZV`XZThdÄx`wRkZUvXZTW\T>%M[YhJwRkTváäLXZTWTTWdT>[]\kT>VYT>[ÙvT>k&%M~VY\ UWad[`[`hd\xYT>adcfiT>cRcEfwz\Z[`hJwRkThdX_fwR\ZkT>]V`T>[ÙvT>k_hd´XZT>adc áäQadcRVYT>xwy[YVY\Zkläk~hdXb[`nT * ÇXowRkZUvh>T %IO>¤d¤JdnoP i²KLDD´¸=Èà§À¨üCêÈÐºCED´¸ª©zÔ \di²ààbÔ ayS Ä¶²¶k¯aQXZT fwyxYT>czV`kZT>xÕaQ``XZhJadUoYT>k\h-kT>VYT>[ÙvT `XbhUvT>kZkbwR[YYP ³_YTW)aQXZT fiT>cRc ágkZV`wR\T>x"LhdX[YhJwyk"wR[`VY\Zkµad[`xÕaQXZT wy[QaQXbwRad[]\\hÁ[YhJ[ágcRwy[YT>aQX\T>hdXbadc k\XbTW\ZUb`wR[`YP_³_`wRk_ aQjdT>k Ä¶²¶kT>kT>UWwRadcRcR2kZVUWUvT>kZkmV`cIwR[kTWT>Ub¯XZT>UvhdJ[`wz\ZwzhJ[ lL\ZYTWxYh2[Yhd\UWaQXbT%LhdX \ZYT¯x`ßw TWXZT>[ÙvT~TW\gfiTWT>["kbczhf8ad[`xmadk\ dTWXbkZwRhJ[`k hdaJwzdT>["khdjdT>[ÁfihdXbxMnoP)ÇVY\ LhdXÝ ad[] hd\ZYTWX \Zadkj-k Ä¶²¶kaQXbTÝczT>kZkkZV`wR\T>ex %G~@T>UWadVk>T %ÄV`[`cRwzjdTÝS^^&k %G\ZYTWÑaQXZT2cywR wz\T>x\hïx`wRkbUvXZTW\T2kZ\ZaQ\T2kMadUvT>kWP ³_wRk} aQjdT>k\Z`T>wzXiaQcRwRUWaQ\ZwzhJ[Ý\h ad[] \ZwRTkTWXbwzT>ki\Zadkj2UWV`%~@TWXokhJTad[x2wy[Y&T UWwzT>[]\WPÄ®YhdXTvad czT LhdX%kZwyczT>kqUvhJV[\ZwR[`\Zadkj&k %!Ä¶²¶k[YTWT>xÑadk% ad[ k\ZaQ\T>kadk%\ZYT \Z`TÝ[V`&~TWX%hd_k%~hJcRkhJ[Ñ\ZYT czhJ[`dT>k\kT>VYT>[ÙvT%\ZàQ\kZYhJV`cRx~@TUvhJV`[]\T>xEP_)YTWXZT>adkfwz\Z¯S^^k\ZYT[YT>UvT>kZkZaQXZadcRdhdXbwz\Z` UWad[¯~T

|{ ËØ

î>ï dï

Ú ÃÙÛ

ò !ÿPð

¤

Rw [k\Zad[\ZwyaQ\T>x¯fwz\Zï[YTW\gfihdXZjkhdsBáúrµV[`wz\Zkl~}qadcRwR[`jdT* ±IT>` ad[`[P%GO>¤d¤d¡d§GS_hxYXbwRJVYTWæ*Æ)wRczT>k%O>¤d¤d¡d§ uTWXok"*ËUb wRx`VY~TWX%sQÊdÊdÊQTBnoP³]V`k%]wR[XbwR[ÙWwzMczTS^^kiaQXZTaQ`cRwRUWaQ~MczT\h%\ZadkZjk~TWdhJ[`x \Z`TXZT>adUo hd]Ä¶²¶¯kWP =ÓGFe?ºC@Fe?BÓ´FI?åÀ´KLDD´¸=óà§À¨üCê ÐºCED´¸ªL7 Ô \Uv\ZVYXbTÝlmÇiT>[YJwzh * ®YXbadkZUvhJ[`=w %O>¤d¤drJnUvhJ%~wR[`T>k_T>czT>T>[]\Zk_hdÄ wz\ZVYXbTváähdyáäTvTWXZ\Z&k %`S^^&k %ad[`xwRx`xYT>[ ¶¯aQXZjdhhxYT>cRk %àd[`xwRkadxàQ`\T>xwya%\Z`T&¶ëadczdhdXbwz\ZP³IhhJV`X_j[`hfczT>x`d>T %Y\Z`wRkaQXbUo`wz\T>Uv\ZVYXZT%àdk [Yhd\dTW\_~TWT>[µaQ`cywzT>x \h\ZadkZjk_UvhJ aQXbaQ~czT\h\ZYThJ[`T>kx`wRkZUWV`kbkT>xµYTWXZTdPÇVY\wz\fiadkkZYhf[µ\h khJczdT kZwyczT\ZadkjkwR[]dhJczwR[YczhJ[Y\ZwR TqcRaQJkWP ÷µ¸=¸]?BKLH¾qABCM¹A¨§ÐåÐÑKm=¹ç§=ID°¾qABCM¹A¨§Ð ¬E¸§ÀBHQ3À S uT>[`TW\ZwRUµ|XZhddXbad wR[Y lmkTWTµTdP£Y'P %}âwRUZj]á ad[[`kITW\adcä'P %Ò>¤d¡J} §YiXoadTWX %Ò>¤d¡ddr §,}hdæ>1a %O>¤d¤dsJnGad[`x|}XZhd~aQ~wycRwRk\ZwRUGÂÃ[ÙvXZT> T>[\ZadcY|}XZhddXbad/dhJcyVY\ZwzhJ[ lm|}Âò|iniläEa cRV`kZ\hfwRUviæ *ÕYUb` wyx`]V`~@TW(X %O>¤d¤JdnwR[`XbwR[UWwzczT}UvhJVcRxkT>aQXbUo wR[dT>[YTWXbadcadczdhdXbwz\Z`/kZadUvT>k ~V`\aQXbT&kZczhBf x`VYT\h \Z`TqaQ~kT>[ÙvTqhdGdXbadxwzT>[\_wy[Y©hdXo aQ\ZwzhJ[µ`XZhBwyx`wR[Ya kT>aQXbUox`wRXZT>Uv\ZwzhJ[EP [x`kkhJcRVY\ZwRhJ[`kmadk\TWX ö2§=DC@Ð ¹MF¸]Ô>ÔBKL=¹ S ®YhdXµkhJTkZwyczT2~T>[Ùb` aQXZjk fT>wzJ]\ JVYT>kbkZwR[Yé \Zàd[ÁT>cRaQ~hdXbaQ\TdXbadx`wRT>[\ adcRdhdXbwz\Z` k¯@l ÄhUbYXbT>wz\TWX *Ub` wRx`VY~TWX %O>¤d¤dd{ %O>¤d¤ddr §&Uo` wRx]VY~TWX * ÄhUoYXZT>wz\TW( X %eO>¤d¤d{JnoP

bÿ ¦iA¨§`D´K©?BKLC@=I§ª«¬E¦mS àQ`\TWX sïxYT>kZUvXowz~T>k&\ZYT\Xoadx`wz\ZwzhJ[àdc±G³¶ adczdhdXowz\Z`ÌadkwR[]\XZhx`VÙvT>xå~] ÄhUoYXZT>wz\TWXad[`x¯Uo` wRx`VY~TWXlÃO>¤d¤JdnoP »CA¨¹¸]? ÷2§Y?B¸]EÔ S Âò[àQ\TWXfiTµwyxYT>[\ZwR©a²fT>aQj[YT>kZkhd±´³¶Íwy[ÑxYT>adcywR[Yfwz\ZºUvhJ[]\ZwR[Vàdc wR[`VY\_k\XZT>ad k\ZaQ\aQXbT[Yhd\ 2o9:0|?A750?´kTWJT>[]\T>xwR[]\hÝkTWaQXoaQ\T\XoadwR[`wR[YÝkT>]V`T>[ÙvT>&k %EkZVÙo\ZàQ\wz\wRk [Yhd\UWczT>aQXf`T>[\hXZT>kZTW\\Z`T[YTW\úfhdXb:j kqwy[\TWXb[adck\ZaQ\TdP&T wR[\Xbhx`VUvT¯ÞÃLhdXZdTW\JaQ\T>kZßadkaÝXZT>T>xY läuTWXo&k %@Uo` wRx`VY~TWX %1*°V` wR[`&k %YsQÊdÊd1Ê %O>¤d¤d¤¨~@noP Ï2AQK©?BÀÐÑ¸]?BKLH CÓ¸A¨§Y?BKLCM=EÔ S Âò[àQ`\TWX¢%fiTXZT>kT>[]\}\ZadkjkwR[dhJcRwR[`qaQXbwz\Z`TW\ZwyU_hdTWXbaQ\ZwzhJ[`k}hJ[ UvhJ[]\ZwR[Vàdcwy[YVY\_k\XbT>ad k\ZàQ\\Xoadx`wz\ZwzhJ[àdcI±´³¶ UWad[[Yhd\kZhJczdTdPÇVY\±´³¶Tv\T>[`x`T>xfwR\ZLhdXZdTW\ JaQ\T>kadkÄkZVYTWXbwzhdXGaQXbwz\Z` TW\ZwRUUWaQMaQ~wRcRwz\ZwRT>kad[`xxYhT>kÄkhJczdT\Z`Ti\ZadkZjk_läuTWXo_k *ÕUb wRx`VY~TWX %¨sQÊdÊdÊdUnoP ¦KmÐÑKm=¹ h¸éE?B¸=D´Km=¹«¬E¦ ¿ÝK©?B À dÓ¸¸Ó´ÀCMªL¸ HCM=´=I¸Hd?BKLCM=lÔ S ÂÃ[ÝàQ`\TWXirfTwR[dT>kZ\ZwzJaQ\T \ZadkjkfYTWXZT_\Z`T_\T>hdXbadcxwRk\Zad[ÙvT_~TW\úfTWT>[2TWdT>[]\ZkiUvhJ[]dTWkT>kbkT>[\ZwyadcwR[YLhdXb aQ\ZwRhJ[lL\Z`wRkwRkÄ\ZYTUWadkT LhdXq[]VTWXZhJV`kkT>VYT>[]\ZwRadcG\ZadkjkqkZVÙoadkqhd\hdX&UvhJ[]\XZhJcad[`x²Xb]\Z`ÈxYTW\T>Uv\ZwzhJ[MnoP®GwzXbkZ\fiT wRxYT>[]\ZwzL afiT>aQj[YT>kZkwy[±´³{ ¶ k_UvhJ[`[`T>Uv\ZwzhJ[kZUoYT>>T %`XbTWJaQXbx`wR[Y\Z`TqfwzXbwy[Y%hdG\ZYTq[`hJ[`cRwR[YT>aQ(X %V`cz\ZwRcRwRUWaQ\ZwzdT JaQ\T>kÄkZV`XZXZhJV`[`xwR[Yad[`x%`XZhd\T>Uv\Zwy[Y±´³{ ¶ kGUvhJ[`k\Zad[]\ÄTWXZXZhdXÄUWaQXZXZhJVkT>cRkläinoPBTTv\T>[`x±´³¶ ~] wR[]\XZhx`VÙWwR[`ÝÞÃTWTWYhJczT_UvhJ[[YT>Uv\ZwzhJ[`kZßLXZhJ/\Z`TikG\h&\ZYT_JaQ\T>k}ad[x [`x\ZàQ\}±G³¶ adVYJT>[]\T>x ~qTWTWYhJczTiUvhJ[`[YT>Uv\ZwzhJ[kÚWad[czT>aQXo[&`XbT>UWwRkTi\ZwR wR[YYPÂú\GczT>aQXb[`T>ex %Q©hdX´Tvad cz>T %d\ZY"T [YTx`wRk\Zwy[Ùv\ZwzhJ[~Tvá \úfTWT>[kT>VYT>[ÙvT>khdkwRjdT>kIkTWaQXoaQ\T>x%~T>wz\ZYTWXÄrQÊhdX¢¤x`wRkZUvXbTW\T}\ZwRTik\TW&k %QfwR\ZYhJVY\e\ZYTiYT>cz%hdad[ kZ`hdXZ\\XbadwR[`wy[Y%TvT>cyaQXbk&läuTWXbk *°Uo` wRx`VY~TWX %sQÊdÊdÊQ>T §EuTWXo&k %@Uo` wRx`VY~TWX %1*°Ub`XbadV`xYhJczMe %YnoP ÄhUoYXZT>wz\TWXïad[`x ½CM=@?¸]éE?çùgA¨¸]¸ §M=IDëHJCM=@?¸]éE?çÔ>¸=IÔK©?BK©ê@¸ ªL§=¹MF§`¹¸EÔ S |XZTWwzhJV`kfhdXbj-~ Uo` wRx]VY~TWXlÃO>¤d¤Jdnad[`xÁ\ZYThJVYX fihdXZjÕlmkTWT²aQ`\TWX JnkZYhBfT>x\ZàQ\ ±G³¶ hJV`\@TWXb©hdXb k\Xba¨á x`wR\ZwzhJ[àdcS^^khJ[åcRT>aQXb[`wR[YXZTWJV`cRaQXcRad[YJVàQdT>k%LXZhJTvT>McRaQXZ\XbadwR[wR[YkT>VYT>[ÙvT>k>P¯ÂÃ[ àQ`\TWX {fT2xYT>hJ[`k\XoaQ\TÝ±Ý³N ¶ k%kZVYTWXbwzhdX&@TWXb©hdXb ad[ÙvT hJ[ºUvhJ[]\Tv\LXZTWT2cRad[YJVàQdTïlä®G±ń~T>[Ùo` aQXZjk LhdXXZT>UWVYXZXbT>[\[YT>VYXoadc[YTW\úfhdXbjk lmS^^konoP³Ih\Z`T~@T>kZ\hdihJVYXj[`hfczT>x`d>T %I±´³¶ äQXowRad[\ZkaQXbT adcRkh \ZYR T `Xbk\ÝS^^kÝ\håcRT>aQXb[aåkZwRMczTUvhJ[\Tv \ a&,&B,ÝcRad[YJVàQdT lä_±ń %_[adT>cz 6,[ läuTWXbÃk * Uo` wRx]VY~TWX %sQÊdÊY5O %noP ¦KmÐº¸Ô¸]AQKL¸]ÔÓÁB¸DGK©HJ?BKLCM3= S ÂÃ[àQ`\TWX ¯±´³¶Íwyk%aQ`McRwzT>x \hï\ZwR T2kTWXowzT>k&XZT>x`wRUv\ZwzhJ[Ñ\Zadkjk khJcRäQ~cRT²~)\ZwRTfwy[`xYhfÉaQ`XZhJadUb`T>kW\ZYTº¶¯adUZjdTW]áòucRadkbk²kZTWXbwzT>kad[`xÕ\ZYTåad[]\ZaÁ®YTå®ÂÃScRadkTWX T> wykZkZwzhJ[µkTWXbwRT>k&läTW\nläuTWXbk %}Ub:j %:* Ub` wRx`VY~TWX %sQÊdÊd1Ê %sQÊdÊYOnoP ÅÆ

op

OWÊ

×]êëìÚÄÛ_í»î>ïðòÙÚ]íqØqñÃò×Ú$ðWØÙ

¨

©«ªX¬®~¯Z°

³

´

Ä6 4(5 ¸ ¹¸

³_`T ~adkZwRU V`[`wz\&wy[\ZYTÝwRx`xYT>[ cya>dTWXhd_ad[Ñ±G³¶([YTW\gfihdXZj wRk&\ZYTg8,&87>0¿{/4'7K&`:§}wz\&XZTWcRadUvT>k%\ZYT `wyx`xYT>[qV`[wz\ZkIwR[%aÝÞÃ\Xoadx`wz\ZwzhJ[àdcyß_S^^ lm®GwzJVYXZTsPROnoPÁT>hdXb~czhUZjqUvhJ[]\ZadwR[`k´hJ[YT}hdXÄhdXbT8,/8~750|¿ K|,/4c4Ðaad[`xaadwzXhdadxaQ`\Zwzd> T %`V`cz\ZwzcywRUWaQ\ZwzdTJaQ\Zwy[Y V`[`wz\Zkf`wyUb2JaQ\T&wR[YMVY\ad[`x2hJVY\MVY\\h adcRcUvT>cRcRk wR[\ZYT~cRhUbjMPÄ¶T> hdXZ%~MczhUbjkÄadcyczhfÁUvT>cRcyk\h%kZàQXZT\ZYTkZadTJaQ\T>klL`XZhBwRx`T>x%\Z`T\Zadkj%TWXb wz\Zk´\Z`wRkon % \ZV`k´XZT>x`VÙWwy[Y\ZYT[V`%~@TWX´hdMadxaQ`\ZwzdTaQXbadTW\TWXokWPádUb T>hdXZ&UvT>cRcàdkGaQ\wz\ZkGUvhdXZTaXZT>UWVYXbXZT>[\ZcR kT>cRyágUvhJ[`[`T>Uv\T>xÝcRwy[YT>aQX}V`[`wR\UWadcRczT>x \Z`TÝÞbihJ[`kZ\Zad[\iXZXZhdXaQXZhJV`kZT>cRß lä_n %f`hJkTadUv\ZwzQaQ\ZwzhJ[ÝfTUWadcRc \ZYT2UvT>cRic a|;=2>;,WP²³_`Tµ k&khJcRdTÝ\ZYT2Qad[`wRkZ`wy[YTWXZXZhdX`XZhd~cRT>wy[ \ZYTµaQ~kT>[ÙvT2hd[YTWf wR[YMVY\hdX Out

Output

Hidden

In

Input

Out

Memory Block with one Cell

Output Gate

InputGate

In

®GwzJVYXbT²sPROQÑ±ITW©\WåS^^Èfwz\Z"hJ[YT²LVcRcz XZT>UWVYXZXbT>[\µ`wRx`x`T>[cRadTWX>P/SwzJ]\Wº±Ý³¶ [YTW\gfihdXZj"fwz\Z T> hdXZÝ~MczhUbjkwR[µ\ZYTq`wRxxYT>[µcRa>dTWXlLhJ[`cz2hJ[YT&wRkkZ`hf[MnoP OdO

×]êëìÚÄÛ_í/1ïàÚ]íiëñðWÚ$ð>ØÙëÿ[ÿ´ÜeÚ#

O>s

y output gating

output squashing

hy h( s )

c out

wout

net out

win

net in

ouput gate

c

s=s+gy

in

1.0

c

c

y

out

CEC: memorizing

gy

input gating

input squashing

in

y

in

input gate

g(net ) c

wc net c

®GwzJVYXbT sPts%³_YT\Xoadx`wz\ZwzhJ[àdcÄ±G³¶UvT>cRcàdk%acRwR[YT>aQXqV`[wz\fwz\Z aXbT>UWVYXZXZT>[]\qkT>cz©ágUvhJ[`[YT>Uv\ZwzhJ[ fwz\Z fiT>wzJ]\OQP Ê¯läi_noPYÂÃ[YV`\iad[`x2hJVY\MVY\JaQ\T&XZTWJV`cyaQ\TXbT>adxad[`xµf_XbwR\TadUWUvT>kZk\h\ZYTqUvT>cRc@fYhJkZTqk\ZaQ\T wRkxYT>[Yhd\T>x D P³_`TmV`[Ùv\ZwzhJ[ kZVàdkZ`T>k}\ZYTUvT>cRAc kiwR[YV`&\ §[µkZVàdkZYT>k}\ZYTUvT>cyAc khJVY\MVY\lmkTWT\Tv\LhdX xYTW\ZadwycRkonoP TWXZXbhdXkZwzJ[adcRk\hÝ\Z`T&UvT>cRcA%`\Z`T k_czhUWadceTWXZXbhdX~adUbjw`hfàXZT> adwR[k_UvhJ[`k\Zad[]\&%@[`T>wz\ZYTWX_dXZhBfwR[Y2[YhdX xYT>UWawy[YYP²³YTµ wyk&`Xbhd\T>Uv\T>x ©XZhJë~hd\Zå©hdXZfaQXb[ x `hfwR[Y¯adUv\ZwRäQ\ZwzhJ[ ad[`xÑ~adUbj]faQXbx `hBfwR[Y TWXZXbhdX~]å\ZYTwR[`VY\ad[`xhJVY\V`\JaQ\T>kÝXZT>kT>Uv\ZwzdT>cRdP-YT>[ÁJaQ\T>kÝaQXZTUWcRhJkT>x lmadUv\ZwzQaQ\ZwzhJ[aQXbhJV`[`x æWTWXZh]n %EwzXbXZT>czTWQad[\wR[YVY\Zkad[`x¯[`hJwRkT%x`hÝ[Yhd\T>[\TWX\ZYTUvT>cRAc %ad[`x¯\ZYT%UvT>cRc´k\ZaQ\TxYhT>k[`hd\TWXZ\ZVYXZ~¯\ZYT XZT> adwR[`xYTWXhdG\ZYTq[YTW\úfhdXZj@P®GwzJV`XZT&sPtskZYhBfk_a T>hdXZ2~czhUZj2fwz\Za kZwy[YJczTUvT>cRcãP ønÆÅ

p¥ e

O

:

_³ `TUvT>cRc@k\ZaQ\T>%y%wRkiVYExàQ\T>x ~MadkT>x2hJ[Ýwz\ZkUWVYXbXZT>[\ik\ZaQ\Tad[`xÝ\ZYXZTWTkhJVYXbUvT>kihdIwR[`VY\WGØGÙÔ5wRkiwR[YVY\ \h \Z`TqUvT>cRcewz\ZkT>cRIf`wyczTØGÙÔ> ad[`xwØGÙ&Ô 5¡l¢aQXZT&wy[YVY\Zki\h \ZYT%wR[YVY\ad[`xhJV`\VY\JaQ\T>kWP T UvhJ[kZwRxYTWXx`wRkbUvXZTW\T\ZwRTk\TWmk ÔiÕ5O Öbds Ö&×&×&×BP% kZwR[YJczTk\TW wR[]dhJczdT>k\ZYTVYExàQ\ThdadcRcGV`[`wz\Zk lLLhdXZfaQXbx2adkZkonad[`xÝ\ZYTUvhJVY\ZaQ\ZwRhJ[µhdITWXZXZhdXkZwRJ[àdcRk©hdXadcRc@fiT>wzJ\ZkqlL~MadUZjfiaQXbxµMadkZkonoPÄÂÃ[YVY\JaQ\T adUv\ZwzQaQ\ZwzhJw[ Ó Ýad[`xhJVY\VY\JaQ\TadUv\ZwzäQ\ZwRhJr[ Ó >¡D¢ aQXZTqUvhJVY\T>x¯adkLhJcRczhfkW Ï Ú ØGÙÔ 5¡l¢|£ cl Ôn Õ Í 5¡l¢|£ Ï Ó cl Ô5Þ O n §Ó 5¡l¢|£ cl ÔZ!n Õ¹à 5¡l¢|£ cl ØGÙÔ 5¡l¢|£ cl Ônþ n Ö lãsPROn Ï ØGÙ&Ô5 £cl Ôn lãsPtsJn Õ Ú Í £ Ï Ó Ï cl Ô5Þ On §Ó £ cl Ôn ÕàJ £Jcl ØGÙÔ> £cl ÔZn n × Ï

¤

IÛ Ù ÃÙ¥

O>

1ï 1ïüÿ _ëí ð

³_`XZhJVYJYhJVY\\ZwRk\ZYT>kZwykM¦²wR[`xYTvT>k& T>hdXZï~czhUZjk&§ §wR[`xYTvT>k&T> hdXZUvT>cRcRk&wy[~czhUZjk¦ lLfwz\Zt¨.© UvT>cRcykon %´kbVÙbÑ\ZàQ\ ©ª xYT>[`hd\T>k&\ZY@T §Jáä\Z UvT>cRc}hd_\ZYT ¦Qáä\Z T>hdXZº~czhUZ j §!ÍfÎÜÏówRkq\Z`TÝfiT>wzJ]\%hJ[Ñ\ZYT UvhJ[`[`T>Uv\ZwzhJ[ ©XbhJ V`[`wz-\ Ñ \h V`[`wR-\ ÒòP Âò[xYTv[ Ñ XbÏ ad[`dT>khdTWX2adcRckhJVYXbUvTV`[wz\Z&k %}adk kZ@T>UW¤w `T>xå~]å\ZYT [YTW\úfhdXbjÏ \hd@hJcRhddºlmwzÄa khJVYXoUvT&V`[`wz\_adUv\ZwRäQ\ZwzhJR [ Ó cl Ô>Þ OnXZTW©TWXok_\hÝad[²wR[YMVY\V`[`wz&\ %UWV`XZXZT>[]\_Tv\TWXo[àdc wR[`VY"\ Ó cl ÔZnwRkV`kT>xwR[`kZ\T>adxMnoP}®YhdX\ZYTJaQ\T>&k % àïwyka czhdJwRk\ZwyUkZwzJhJwRx lLfwz\ZµXbad[YdAT « 1Ê Ö>O ¬Lno lãsPtJn àÄl E n Õ O3®åO Tb¯G° × ³_`Twy[YVY\\h \ZYT%UvT>cRcEwz\ZkT>cz´wRk ØGÙÔ £ ± cl Ô! n ÕÚ Í £± Ï Ó Ï cl ÔVÞ Oá n Ö lãsP£¢]n Ï

fwRUbwRk}wRkkZVàdkZYT>x ~]W%aqUvT>[]\TWXZT>xÝcRhdJwRk\ZwRU_kbwzJhJwRxmV`[Ùv\ZwzhJ[ fwz\ZXbad[YdT«ßÞsdÖbsy¬GlmwR@[Yhd\ikT>UWw'`T>x x`'w @TWXZT>[]\Zcz`no @l n Õ OC®å¢ T ¯G° ÞºË s × lãsPtrJn ³_`TwR[]\TWXb[àdc@k\ZaQ\ThdIT> hdXZ UvT>cRIc D>cl ÔZn}wRkUWadcyUWV`cRaQ\T>xµ~ adx`x`wR[`q\ZYTkZVàdkZYT>ex %]JaQ\T>xwR[YMVY\\h\ZYT k\ZaQ\T%aQ\_\ZYT&cRadkZ\\ZwRTqk\TWW cl Ô5Þ Ono £± lmÊn Õ)wÊ § £± cl ÔZn Õ £± cl Ô5Þ On ®[Ó £ cl Ôn @cl ØGÙ&Ô £± cl ÔZnn LhdfX Ô³²ÁwÊ × lãsPt{Jn ³_`TµUvT>cychJVY\VY\ Ó wRkUWadcRUWV`cRaQ\T>x~ kZ]VadkZ`wR[Y²\ZYTwR[]\TWXb[àdckZ\ZaQ\4 T wRa²\ZYT5¡lhJ¢ VY\V`\ kZ]VadkZ`wR[Y mV`[Ùv\ZwzhJk[ P%ad[`xµ\ZYT>[%V`cR\ZwzczwR[YµlLJaQ\ZwR[Y]nwR\~]2\ZYThJVY\V`\JaQ\T%adUv\ZwzQaQ\ZwzhJr[ Ó £± cl ÔZn Õ¥Ó 5¡l¢|£ cl ÔZ³n I´l £± cl Ôn n × Ó lãsP dn wRk_aUvT>[]\TWXZT>x¯kbwzJhJwRxµfwR\ZµXbad[Yd T «ßÞq5O Ö>O ¬ä Il En Õ OC® s Tb¯G° Þ®O × lãsPt¡Jn ®GwR[àdcycz> %MadkZkbV` wR[Y2aÝcya>dTWXZT>x[YTW\úfhdXbj\hdhJczhddfwz\Zaµk\Zad[`xaQXbx¯wy[YVY\cRa>dTWX %Eaµ`wRxxYT>[cRadTWXUvhJ[á kZwyk\ZwR[Y¯hdT>hdXb ~czhUZj&k %}ad[`x aïk\Zad[`xaQXbxåhJVY\VY\cRa>dTW(X %}\ZYTµT>VàQ\ZwzhJ[`kLhdX\ZYTÝhJV`\VY\V`[`wz\Zk å aQXZTd ØGÙ&Ô cl ÔZ n Õ Ú Í ä ÏÓ Ï cl ÔVÞ On ÖÓ ä cl ÔZn Õ¹à ä cl ØGÙÔ ä cl ÔZn n Ö lãsPt¤Jn ä Ï

f`TWXZTÑÌXbad[YdT>khBdTWXadcRcÄV[`wz\Zk©TWT>xwR[Yµ\ZYT hJVY\MVY\V`[`wz\Zk lL\ú]wyUWadcRcz¯adcycÄUvT>cRcRkwR[ï\ZYT wRx`xYT>[ïcRadTWX% \ZYTwR[`VY\V`[wz\Z&k %~VY\[Yhd\\ZYTT>hdXZ¯~czhUZjJaQ\T>k noPkkZVàdkZ`wR[` mV`[Ùv\ZwzhJ [ à ä fiTaQJadwy[V`kT\ZYT czhdJwyk\ZwRUkZwzJhJwyx lãsPtJnoPi³_wRkUvhJ[ÙWcRV`xYT>k_\Xbadx`wz\ZwzhJ[adc@±G³{ ¶ kLhdXZfiaQXoxadkZkWP

µ

ÿ TWT-ÄhUb`XZT>wz\TWX*ÆYUb` wyx`]V`~@TWX&lÃO>¤d¤JdnLhdXxYTW\ZadwRcRkhd\Xbadx`wz\ZwRhJ[àdce±Ý³¶N k~adUbj]faQXbx²adkZkWPÂú\fwRcRc ~TXZTvágxYTWXbwRdT>xad[`xx`wykZUWV`kZkT>xµwR[x`TW\ZadwRcewR[T>Uv\ZwRhJ[PtsPts aQL\TWX\ZYT%wR[\Xbhx`VUv\ZwzhJ[2hd´LhdXZdTW\_JaQ\T>k>P kZkT>[]\ZwRadcRcz> %adkwR[Ý\XbV`[ÙWaQ\T>xÇ|³_n³ %TWXZXZhdXbkaQXZXowzwy[Y%aQ\[YTW\wR[YMVY\Zk}hd´T>hdXZÝ~czhUZjkad[`x2\ZYT>wzX JaQ\T>kxYhq[Yhd\ÄdTW\`XbhdaQJaQ\T>x ~adUbj%mVYXZ\Z`TWXwR[\ZwR>T %adcR\ZYhJVYJ\ZYTWw l7_kZTWXZdT_\hqUbad[YdT_\ZYT_wR[UvhJ wR[Y ønÆJø

P

×]êëìÚÄÛ_í/1ïàÚ]íiëñðWÚ$ð>ØÙëÿ[ÿ´ÜeÚ#

Ov¢

fiT>wzJ]\ZkWP_ÂÃ[T>kZkZT>[ÙvT>%hJ[ÙvTad[¯TWXZXZhdXkZwzJ[adcIaQXZXbwRdT>kaQ\a T> hdXZUvT>cRchJVY\VY\&%@wz\dTW\ZkkZUWadcRT>x~\ZYT hJVY\MVY\JaQ\Tad[x\ZYT&hJV`\VY\[YhJ[cRwR[YT>aQXbwR\g@ P§\ZYT>[wz\_T>[]\TWXbk\Z`T&T>hdXZUvT>cR=c kcRwR[`T>aQXi %fYTWXZT wz\UWady[ `hBf~adUbj wR[`xY&T [wz\T>czqfwR\ZYhJVY\ÄTWdTWXi~@T>wy[YqUbàd[`dT>x²lL\Z`wykwRkf±´³¶ UWad[ ~`XbwRx`dT_aQXZ~wz\XbaQXb \ZwR T²cRaQJk2~TW\gfiTWT>[çwR[YMVY\ TWdT>[]\Zkad[`x-\ZaQXZdTW\kZwzJ[àdcykonoP°[`czf`T>[\ZYT²TWXZXZhdXµT>kZUWaQT>kµLXZhJ(\ZYT T> hdXZåUvT>cyc\ZYXZhJVYJad[hdT>[`wy[YwR[YVY\JaQ\T¯ad[x \ZYTadx`x`wz\ZwRhJ[àdcwR[YVY\[YhJ[`cywR[YT>aQXbwz\úp W%xYhT>kwz\ dTW\kZUWadczT>x¯hJ[ÙvThdXZT%ad[x\ZYT>[²kTWXZdT>k\hµUbàd[YdTwR[UvhJ wR[YfiT>wzJ]\Zk~TWLhdXZT&~T>wR[`\XbV`[ÙWaQ\T>xEP³_YT UvhJ[`kZT>]VYT>[UvTqhd\ZwRk\XbV`[ÙWaQ\ZwRhJ[wRk\ZàQ\_T>adUo±G³¶Å~czhUZj2XZT>cRwzT>khJ[TWXbXZhdXbkLXZhJó\ZYTqhJV`\VY\LhdXwz\Zk adxàQ\ZaQ\ZwzhJ[EP}wR[ÙvT~MczhUbjk}xYhq[Yhd\TvYUbàd[`dTTWXZXbhdX}kZwzJ[àdcy&k %wz\wRk}àQXbxLhdX±Ý³¶\h%czT>aQXb[\ZadkjkfYTWXZT hJ[YT~czhUZj TvUWcRVkZwzdT>cz kTWXbdT>khd\Z`TWX~czhUZjklLTdP£Y'P %adkahJwR[]\TWXiwR[]\ha®´ÂÃ®àVYT>VYTBnfwz\ZYhJVY\ix`wzXZT>Uv\ZcR XZT>xVÙWwR[Y\ZYTqhJVY\MVY\TWXZXZhdX>P ý É dP àÿ3þ ýRe ÿJÿ f µ ý7¶ R ÄhUoYXZT>wz\TWXad[x Ub` wRx`VY~TWXilÃO>¤d¤JdnÄadcRXZT>adxY%kZhJczdT>xafwRxYTiXbad[YdThd@\ZadkjkÄfwz\Z\Xbadx`wR\ZwzhJ[àdc±´³¶ lÃOn³_YTT>%~@T>xxYT>xÝSTW~TWXidXoad aQXqlma@hdMV`cRaQX}XbTWJV`cRaQXdXbad aQX~T>[Ùo` aQXZj`n §elãsJni^hJwRkZTLXZTWTad[`x [YhJwyk kT>VYT>[ÙvT>kfwR\ZÝ\ZwyTcRaQJkhdV`2\h¯OWÊdÊdÊ k\TWkqlLTdP£Y'P §`\Z`TÞZsBágkT>]V`T>[ÙvTq`XZhd~MczT> ß%`XZhdhJkT>x2~] ÇiT>[YJwzhTW\_adcã'P %eO>¤d¤¨¢]n §GlãJnihJ[]\ZwR[]V`hJV`kÃáäQadcRVYT>x2\Zadkjk\ZYTXZT>V`wzXZT\ZYTk\hdXbaQdTqhd´QadcRVYT>ki©hdX_cRhJ[Y\ZwRT TWXbwzhx`kad[`x \Z`T>wzXkZV` aQ\ZwRhJ[Ýad[`xµV`cz\ZwzMcRwRUWaQ\ZwzhJ[lmV`2\haUvTWXZ\ZadwR[µXZT>UWwRkZwzhJ[@n §lL¢]n³T>hdXbadc@hdXbxYTWX `Xbhd~czT> kifwz\ZµfwRcRx`cz kZTWaQXbaQ\T>xwy[YVY\ZkWP ÂÃ[\ZYT_LhJcRczhBfwR[YUoàQ`\TWXb&k %YhfiTWdTWX %fTfwRcRc`XZT>kT>[]\Ä\ZadkjklLaQXb\Zczx`TWXbwzdT>xLhdXb/\ZYT_\ZadkZjkcywRk\T>x aQ~hdTBnhJ[f`wyUbµ\Xbadx`wR\ZwzhJ[àdcE±Ý³¶ÈmadwRcRkad[`x@hJwy[\hJVY\wR\Zk`XZhd~cRT> kWP

5N

ønÆ

¨

©«ªX¬®~¯Z°

¸7

³

½ N

)7«´ ³

À

5 ¸ 5

· ¹

6"4

Ë6¹¸4(5

¸q·

ºp4 v¹7Æ6¼» ¾ Ä6¿ ³

3 4(5(6¹¸5 ¶

¸

Eÿ ÄhUoYXZT>wz\TWXÝad[`xUb wRx`VY~TWXµlÃO>¤d¤Jdn xYT>hJ[`kZ\XbaQ\T>x\ZàQ\ ±G³¶ UWad[ÁkhJczdT[V`TWXZhJVk\ZadkjkÝ[Yhd\ khJcRäQ~cRT~ÝXZTWwRhJV`kczT>aQXb[`wy[YadcRdhdXbwz\Z` kLhdX_S^^kWPÂÃ[µ\Z`wRkUbaQ`\TWX%YhBfTWdTWX(%`fTfwRcRckZYhBfç\ZàQ\ TWdT>[±G³¶,madwRcRki\h czT>aQXb[2\h UvhdXZXbT>Uv\Zcz `XbhUvT>kZkUvTWXb\ZadwR[2dTWXZ2cRhJ[Y%hdX_UvhJ[]\ZwR[Vàdc\ZwRTkTWXowzT>ki\ZàQ\_aQXZT [Yhd\~29:0?A750|?´kZTWJT>[\T>xÑwR[]\hµaQ`XZhd`XbwyaQ\T%\Xbadwy[`wR[Y2kZVY~kT>VYT>[ÙvT>kfwR\Z²UWcRT>aQXbczxY&T M[YT>x²~@TWJwy[`[`wR[YJk ad[`xT>[`x`k_aQ\f`wRUo\ZYT&[YTW\úfhdXb:j kwR[]\TWXb[àdcek\ZaQ\TUvhJV`cRx~@TXbT>kTW\WP³_YTq`Xbhd~czT>ówRk\ZaQ\a UvhJ[]\ZwR[Vàdc wR[`VY\kZ\XZT>adTWdT>[]\ZVàdcRcz2 a>UWadV`kZT\ZYTqwR[]\TWXb[àdcQadcRVYT>khd´\ZYTqUvT>cRcRk\hdXZhBfÕfwz\Z`hJVY\~@hJV[èx %TWdT>[ wz\ZYTXZTWTW\Zwz\ZwzdT_[àQ\ZV`XZThdM\ZYT`XZhd~czT>ËkZV`ddT>k\ZkÄ\ZYTWkZYhJV`cyx%~TXZT>kZTW\ÄhUWUWadkZwRhJ[àdcRczdP´Âò[ \Z`wRkGUbàQ\TWX fwycRcXZT>kT>[]\aXZT>T>xYdP )`wRcRT2fiT`XbT>kT>[\ aºkT>UW'w UµkhJcRVY\ZwRhJ[ \h \ZYT`XZhd~MczT> hdLhdXZdTW\\ZwR[`ºwR[±Ý³¶ [`TW\gfihdXZjk %ifiT XZT>UvhdJ[wzæWTÝ\ZaQÃ\ 25BW¿ \XbadwR[`wy[Y`XZhUvT>x`VYXbT ©hdXS^^kf`wRUoºwRk%hfiTWXZLVcT>[YhJVYJå\hïkad[åczhJ[Y¯\ZwRT cRaQJk%Vk\}adcRkhadx`xYXZT>kbkÄ\ZYTwykZkZVYThdLhdXZdTW\\ZwR[YwR[ kZYhdXb\\TWXbÆT>hdXZlmV`[`wR\adUv\ZwzäQ\ZwRhJ[`konoPÄTj[YhBf hdE[Yhqhd\Z`TWXUWVYXZXbT>[\\XoadwR[`wR[YTW\Z`hx ©hdX}S^^kf`wRUb wRkkZ[\ZcR%hfiTWXZmV`cY\h&a>dTT>[ÙvhJV`[]\TWXZT>x \Z`wyk`XZhd~czT>P ¶ k&fiT>aQj[`T>kZkwR[Ñ`XZhUvT>kZkZwR[Y²UvhJ[\Zwy[]VàdciwR[YMVY\%k\XbT>ad kWP Ò FI?BªmKm=I+¸ S T>Uv\ZwRhJ[ PROQPROµTvcRadwy[`k&±G³{ T>Uv\ZwRhJ[ PtsµwR[\Xbhx`VUvT>kaµXbT>T>xY¯UWadcyczT>x)ÞÃ©hdXZdTW\&JaQ\T>kWP ß®YhdXZdTW\%JaQ\T>kqczT>aQXb[ï\hXbT>kTW\qT>hdXZ²UvT>cRc UvhJ[]\T>[\ZkhJ[UvT\ZYTW aQXbT[Yhd\i[YTWT>xYT>x2ad[ÝhdXZTdP®YhdXZdTW\\Zwy[Y a> hUWUWVYXiXb\Z` wyUWadcRczhdXwy[ ad[µwR[YVY\Ãá xYTWT>[`x`T>[\madkZ`wzhJ[EPÂÃ[\ZYT kbadT kT>Uv\ZwzhJ[ºfTÝxYTWXbwzdT adXoadx`wzT>[]\Ãáä~adkT>x czT>aQXo[`wR[YadczdhdXbwR\Z`ÈLhdXq\ZYT ±G³¶ Tv\T>[kZwzhJ[ fwz\Z LhdXZdTW\iJaQ\T>kWPiT>Uv\ZwzhJ[2Pt&x`T>kZUvXbwz~T>kTvTWXbwRT>[]\ZkW´fT\Xbad[`kLhdXb°fT>cRczáäj[YhBf[ ~T>[Ùo` aQXZj`XZhd~MczT> kIwy[\hhdXZTUvhJczTvp %dUvhJ[\ZwR[Vàdc]\Zadkj&k %QXZTWhdXZ\I\ZYTiTWXZLhdXb ad[ÙvThdYäQXbwRhJV`kS^^ adczdhdXowz\Z` &k %ad[`xºad[àdcR]æWT2ad[`x UvhJaQXbT \ZYT [`TW\gfihdXZjk%LhJV`[`x~ï\Xbadx`wz\ZwzhJ[adc±´³¶ad[`xºTv\T>[xYT>x ±G³¶ºP ÆÅ

m e ~$

T > Á& v&$EB@02$E '(3&´$& "!#'Ác; Â@Ã

G± ³¶8adcRczhBfk}wR[`©hdXb aQ\ZwzhJ[\h~@Tk\hdXZT>x2adUvXZhJkZkaQXZ~wz\XoaQXZ%\ZwyTcRaQJk%ad[`x TWXZXZhdXikZwzJ[àdcRk}\h%~TUWaQXbXbwzT>x maQX~adUbj)wy[)\ZwRTdP³_`wykÝhd\T>[]\ZwRadcqk\XbT>[Yd\Ze%YhfiTWdTWX%&UWad[ UvhJ[\XbwR~VY\T\hafiT>aQj[YT>kZkwy[çkhJT O>r

O>{

×]êëìÚGÛ_íþúTïÿIÛëíÙiðÃÙÄ¥ Ú_ØvÅØíc¥ÛÚ³Æ×ØÙÚ]ðòÙò¦ëÿ{ìíÛñð>×_Ú$ð>ØÙ¸Rð>Ú]êáÿ´ÜeÚ#

kZwR\ZVàQ\ZwzhJ[`kWå\ZYT UvT>cRckZ\ZaQ\T>k4yÝhdL\T>[Õ\T>[x)\hÁdXZhfëcRwR[YT>aQXbcRx`VYXbwy[Yº\ZYTXZT>kT>[]\ZaQ\ZwzhJ[çhda \ZwRT kTWXowzT>k%lL\ZYT[YhJ[`cRwy[YT>aQX_adkT>Uv\ZkhdkZT>]VYT>[UvT&`XZhUvT>kZkZwy[Y aQXZTczTW©\\hÝ\ZYTkZVàdkZ`wR[`mV`[Ùv\ZwRhJ[`kad[`x\ZYT `wRJ`cz[YhJ[cRwR[YT>aQXJaQ\T>konoPµÂòfiT `XbT>kT>[\%aUvhJ[\Zwy[]VYhJVk&wR[YMVY\k\XZT>adw%\ZYT UvT>cRck\ZaQ\T>k% a>ïdXZhfÆwR[ V`[]~hJV`[`xYT>xÝmadkZ`wzhJe[ %YUWadV`kbwR[Y kZaQ\ZVYXoaQ\ZwzhJ[hd´\ZYTqhJVY\V`\_kZ]VadkZ`wR[YmV`[Ùv\ZwzhJe[ %GeP}³_`wRkaQ`T>[`kiTWdT>[ wz\ZYT [àQ\ZVYXbT hd\Z`T `XZhd~cRT>ÈkZVYddT>k\Zk&\ZàQ\q\ZYT UvT>cRc}k\ZaQ\T>k&kZ`hJV`cRxï~TXZT>kTW\qhUWUWadkZwzhJ[adcRcz> %eTdP£Y'P %GaQ\ \ZYT~TWJwR[[`wR[YJkhd[YTWf°wR[`VY\kT>VYT>[ÙvT>kÝlLfYhJkT k\ZaQXb\Z&k %YhBfTWdTW(X %GaQXbT [Yhd\qTvcRwRUWwR\ZczwR[`xwRUWaQ\T>xï~] a2\T>adUb`TWX noP2aQ\ZVYXoaQ\ZwzhJ[²fwRcRclman aQjdAT P kxYTWXbwzQaQ\ZwzdTQad[`wRkZP %@\ZV`k~cRhUbjwR[`2wR[UvhJ wR[Y2TWXZXZhdXbk %Iad[`x lL~@n aQjdT\ZYTUvT>cRc´hJVY\MVY\T>]Vàdc´\ZYThJVY\MVY\JaQ\TÝadUv\ZwzäQ\ZwRhJe[ %E\ZàQ\wR&k %\ZYTT>[]\ZwzXZTT>hdXZ²UvT>cRc´fwRcRc xYTWdT>[`TWXbaQ\T2wy[\hïad[ºhdXox`wR[àQXZÇ|}³³ V`[wz&\ %´kZh¯\ZàQ\\ZYT2UvT>cRc}fwRcRcUvT>adkTµmV`[Ùv\ZwzhJ[wR[YadkaT> hdXZdP ³_`T `XZhd~MczT>Åx`wyxº[Yhd\aQXbwRkT2wR[ \Z`TÝTvTWXbwRT>[]\ZkqXZTWhdXZ\T>xÑ~{ ÄhUb`XZT>wz\TW~X *8YUb` wyx`]V`~@TWX2lÃO>¤d¤Jdn ~T>UWadV`kTqUvT>cycek\ZaQ\T>k_fiTWXZTqTvcRwyUWwz\Zcz XZT>kZTW\_\h æWTWXZh ~TWLhdXZT\ZYT&kZ\ZaQXZ\_hd´T>adUb²[YTWf kT>VYT>[ÙvTdP Ähf UWad[ÑfT2khJczdT \ZwRk`XZhd~MczT>Èfwz\Z`hJVY\&czhJkZwy[Y±´³N ¶ k%adxYäd[]\ZaQdT>khdTWX\ZwRT2xYT>cRaï[`T>VYXbadc [YTW\úfhdXbjklm³â^^nlL adwz~T>Ac %O>¤d¡d¤JnhdX ^S f8[YTW\úfhdXbjklm±wR[ TW\ adcã'P %O>¤d¤d{Jn %f`wRUoxYTWT>[`xåhJÛ [ 2 9:0|?A750?j[YhfczT>xYdTqhd´\gwRUWadcE\ZwyTqcRaQ kZwRæWT> kÇ ³YT&k\Zad[`xaQXbx\T>Uo`[`wRVYTqhdÄfiT>wzJ]\xYT>UWa> %@f`wRUbYT>czk\hµUvhJ[]\ZadwR[\Z`T&czTWdT>chdÄhdTWXbadcRcádUv\Zwzwz\g fwR\Z`wR[q\Z`T[`TW\gfihdXZ:j %fiadkÄLhJV`[`x%\hdT>[YTWXbaQ\TkhJcRVY\ZwzhJ[kIfwRUbfiTWXZTaQXZ\ZwRUWV`cyaQXbcz`XZhJ[`T\hqV`[]~hJV`[`xYT>x k\ZaQ\T&dXZhf_\ZeP baQXowRad[\Zkhd_LhUWV`kT>xÑ~adUZj]áä`XZhdaQJaQ\ZwRhJ[Õlã¶hdæWTW X %O>¤d¡d¤JnadcRkhïxYhï[Yhd\%fihdXZj fiT>cRcãP²³_YT>kTµczTW\\ZYT wR[]\TWXb[àdck\ZaQ\TÑxYT>UWa>)wRa akT>cz©ágUvhJ[`[YT>Uv\ZwzhJ[çfYhJkTfiT>wzJ\wykkZ adcRczTWXµ\Zàd[/OQPóÇVY\\ZYTWXbT wRk[Yh `XowR[ÙWwzcRT>x&fa>hdexYT>kbwzJ[`wR[YqaQ`XZhd`XbwyaQ\T_xYT>UWa> UvhJ[k\Zad[\Zk>"hd\T>[]\ZwRadc`Jadwy[ LhdXkhJT\ZadkjkwRkadwRx LhdX~aqcRhJkZkÄhdEaQ~wRcRwR\g&\h%xYT>adcYfwz\Z aQXZ~wz\XbaQXb> %V`[Yj[Yhf[UWadV`kZadc`xYT>cya>k~TW\gfiTWT>[ wR[`VY\ZkGad[`x\ZaQXZdTW\Zk>P ÂÃ[madUv&\ %k\ZaQ\TxYT>UWa xYhT>k}[Yhd\}kbwzJ[`'w UWad[]\ZczwR`XbhdT_TvTWXbwRT>[]\ZadcYTWXZLhdXb ad[ÙvT&lmkTWTµÞb\ZaQ\TâT>UWaß wR[³áQ~czT&PtsJnoP UvhJV`XbkTfiTï wzJ]\ \XZ\hËÞÃ\T>adUbYTWXLhdXbUvT>ß)Al )dhdXbxàdP[ %qO>¤d¡dd{ §âhBJa *ÉÈhJkZ`wRæ>a>f1a %%O>¤d¡d¤Jn \ZYT wR[]\TWXb[àdcGk\ZaQ\T>_k ~XZT>kTW\\Zwy[Y\ZYT>,hJ[ÙvT a[YTWf/\XoadwR[`wR[YµkT>VYT>[ÙvT k\ZaQXb\ZkWPÇVY\\ZwRkXZT>]VwzXZT>kad[ Tv\TWXb[àdcG\T>adUoYTWXqfYhµj[YhfkYhf \hkZTWJT>[\\Z`T wR[YVY\k\XZT>adÈwR[]\h\XbadwR[wR[YÝkbVY~kT>VYT>[ÙvT>kWP%T aQXZT`XZT>UWwRkT>cRwR[]\TWXZT>k\T>Px %YhBfTWdTW(X %wy[\ZYhJkTkZwz\ZVàQ\ZwzhJ[kÄfYTWXZT\ZYTWXZTwRk[Yyh 2i9:0|?A750?j[YhBfczT>xYdThd\Z`wRk jwR[`xEP

N

Eÿ ie] ù "p V`XkhJcRVY\ZwzhJ[\h&\ZYT`XZhd~cRT>/aQ~@hBdTwRkÄ\h%V`kT_adxàQ`\ZwRdT ÞÃ©hdXbdTW\JaQ\T>kZß%f`wRUoczT>aQXb[\hqXZT>kZTW\}T>hdXZ ~cRhUbjkhJ[UvT \ZYT>wzXUvhJ[\T>[]\ZkaQXZT hJVY\%hdxàQ\TÝad[`xÑYT>[ÙvT VkT>czT>kZkWPµÇiïXZT>kTW\Zk%fT2xYh[Yhd\%hJ[`czïT>ad[ wR T>x`wRaQ\TXZT>kTW\Zk\h æWTWXZh ~MVY\_adcRkhdXbadxVàdcEXZT>kTW\ZkUvhdXZXZT>khJ[`x`wy[Y\h kZczhBfcz madx`wR[ÙvT>cycek\ZaQ\T>kWP ¶hdXZTkT>UW'w UWadcRcz> %dfiTXbTWcRadUvT_\Xbadxwz\ZwzhJ[àdc`±G³{ ¶ k}UvhJ[`kZ\Zad[\iåfiT>wzJ]\OQP Ê%~]\ZYT%V`cR\ZwzcRwRUWa¨á \ZwzdTLhdXZdTW\JaQ\T%adUv\ZwzQaQ\ZwzhJw[ Ó.Ê@PiTWT&®GwzJV`XZTqPROQP ÆJø

o w#

T Ë EÌ'I 6(z'IBbB`"02 $b+!3(#+(ÍÁ); Â@Ã Ì4&´$qË 1".$pÎp'+$EB

cRcT>VàQ\ZwzhJ[`kÄhd@\Xbadx`wR\ZwzhJ[àdcY±Ý³N ¶ k©hdXZfaQXbxMadkZkÄTvYUvTW`\LhdX}T>VàQ\ZwzhJ[lãsPt{JnÄfwRcRcXZT> adwy[ädcRwyxadcRkh LhdX_Tv\T>[`xYT>x±G³¶Åfwz\ZµLhdXZdTW\_JaQ\T>kWP ³YT©hdXZdTW\JaQ\TadUv\ZwzQaQ\ZwzhJw[ Ó.ÊÝwykUWadcRUWV`cRaQ\T>xcywzjdT\Z`TqadUv\ZwzäQ\ZwRhJ[`khd´\ZYThd\ZYTWX_JaQ\T>k59)kTWTqT>Và¨á \ZwzhJ[kqlãsPROnad[`xºlãsPtsJno ØGÙ&Ô £Jcl ÔZ lãPROn Ê n ÕÚ Ï Í Ê £ Ï Ó Ï cl ÔVÞ On §Ó Ê £ cl ÔZn Õ¹à Ê £Jcl ØGÙÔ Ê £cl ÔZn n × ÄTWXZ T ØGÙ&Ô Ê £wRk\ZYTwR[YMVY\_LXZhJ,\ZYT[YTW\gfihdXZj¯\h\ZYT©hdXZdTW\JaQ\TdPT V`kT\ZYTczhdJwRk\ZwyU%kZwRJhJwRx¯fwz\Z Xbad[`d T « 1Ê Ö>O ¬adkkZVàdkZ`wR[`ºmV`[Ùv\ZwzhJ¹ [ à Ê £ P Âú\ZkhJVY\VY\2~T>UvhJT>k\ZYTïfiT>wzJ\hd%\ZYT kT>cz&XZT>UWVYXZXZT>[]\

¤ )ÜeØ

úTï 1ï

Ú >ØÙÆKÅiØqí]¥ÛÚ/¥qëÚGÛiÜ

O

qÿiò $ð

y output gating

output squashing

hy h( s )

c

out

y

net out

wϕ

net ϕ

win

net in

ϕ

in

c

memorizing and forgetting

input squashing

wout

ouput gate

s=sy+gy input gating

out

c

ϕ

c

y

forget gate

gy g(net )

in

y

in

input gate

c

wc net c

®GwzJVYXbT PROQ¶T>hdXZï~czhUZj²fwz\Z hJ[`cz²hJ[YT UvT>cRcÄLhdXq\ZYT Tv\T>[`x`T>xº±Ý³¶ Pe V`cz\ZwzMcRwRUWaQ\ZwzdTLhdXZdTW\ JaQ\TUWad[XZT>kTW\_\Z`T&UvT>cRcA kwR[[YTWX_k\ZaQ\T P

UvhJ[`[`T>Uv\ZwzhJ[hdÄ\ZYT%wR[\TWXo[àdcIk\ZaQ\T wR[T>VàQ\ZwzhJ[ lãsPt{JnoP_³_YTqXZTWwRkT>x¯VYExàQ\TT>VàQ\ZwzhJ[¯©hdX) wy[\ZYT Tv\T>[`xYT>x±G³¶ëadczdhdXbwz\ZówRkqlL©hdXÔ²Êno £± cl ÔZ n Õ Ó Ê £ cl ÔZn £± lcÔ5Þ On®IÓ £ lcÔZnÏ@lcØGÙÔ £± lcÔZnn Ö lãPtsJn fwR\ZH £± lmÊn Õ ÊPÄ\T>[`xYT>x±Ý³N ¶ kmV`cRc©hdXbfiaQXbx²adkbkwRkhd~`\ZadwR[YT>xï~¯adx`xwR[Y2T>]VàQ\ZwRhJ[`klãPROn\h \ZYhJkZTqwR[àQ`\TWXsad[`xXZTWMcRadUWwR[YT>VàQ\ZwzhJ[ºlãsPt{Jn~] lãPtsJnoP ÇwRadkfiT>wzJ]\ZkLhdX_±´³¶ÉJaQ\T>kaQXZTqwy[`wz\ZwRadcRwRæWT>x fwR\Z2[`TWJaQ\ZwzdTqQadcRVYT>ki©hdXwR[YVY\iad[`xµhJVY\V`\JaQ\T>k lmkTWTT>Uv\ZwzhJ[²PtPtsJn %@hJkZwR\ZwzdTqQadcRVYT>kLhdXLhdXZdTW\JaQ\T>kWP_³_`wRkwRcRwRT>5k 9-UvhJaQXbTT>VàQ\ZwzhJ[k%lãPROnad[`x lãPtsJn 9"\ZaQ\wR[\Z`T~@TWJwy[`[`wR[YhdG\ZYTq\Xbadwy[`wR[YàdkZT\ZYTqLhdXZdTW\_JaQ\T%adUv\ZwRäQ\ZwzhJ[fwRcRc@~TqadcRhJkZ\&OQP 1Ê % ad[`x \ZYTT>[]\ZwzXZTUvT>cRc`fwycRc~T>à>dTcRwzjdTa&\Xbadx`wz\ZwRhJ[àdc`±Ý³¶8UvT>cRcãPGÂú\fwRcRcY[`hd\}TvcRwRUWwz\ZcR&LhdXZdTW\iad[]\Z`wR[Y V`[]\ZwRcwz\_àdkczT>aQXb[YT>x\h ©hdXZdTW\WP

T : Ð'I*6Ñ¤d¡JdnoPÆV`\VY\ÝV[`wz\Zk2V`kTïÇii| §hJV`\VY\ JaQ\T>kV`kTkZcRwzJ]\Zcz2hx'w `T>ex %\XbV`[ÙWaQ\T>xÇi|³_³PT>wzJ]\Zk_\h UvT>cycR&k %YwR[YV`\JaQ\T>kad[`x\ZYTq[YhBdT>cELhdXZdTW\ JaQ\T>&k %YhfiTWdTWX %IV`kZTa2\XbV`[UWaQ\T>x¯dTWXokZwzhJ[²hdSG³_S±P³IXbV[ÙWaQ\ZwzhJ[ïT>ad[`k\ZàQ\qadcRcTWXZXZhdXokaQXbTUWV`\5h hJ[ÙvT2\ZYTWºcRT>aQj hJVY\hd_a² T>hdXZ UvT>cRchdXJaQ\>T %}adcz\ZYhJVYJÑ\ZYTWºx`h²kTWXbdTÝ\hUbad[YdTµ\ZYTÝwR[UvhJ wR[Y

O>¡


fiT>wzJ]\ZkWP³_YT&T/T>Uv\wRk\ZàQ\\ZYTkaQXbT&\ZYT%hJ[`czµaQXZ\_hdÄ\Z`T%kk\T>\Z`XZhJVYJf`wRUbTWXZXZhdXbkUWad[ `hBf ~adUbjµ©hdXZTWdTWXPi³_wRk aQjdT>k±´³{ ¶ k_VYExàQ\T>k_&T UWwzT>[]\fwz\Z`hJVY\_kZwzJ[`¤w UWad[\ZcRÝVa T>Uv\ZwR[YÝcRT>aQXb[`wR[Y hfiTWX>qTWXbXZhdX `hBf°hJVY\ZkZ¢ÔÓ wyxYThdUvT>cycRk\T>[`x`kq\hxYT>UWa>TvhJ[YT>[]\ZwRadcRcz²ad[fia@l ÄhUb`XZT>wz\TWX %iO>¤d¤OnoP2ÂÃ[ \ZYT T>]VàQ\ZwRhJ[`kq~T>czhBf % Õ fwRcRcGwR[`x`wyUWaQ\T fYTWXZTfiTÝVkTTWXZXZhdX%\XbV`[ÙWaQ\ZwzhJ[ºad[èx %ILhdX%kbwRcRwyUWwz\g> %IV[`czT>kZk hd\ZYTWXbfwRkTwR[`xwRUWaQ\T>ex %YfiTqadkZkZV`T&hJ[`czÝa kZwR[YJczTUvT>cRcTWX_~MczhUbjMP T&kZ\ZaQXZ\_fwz\Zµ\Z`T&V`kZVàdcEkb]VàQXbT>xµTWXZXZhdX_hd1~ âT>Uv\ZwzdT%LV`[Uv\ZwzhJ[µ~adkT>xhJ[\ZaQXZdTW\Zik ÔJäJ O Ú Ù lcÔZnçæ®§iÙ lcÔZnièÕ¥Ô ä lcÔZn]ÞéÓ ä lcÔZn Ö ãÝlcÔZnÕ lãPtJn ä s ä ä f`TWXZoT Ù ä x`T>[Yhd\T>k\Z`T&Tv\TWXb[àdcRczwR5[ âÃT>Uv\T>xTWXZXZhdX>PT wR[`wy wzæWT ãÆwyadXbadx`wzT>[]\x`T>kZUvT>[\~µadxx`wR[Y fiT>wzJ]\qUbàd[YdT>-k óÍfÎÜÏ°\h\ZYTfT>wRJ\Zsk ÍfÎÜÏ lLLXZhJÉV`[wz\ ÑÌ\hV[`wzm\ ÒãnV`kZwR[YµczT>aQXb[wR[Y2XbaQ\yT õà´l Õ © wRk \ZY]T }qXZhJ[YT>UZjdTWXxYT>cz\Zano ãÝcl ÔZn Ódäcl Ôn Ó1ä]cl ÔZn ãÝcl ÔZn ö ö ÕáÞiõ®Ú Õõ¾Ú Ù lcÔn ö óÍfÎÜÏ%lcÔZn Õ Þiõ ö ä ÍfÎÜÏ Ó ä cl ÔZn ÍfÎÜÏ ÍfÎÐÏ ö ö ö ö ä ä ÎÖ ØGÙ&ÔJÎ Ö cl Ôn Ódäcl Ôn Ó cl ÔZn ö Ú Ú ö Õ õ Ù cl Ôn ö Î¤Ö ä ØGÙ&Ô Î¤Ö lcÔZn ÍfÎÜÏ Ó lcÔZn Î|Ö ö ö ö ä ÎÖ Ódäcl Ôn Ó cl ÔZn ØGÙ&ÔJÎ Ö cl ÔZn Ï Ú Ú ö ö Ï Õ õ Ù lcÔn ö Î¤Ö /Î¤ÖßÎeÓ Õ cÔVÞ On® l ä ØGÙ&ÔJÎ¤Öúcl ÔZØ n × Ó cl Ô5Þ Ob nÙ Ó cl ÔZn ÎÖ ö ö ö ä }XZXZhdXbkaQXbT\XbV[ÙWaQ\T>x fYT>[ \Z¢ÝYÜ TW ¢àß cR¢ÔT>Ó a>dTa%T> hdXZ~czhUZj~ kTW\\ZwR[Y%\ZYTLhJcRczhBfwR[Y&x`TWXbwzQaQ\ZwzdT>k}wR[ \ZYT aQ~hdT&T>]VàQ\ZwRhJ[\h æWTWXZhY ÚwÚ áãbâ Û Þ ¢ Ö ¯yÞ ä ß Õ2ÊLhdiX ÒåæWçJè_Ödé=Ø!Öã©.ª ê P ¢ÔÓ õ Ú Ù lcÔn ö Ó1äÎ cl ÔZn ö Ó Î cl Ôn Ó Ï lcÔVÞ On óÍfÎÜÏlcÔn Õ ä Ó cl Ôn ØGÙ&ÔJÎgcl ÔZn ö ö ä Î Ó cl ÔZn Ó ä cl ÔZn Ï ö Ú ö Ù lcÔZn5ì Ó lcÔ5Þ On × lãP£¢]n Õ õ Î ØGÙ&ÔJÎgcl ÔZA n ë Ó cl Ôn ä ö ö

í

ò%ó6îðô ï Ü Þ ¢àß ä

ñ

®YhdXad[åaQXZ~wz\XbaQXb²hJV`\VY\&V`[wz\ @l Ò:ÕZåKåyn\Z`TÝkZV`ëwR["lãP£¢]nqXZT>x`VUvT>k&\hÙ lLfwz\ZIåsÕ åKåRnoPÇi x`wßTWXá T>[]\ZwRaQ\ZwR[Y²T>VàQ\ZwzhJ[çlãsPt¤Jn%fT2hd~`\ZadwR[å\ZYTµV`kZVadc}~adUbjáäXZhdaQJaQ\ZwzhJ[ fiT>wzJä ]\Uoàd[YdT>kLhdX\ZYTÝhJV`\VY\ V`[wz\ZkW Ó ä cl Ôn ö Õ¹à å lcØGÙÔ lcÔnnÕõ Õ lcÔnÕà ä å lcØGÙ&Ô ä lcÔnnÙ ä lcÔZn × lãPtrJn ä ä ä ØGÙÔ cl Ôn ä ö ³h¯UvhJMVY\T \ZYTÝfT>wRJ\Ubàd[YdT>k%LhdX%\ZYTÝhJVY\VY\&JaQ\T>k óÍ8>¡D¢|£ Ï fT2kTW\2@l ÒiÕ÷öJøWÔZnwR[lãP£¢]noP²³_YT XZT>kbV`cz\ZwR[Y\TWXb kUWad[~TqxYTW\TWXb wy[YT>xµ~]µx`ßw TWXZT>[]\ZwRaQ\ZwR[YT>VàQ\ZwzhJ[`k%lãsPROn %GlãsP dn_ad[`xºlãsPt¤Jno 5¡l¢|£ cl ÔZn Õ¹à å lcØGÙ&Ô 5¡l¢|£lcÔZnn Ö ö Ó1äcl ÔZn Ù lcÔnÕIl´ lcÔZnn"Í Õ lcÔn × Ó ö £± ä £± ä 5¡l¢|£ 5 l ¡ | ¢ £ ä ØGÙ&Ô 5¡l¢|£cl ÔZn Ó cl Ôn ö ö ÂÃ[`kTWXZ\Zwy[YÝ~hd\Z\TWXo kqwR[ïT>VàQ\ZwzhJ[lãP£¢]nJwzdT>_k Õ 5ª ¡l¢|£ %e\ZYT UvhJ[]\Xbwz~V`\ZwzhJ[ïhd\Z`T ~czhUZ j )k §Jáä\ZÑUvT>cRcG\h Õ 5¡l¢|£ Pik_TWdTWXZUvT>cycewR[aÝT>hdXZµ~czhUZjµUvhJ[]\Xbwz~VY\T>k\h \ZYT&fT>wzJ]\Uoàd[YdT%hd´\Z`T&hJVY\V`\JaQ\>T %@fiT

¤ )ÜeØ

úTï 1ï

Ú >ØÙÆKÅiØqí]¥ÛÚ/¥qëÚGÛiÜ

O>¤

qÿiò $ð

àdT&\hµkZV`óhBdTWXadcRceUvT>cRcRk§wR[~czhUZj¦Ý\h2hd~`\ZadwR[\Z`Tq\hd\ZadcÕ >¡D¢|£ hdG\ZYT)¦Qáä\Z²T>hdXZ~MczhUbj lLfwz\Z ¨ © UvT>cRcRkono

£

Õð5¡l¢|£l n 5å l¡ ¢|£ l 5¡l¢|£l Znn¼ùúpû Il´ £± l Znn ªò ä cÔ _Õ¹à

cØGÙ&Ô

cÔ

Ú

cÔ wzJ]\Ubàd[`dT>k©hdXhJVY\V`\V`[wz\Zkad[`x²hJVY\MVY\JaQ\T>khd T> hdXZ~czhUZjkWPÄ³_YT>wzXixYTWXbwzQaQ\ZwzhJ[ fiadkadcRhJkZ\k\Zad[`xaQXbxÝÇ|³_³n%fwz\Z TWXZXbhdXikZwRJ[àdcRk\XoV`[ÙWaQ\T>xÝhJ[ÙvT \ZYTWczT>adTÝT> hdXZï~czhUZjkÝlmwR[ÙWcyV`x`wR[Yµwz\ZkJaQ\T>k noP³_`wRk\XbV[ÙWaQ\ZwzhJ[ xYhT>k&[`hd\&aV@T>Uv\±Ý³¶N k&czhJ[Y \ZwR TcyaQ UWaQaQ~wRcywz\ZwzT>ki~VY\wRkUvXbVUWwRadc©hdXadcRcT>]VaQ\ZwzhJ[`khd´\ZYT~adUbj]faQXbxMadkZkad[`xkZ`hJV`cRx2~TjdTW`\_wR[ wy[`xEP ®`hdXfiT>wzJ]\Zk\hµUvT>cyAc %@wy[YVY\_JaQ\Tad[`x¯LhdXZdTW\JaQ\TfTadxYhd`\ad[ïS´³S±EáähdXowzT>[\T>x¯TWXbkT>Uv\Zwzd>T %@~] `Xok\ikZ\ZaQ\ZwR[Y\ZYTqwRC [ VYT>[ÙvThdáUvT>cyAc kwR[]\TWXb[àdck\ZaQ\_T y £± hJ[\ZYTTWXbXZhdX_ad[`xµ\ZYT>[ad[àdcR]æ>wR[``hfÕT>adUo fiT>wzJ]\\h\ZYTUvT>cRchdX\ZYT~cRhUb:j kJaQ\T>k2UvhJ[\Xowz~VY\T>k\þ h £± PÁhïfT¯kcRwz\%\ZYTdXbadx`wRT>[\ wR[ aïfa> x`'w @TWXZT>[]\©XbhJó\ZYTqhJ[YT%V`kT>xwR[µT>VàQ\ZwzhJ[ºlãP£¢]n %[YTWJczT>Uv\Zwy[YkW ¢ÔÓ Þõ ö ã2cl Ôn ö £± cl Ôn Õ¹õ Ù £± cl ÔZn ö £± cl ÔZn × lãP dn óÍfÎÜÏ%cl ÔZ n ÕáÞõ ö ãÝÍfÎÐcl Ï ÔZn Õá £± lcÔZn ÍfÎÐÏ ÍfÎÜÏ

%ò í ó ¯ îðÛ5ï ÿ £± ñ Þ ¢àß ³_`T>kT%\TWXo kaQXZT\ZYTwR[]\TWXb[àdc´k\ZaQ\TTWXZXZhdX £± ad[xaÝMaQXZ\ZwRadc Ú Ú Ü £± hd³ £± fwz\ZXbT>kT>Uv\\hfiT>wzJ]\Zk â é nqhdX\ZYTµ~cRhUbj kLhdXZdTW\JaQ\T ÆLTWT>x`wR[`¯\ZYTUvT>cRc³©ª l ©ª n&hdX\ZYTµ~czhUZj kwR[YV`\qJaQ\TÑl ÷ l èGn adkadcRc\ZYT>kT&fT>wzJ]\ZkUvhJ[]\Xbwz~MVY\Tq\h \ZYT%UWadcRUWV`cRaQ\ZwzhJ[µhd# £± l ZnoP}T&\XbT>aQ\_\ZYTqaQXb\ZwRadcELhdX_\ZYT wR[]\TWXb[àdcEkZ\ZaQ\T>k_TWXZXZhdX £± ad[adczhddhJV`kZcz2\h¯lãP£¢]n_ad[`xhd~`\ZadwR[e £± Ýl Zn ¢ÔÓ Ýl Zn l n £± l Zn l Zn l Zn l£± n

± ± £ £ D £± l Zn l Zn l n D £± l n D £± l n í ò l £± îðZÜ ïn ô Þ ¢àß ñ âw TWXZT>[]\ZwRaQ\ZwR[Y\ZYTLhdXZfaQXbxadkbkT>]VaQ\ZwzhJ[ºlãsP dn @fThd~`\Zadwy[E £± 5¡l¢|£ l n å l´ £± l nn £± l n V`~k\Zwz\ZVY\Zwy[Y&\Z`wyk\TWXbwR[µ\Z`TqT>]VaQ\ZwzhJ[LhdX £± lãPt¡Jn £± l n 5¡l¢|£ l n å l´ £± l nn ë £± Õ l n5ì ³_wR± kwR[]\TWXb[àdcGk\ZaQ\TTWXZXbhdXq[YTWT>x`k\hµ~@TUWadcRUWV`cyaQ\T>x¯LhdXT>adUbºT>hdXZUvT>cycãP³IhUWadcRUWV`cyaQ\T\Z`TMaQXZ\ZwRadc Ú Ü £ wR[µT>VàQ\ZwzhJ[ÑlãP dnfiTqx`w TWXZT>[]\ZwRaQ\TqT>VàQ\ZwzhJ[ºlãPtsJn_ad[`xhd~`\ZadwR[akZV`óhd´©hJV`X_\TWXb kWP Ú â £± l Zn £± l On Ê £ l Zn ® £ l n @ l £± l nn í ò îðï Ê £± ñ í ò îðò ï £± Þ ß ñ ® @l £± l Znn £ l Zn ® £± l On .Ê £ l Zn lãPt¤Jn í ò ò îðï Þ ñß ò í ò Ê îðï Þ ñ ß ö

ö

ö

ö

sÙ

ÍfÎÐÏ

@ÒPÕ

@ÒmÕ

@ÒnÕ

%

:

cÔ

Ù

Ù

Ø

cÔ

ö

èÕ»Þ

ã

cÔ

cÔ

ö

ö

ÕáÞ

ö

ã

cÔ

Ó ä cÔ

ö

Ó ä cÔ

Ó

ö

Ó

ö

cÔ

cÔ

cÔ

ö

Ó

ö

Õ

Ú

cÔ

ö

Ó ä cÔ

ö

ö

Ó

ä

ß

Ù

cÔ

%

Ó

ö

cÔ

ö

Õ¥Ó

cÔ

Ù

Ù

cÔ

Õ

Ó

cÔ þ×

cÔ

Ú

cÔ

Í

ä

ä

cÔ

×

ä

ß

cÔ

ö

ö

ÍfÎÜÏ

Õ

cÔ5Þ

ö

ö

Ó

ÍfÎÜÏ

Î

cØGÙ&Ô

cÔ

Ó

Ó

Î

cÔ ®ö

ö

cÔ

ÍfÎÜÏ

#$&%('()+* ),!

cØGÙÔ

ö

cÔ

ö

Î

cÔ

ÍfÎÜÏ

"!

cÔ5Þ

Ó

Î

®ö

ö

cÔ

ÍfÎÐÏ

*!")* ),!

×

ä

cÔ

sQÊ


â wßTWXZT>[]\ZwRaQ\ZwR[Y \ZYT LhdXZfaQXbxçadkbkT>]VaQ\ZwzhJ[`kºlãsPt{Jn%ÝlãsPtsJnad[`x lãPROnµLhdX`W%nÓ W%ad[`x¹Ó.Ê-fiTºUWad[ kZV`~k\Zwz\ZVY\T\ZYTV`[YXbT>khJczdT>x²aQXZ\ZwRadcykad[xkcRwz\\ZYTTvXZT>kZkZwzhJ[²hJ[ï\ZYTXowzJ\àd[`xkbwRxYThdlãPt¤JnwR[]\h \ZYXbTWTkTWaQXbaQ\TT>]VaQ\ZwzhJ[`kLhdX}\Z`TUvT>cRcIl@ÒGÕv ©ª n%\ZYTwy[YVY\GJaQ\Tl@ÒGÕÏé=ØnÄad[`x \ZYTLhdXZdTW\}JaQ\Tl@ÒPÕèGno £± cl ÔZn Õ ö £± cl Ô5Þ On Ó Ê £ lcÔZn®r å lcØGÙ&Ô lcÔZnnqÓ £ lcÔn"Ó Ï lcÔ5Þ OnÖ ö lãPROWÊn ö

Í

ö Í

ö

£±

Ï

£± l Zn £ £± l Zn Ê£ Ï

cÔ

ö ö

ö Õ

Í

Ï

Õ

ö

cÔ5Þ

ö

cÔ5Þ

Ï

Í

Ê £ l Zn®r@l 5 £± l nn å £ l £ l Znn Ê £ l Zn® £± l On Ê å £ l Ê £l Znn l

Ó

Ï

Í

ö

£±

Ï

£± l On £ £± l On Ê£ ö

cÔ

£±

Í

Ó

cÔ

cØGÙÔ

cÔ

cÔ5Þ

cÔ fà

à

cØGÙÔ

cØGÙÔ

cÔ "Ó

cÔ "Ó

Ï

lcÔVÞ OnnÖ

lãPROdOn

On×

lãPRO>sJn

Ï

cÔVÞ

®`V`XZ\ZYTWXbhdXbT\ZYTqwR[`wz\Zwyadc@kZ\ZaQ\T&hdG[YTW\gfihdXZjxYhT>k[Yhd\_xYTWT>[`xµhJ[\ZYTqfiT>wzJ\Zk%khfiT&à>dT £± cl ÔÕ)Ên Õ)Ê ©hdXÝÒJn

^ hd\T\ZàQ\\ZYTXZT>UWV`XbkZwzhJ[`kGwR[T>VàQ\ZwzhJ[`klãPROWÊngáolãPRO>sJn}x`TW@T>[x%hJ[\Z`T_adUv\ZVàdcàdUv\ZwzQaQ\ZwzhJ[hd@\ZYT~czhUZj k LhdXZdTW\_JaQ\TdP-YT>[\ZYTqadUv\ZwzQaQ\ZwzhJ[dhT>k\h æWTWXZhÝ[Yhd\hJ[`cz2\ZYTqUvT>cRcA kkZ\ZaQ\T>%~VY\adcRkZh \ZYTaQXZ\ZwyadcRkaQXZT XZT>kZTW\lLLhdXZdTW\\ZwR[YµwR[ÙWcRVxYT>k©hdXbJwzwy[Y `XZTWwzhJV`k wRk\ZaQjdT>konoPdTWXZUvT>cRc[YTWT>x`k\h jdTWTWa UvhdhdÄT>adUo hd´\ZYT>kTq\Z`XZTWTaQXZ\ZwRadcyk_ad[`xVYExàQ\T\ZYT> aQ\_TWdTWXZµ\ZwRT&kZ\TWeP TUWad[µwR[`kTWXb\\ZYTaQXZ\ZwRadcRkwR[ T>VàQ\ZwzhJ[ïlãP dnad[`x2UWadcRUWV`cRaQ\T\ZYTUvhdXbXZT>khJ[`x`wR[`qfT>wRJ\VYExàQ\T>&k % fwR\Zµ\ZYT&wR[]\TWXb[àdcEk\ZaQ\T%TWXZXZhdmX Ù £± cl ÔZniJwzdT>[~]2T>]VaQ\ZwzhJ[ÑlãPt¡JnoP³_YT&xßw @TWXbT>[ÙvT~TW\gfiTWT>[VYExàQ\T>khd fiT>wzJ]\Zk\haUvT>cRcwz\ZkZT>cz}@l ÒGÕv©ª nad[`xµVYExàQ\T>kihdIfiT>wzJ]\Zk\h\Z`TJaQ\T>kwRki\ZàQ\Ubàd[YdT>k_\hfiT>wzJ]\Zk\h \ZYTqUvT>cyGc óÍ £± Ï hJ[`czµxYTWT>[`x2hJ[\ZYTMaQXZ\ZwRadcRkhdG\Z`wRkUvT>cyAc khf[k\ZaQ\Td óÍ Ï lcÔn!ÕõIÙ lcÔZn ö £± cl ÔZn × lãPROv¢]n

£±

£±

ö

Í

£±

Ï

³h2V`xaQ\T%\ZYTfiT>wzJ]\Zkhd\ZYTwR[YVY\JaQ\T ad[`x²hd\ZYTLhdXZdTW\JaQ\T>%IYhfiTWdTWX%efiTà>dT\hkZV`8hBdTWX \ZYTqUvhJ[]\Xbwz~MVY\ZwzhJ[`khdGadcRceUvT>cRcykwR[µ\ZYTq~cRhUbjM £ óÍfÎÜÏ%lcÔZnÕ¹õ Ú û Ù lcÔn ö y £± cl ÔZn LhdXÛÒæHçJè_Ödé=Ø × lãPRO>rJn

± £ ªò ä

ö

ÍfÎÜÏ

ê

³_`TT>]VàQ\ZwRhJ[`k%[`T>UvT>kZkZaQXZï\h¯wyczT>T>[]\\ZYTÝ~adUZjfiaQXoxºadkbk&aQXZTïlãP£¢]n%lãPtrJn%lãPt{Jn%lãPt¡Jn%lãPROWÊn% lãPROdOn %GlãPRO>sJn %GlãPRO>Jn %ÄlãPROv¢]n_ad[xºlãPRO>rJnoP

T :T ?A"7C .#&´$l-

³ hqUWadcRUWV`cRaQ\T\Z`T_UvhJVY\ZaQ\ZwzhJ[adcÙvhJczTvYwz\ú%hd@Tv\T>[xYT>x ±Ý³¶ófiT\ZaQjdTwR[]\hqadUWUvhJV`[]\\ZàQ\fiT>wzJ]\Zk \hwy[YVY\JaQ\T>kad[`xLhdXZdTW\qJaQ\T>kUWadV`kT hdXZT Tv@T>[kZwzdTVYExàQ\T>k\Zàd[ºhd\ZYTWXbk&%~T>UWadV`kT T>adUoºkZVÙo fiT>wzJ]\x`wRXZT>Uv\ZczaV@T>Uv\Zkadcyce\ZYTUvT>cRcRkwR[wR\ZkT> hdXZ~czhUZj@PTTWQadcRVàQ\Ta XbaQ\Z`TWX\ú]wyUWadc\hdhJczhdd V`kZT>xÑwR[Ñ\ZYT2TvTWXbwyT>[\ZkµlmkTWT®GwzJVYXZTPtJnoP cRc T>hdXZ ~cRhUbjkà>dTµ\ZYTkZad T2kZwRæW>T §}JaQ\T>k àdT [YhhJV`\dhJwR[YºUvhJ[`[`T>Uv\ZwzhJ[`&k §hJVY\MVY\ V`[`wz\Zkad[xJaQ\T>kÝa>dTa~MwRadk UvhJ[`[YT>Uv\ZwRhJ[ lL©XbhJ a V`[`wR\fYhJkT adUv\ZwzQaQ\ZwzhJ[wykadczfiak&OQP Ên §hd\ZYTWXUvhJ[[YT>Uv\ZwzhJ[`k\hhJV`\VY\_V`[`wR\ZkikZ\T>ó©XbhJóT>hdXZ2~czhUZjkhJ[`cz §Y\ZYT `wyx`xYT>[cRadTWXwRkmV`cRcR2UvhJ[`[`T>Uv\T>xEP±eTW\ yÖ¨]Ö CÖ x`T>[Yhd\T\ZYT%[V`%~@TWXokhd} T>hdXZ~czhUZj&k %T>hdXZ .-

0/ 01

"Û mì}Û_íið ¯Û}ÙÚÜ

úTïûúTï

sO

32

vU T>cRcykwR[T>adUoï~czhUZj %MwR[`VY\_V`[`wR\Zk&%ad[`xhJVY\VY\V`[`wz\Zk&%XZT>kT>Uv\ZwzdT>czdPT-[`x\ZYT%VYExàQ\T%UvhJczTvYwz\g TWX\ZwRTqkZ\TW\h ~Td

í 6«¤¨ l ï yñð¨kí ®)îO{Jn YT>xµfThd~`\Zadwy[a\hd\ZadcIUvhJV`\ZaQ\ZwzhJ[àdcEUvhJcRTvwz\ú hd# L l l¨ noP³_YT[]V&~TWX )

Õ

4

-65

75

)

! 8

-65

}TWTWMwR[Y

ad[x hd´fT>wRJ\ZkwRkW Õ

5

-65

F

æ

#$&%&'9)

=1C5-G5

=/H5-65

Ö

J/

I1

K

æ

) > '()%('() * ),!

!"' , !"$0)? $&$(!;),# $ 8@ $&:BA&# 8 7/E5 =-65 '()%('()

=-

*!;)* )! 8

¤d¡d¤Jn%ÄczTWTWXZT> ad[`kTW\adcãPlÃO>¤d¡d¤Jn%®àd`cy ad[ lÃO>¤d¤On %pÄhUb`XZT>wz\TWX *°YUb` wyx`]V`~@TWXlÃO>¤d¤JdnoPihJ[`kZwyxYTWX®´wRJVYXZT&PtsP õö2H ÷ S ³_YTÄ\Xbadxwz\ZwzhJ[àdcJ TW\ZYhx%k\ZaQXZ\ZkáQ\I\ZYTiczTWL\ZhJk\[`hxYTÄhdY\Z`T}S_uÁdXbaQe %Qad[`xqkZT>]VYT>[]\ZwRadcRcR dT>[YTWXoaQ\T>k [`wz\Tk&~hJc@k\XbwR[`JklL~TWJwR[[`wR[Y%fwz\ZÝ\ZYTT>`\úÝk\Xbwy[Y]nÄ~ÝkZ\TWwR[Y©XbhJó[Yhx`T\h [YhxYT LhJcRczhBfwR[YÝ\ZYTT>xYdT>khd}\Z`TdXoaQe %Ead[`xaQ`T>[`x`wy[Y \ZYTk&~hJcRkadkZkZhUWwRaQ\T>xïfwz\Z²\ZYTT>xYdT>k\h\ZYT UWVYXbXZT>[\Äk\XowR[YV`[\Zwyc]\Z`TiXowzJ\Z hJk\Ä[Yhx`TwykXbT>adUbYT>xEPxYdT>kÄaQXZT_UoYhJkT>[Xbad[xYhJ czqwzM\ZYTWXZTwRkGaUb`hJwRUvT lL`Xbhd~aQ~wRcRwR\g Õ)1Ê ×trJnoP ÂÃ[YV`\ad[x\ZaQXZdTW\kZ%~hJcRkaQXZTXZTW`XbT>kT>[\T>x~]x`wyT>[`kZwzhJ[adc~wy[àQXZ dT>Uv\hdXo&k %T>adUo¯UvhJhJ[YT>[]\ k\Zad[x`wR[Y LhdXhJ[YT%hd}\ZYT hJkZkZwz~MczTqk&~hJcRkWP ÄT>[ÙvT\ZYT[YTW\úfhdXZj¯àdkq wR[`VY\_V`[`wR\Zkad[`xhJV`\VY\ V`[wz\ZkWPÝ³YT \Zadkj wRk&\hXZT>adxÑk\XbwR[`J&k %IhJ[YT2k&~hJcaQ\a\ZwR>T %Gad[`xº\h¯UvhJ[]\ZwR[]VadcRczï`XZT>x`wyUv\\ZYTÝ[YTv\ hJkZkZwR~czTk&~hJcglmk noPÂÃ[YVY\IdT>Uv\hdXbkàdTiTvYadUv\Zcz%hJ[YTi[`hJ[YæWTWXZhUvhJhJ[YT>[]\WP´³áQXZdTW\ÄdT>Uv\hdXbkÄ a%àdT \úfUWadV`kTkhJ TW\ZwRT>ki\ZYTWXZTwRkaUbYhJwyUvThde\gfihhJkZkZwz~cRTk&~hJcRkiaQ\\ZYT[YTv\kZ\TWePÕ`XbT>x`wRUv\ZwzhJ[ wRkUvhJ[`kbwRxYTWXZT>xUvhdXZXbT>Uv\wzI\Z`TTWXZXZhdXaQ\T>adUohd´\ZYT%hJVY\V`\V[`wz\ZkwRk~T>czhfÕ1Ê ×£¢¤¯lLTWXZXZhdXkbwzJ[àdcRkihUWUWVYX aQ\_TWdTWXb2\ZwyTqk\TW@noP


sds

S B

T

X X

P

S

E

P

B

T P

V T

V

Reber Grammar

Reber Grammar

T

E

P

recurrent connection for continual prediction

®GwzJVYXbTPts³IXbad[kZwz\ZwzhJ[x`wyaQdXbad kLhdXkZ\Zad[`xàQXbxlmcRTW©\onad[`xT>%~@T>xxYT>xlLXbwzJ]\onÄSTW~TWXdXbad aQXokWP³_YT xàdkbYT>xcRwR[YTwR[x`wRUWaQ\T>k\ZYT&UvhJ[]\ZwR[VàdcEQaQXbwRad[]\WP czdhQá `wRxxYT>[ fT>wRJ\Zk czT>aQXb[`wy[Y ÆhdÄkZVÙWUvT>kbk kZVUWUvT>kZk Xowz\Z` V[`wz\Zk XbaQ\T aQ©\TWX SG³_S± O¨Ê ÊP ÊJr ÞkZhJTqLXbadUv\ZwzhJ[`ß OQd % ÊdÊdÊ SG³_S± O>s à¢¤¨¢ ÊPRO ÞkZhJTqLXbadUv\ZwzhJ[`ß sddr % ÊdÊdÊ }±¶ O>r à¢dr Ê ²sQÊd1Ê % ÊdÊdÊ S__ áú¤ OdO>¤BábO>¤d¡ rQÊ O>¡dds % ÊdÊdÊ ³XbaP ±G³¶ ¨~Mcã'P % kZwzæWTqs sJQ{ ÊPtr OWÊdÊ ¡ %£¢d¢JÊ d ³áQ~czTPROQ\Zad[`xàQXbx T>&~T>x`xYT>x S_TW~TWX}dXbad aQXlmS_uqno]TWXbUvT>[]\ZaQdThdekZVÙWUvT>kZkmV`c\XbwRadcRkad[xÝ[V`á ~TWX}hdekZT>]VYT>[UvT`XbT>kT>[\ZaQ\ZwRhJ[`kV[\ZwRcMkZVÙWUvT>kZkLhdXiSG³_S± lLXbT>kZV`cz\Zk}\ZaQjdT>[µ©XZhJó wR\Z ad[`wx ÷wzkTWXO>¤d¡d¤ n %Þ}cR ad[[`TW\\XbadwR[YT>x ~ÑcR ade[ k`XbhUvT>x`V`XZT>ßlLXZT>kZV`cz\Zk\ZaQjdT>[©XbhJczTWTWXZT> ad[kTW\ adcãP)O>¤d¡d¤ºn % ÞST>UWVYXZXZT>[]\2adkZUWadx`TváòihdXZXZT>cRaQ\ZwRhJ[`ß"lLXZT>kbV`cz\Zk\ZaQjdT>[-LXZhJ ®àd`cy ad[ O>¤d¤Onad[`xÁ\Xbadx`wz\ZwzhJ[adc_±´³¶ lLXZT>kbV`cz\Zk\ZaQjdT>[©XbhJv ÄhUoYXZT>wz\TWX_ad[xYUb` wyx`]V`~@TWXO>¤d¤JnoPT>wzJ]\[V`%~TWXbkwy[ \ZYmT `Xok\}¢XZhBfkaQXZT T>k\Zwy aQ\T>kWP U

U

V

W W W

W

³h UvhdXZXZT>Uv\ZcRº`XZT>xwRUv\\ZYTk%~hJci~@TWLhdXZT\ZYTcRadkZ\l ¦ hdX ¾ nwR[ ad[ÁS_u,k\XbwR[`T>%~@TWX\ZYTkT>UvhJ[`xïk&~hJclmadcykh ¦ hdX ¾ n_fwz\ZYhJV`\UvhJ[YmV`kZwR[YÝwz\fwR\ZwRxYT>[]\ZwRUWadck%~hJcRk T>[ÙvhJV[\TWXZT>xcRaQ\TWX>P³_YT wy[`wR adcM\ZwRTcRaQ wRk_µlmaQ\_\Z`TqcRwR wz\hdfàQ\k\Zad[`xàQXbxµXbT>UWVYXZXZT>[]\[YTW\gfihdXZjk UWad[Ý ad[àQdTBn§Y\ZwRTcRaQJk}àdT[`h&VY`TWXÄ~hJV`[`x\Z`hJVYJEPÄ³_YT_TvT>Uv\T>x czT>[`d\Z hdea&kZ\XbwR[YdT>[YTWXoaQ\T>x ~Ýad[S_u/wRkOdOQPtr k%~hJcRkWPÄ³_YTczT>[`d\Zhd\Z`TqczhJ[YdT>k\kZ\XbwR[YwR[µa kTW\hd [YhJ[ágwRx`T>[\ZwRUWadck\XowR[YJk wRki`Xbhd@hdXb\ZwzhJ[àdc`\h cRhd lmk\ZaQ\Zwyk\ZwRUWkhd\ZYTT>&~T>x`xYT>xµSTW~@TWXuXoad aQX_aQXZTx`wRkbUWV`kZkT>x2wR[µ@T>[x`w noPÝ®YhdX%\ZYT \Xbadwy[`wR[Yad[`x\T>kZ\%kTW\Zk%V`kT>x wR[ïhJV`X&Tv@TWXowRT>[]\Z&k %e\ZYT TvT>Uv\T>x QadcRVYThd\ZYT cRhJ[YdT>k\ k\XowR[YwRkdXZT>aQ\TWX\Zàd[rQÊP ³áQ~czTPROkZV` aQXbwzæWT>kÄTWXZLhdXb ad[ÙvThd@XZTWwRhJV`kÄS^^k}hJ[ \ZYTk\Zad[`xàQXbxÝ}S_u)XZhd~czT> lL\T>kZ\ZwR[Y wR[]dhJczdT>xa \T>k\kZTW\_hd}sdrd{ Su \T>k\k\XbwR[YJk noP[`cR \Xbadx`wR\ZwzhJ[àdce±G³¶ëadczfiakczT>aQXb[`k\h khJczdT%\ZYT \Zadkj@PdT>[¯fYT>[fTwzJ[YhdXZTq\Z`T%V`[`kbVÙWUvT>kZkmV`c\XbwRadcykhdÄ\ZYT&hd\Z`TWXaQ`XZhJadUbYT>k %@±G³¶ëczT>aQXb[`k%VÙo madk\TWX>P ½õö2H ÷ S VYXihdXZTx`'w UWVcz\UvhJ[\Zwy[]VàdcMQaQXbwRad[]\}hde\ZYT}S_u `XZhd~cRT> läSu&nxYhT>ki[Yhd\XZhwRxYT wR[`©hdXb aQ\ZwzhJ[ÝaQ~hJVY\\ZYT~TWJwR[[`wR[YJkiad[`xµT>[`x`kihdGk&~hJck\XbwR[`JkWPG-wz\Z`hJVY\wR[\TWXoT>x`wRaQ\TXZT>kZTW\Z&k %Y\ZYT [YTW\úfhdXbjwRk´XZT>V`wzXbT>x%\hqcRT>aQXbe[ %JwR[ad[hJ[ágcRwy[YTmadkZ`wRhJe[ %d©XbhJ/wR[YVY\´k\XZT>ad k}UvhJ[`kZwRk\Zwy[YhdUvhJ[ÙWaQ\T>[àQ\T>x Suçk\XbwR[YJk>PIÂÃ[YVY\Äk\XZT>ad kaQXZT_k\hd`T>xadkkZh]hJ[adkÄ\Z`T_[YTW\gfihdXZj aQjdT>k}ad[ wy[ÙvhdXZXZT>Uv\G`XZT>xwRUv\ZwzhJ[hdX \ZYTµOWÊ áä\ZïkZVÙWUvT>kZkbwzdTkZ%~hJcàdkhUWUWVYXZXZT>xEP±eT>aQXo[`wR[Y2ad[`x\T>kZ\ZwR[Yµadcz\TWXb[àQ\TdaQL\TWXT>adUbï\XoadwR[`wR[Y K

K

X


úTïûúTï

sd

32

Out 1

Out 2

Out 3

Out 4

MemoryMemory Out Gate 1 Block Block 1 1 Forget Gate 1 Cell Cell 1 2

In 2

Out 6

Out 7

MemoryMemory Out Gate 2 Block Block 2 2 Forget Gate 2 Cell Cell 1 2

In Gate 1

In 1

Out 5

In Gate 2

In 3

In 4

In 5

In 6

In 7

G® wzJVYXbTPt³YXZTWTcRa>dTWX±Ý³¶\hdhJczhddfwR\ZXZT>UWV`XZXZT>[ÙvTcRwR wz\T>x\hµ\ZYTwRx`xYT>[¯cRa>dTWXUvhJ[`kZwyk\ZwR[Y hd©hJVYXTv\T>[`xYT>xº±´³¶T>hdXZ²~czhUZjk lLhJ[`cz¯\gfihkZYhBf[MnfwR\Z¯\úfh¯UvT>cRcRkT>adUbePÝ[`czacywR wz\T>x kZV`~kTW\hdÄUvhJ[`[YT>Uv\ZwzhJ[`kaQXZT&kZYhBf[EP k \XbT>ad fiT©XbTWTWæWT\Z`TfiT>wzJ]\Zkad[`xÝ©TWT>xOWÊ\T>k\k\XZT>ad kWPVYXTWXZLhdXb ad[ÙvTT>adkZVYXZTwRki\ZYTa>dTWXbaQdT \T>k\k\XZT>adÌkbwzæWT>§OWÊdÊ1% ÊdÊdÊUvhdXZXbT>khJ[`x`k%\hïa²khQágUWadcRcRT>xàÞÃTWXZLT>Uv\Zß²khJcRVY\ZwzhJ[)lÃOWÊ µkZVÙWUvT>kZkbwzdT2UvhdXbXZT>Uv\ `XbT>x`wRUv\ZwzhJ[`k noP Y

T T : Vþ.$DÌ@ 6ÑÂM" "1- '!3(Í'I '7.$EG 6B

_³ `T wR[YVY\V`[`wR\ZkaQXbT LVcRczUvhJ[`[YT>Uv\T>xº\ha`wyx`xYT>[²cRa>dTWXUvhJ[`kZwRk\Zwy[Yµhd¢T> hdXZ¯~MczhUbjkqfwz\Z s UvT>cRcykT>adUoºlã¡UvT>cRcRkad[`xïO>sJaQ\T>k_wy[Ý\hd\ZadcLnoP}³_YTUvT>cRc@hJVY\V`\ZkaQXZTLV`cycz UvhJ[`[YT>Uv\T>x\h\ZYTqUvT>cyc@wy[YVY\Zk&% \h&adcyc`JaQ\T>k&%ad[`x \h%\ZYThJVY\VY\}V[`wz\ZkWP´³_YT_hJVY\MVY\}V`[`wR\ZkÄà>dTadx`x`wz\ZwRhJ[àdcÄÞkZYhdXZ\ZUWV`\Zß&UvhJ[`[YT>Uv\ZwRhJ[`k LXZhJ \ZYTwy[YVY\}V[`wz\ZklmkZTWT®GwzJV`XZTPtJnoP}cRcMJaQ\T>kad[`xÝhJVY\VY\iV`[`wz\ZkiaQXZT~wRadkT>xEPÄÇwRadkfT>wRJ\Zki\hwR[á ad[`xïhJV`\VY\JaQ\T>k%aQXbT wR[`wz\ZwRadcywzæWT>x~MczhUbj]fwRkTd Þ_1Ê ×trµLhdX\ZY~T `XbkZ\~MczhUb:j %Þ&5O × Ê©hdX&\ZYT kT>UvhJ[èx %Þq5O ×tr LhdX\Z`T\Z`wRXbex %]ad[xÝkhLhdXZ\ZEPGÂò[Ý\Z`wRki ad[`[`TWX %UvT>cycMk\ZaQ\T>kaQXbTwR[wz\ZwRadcRczUWczhJkT\h%æWTWXbkZkT>&k %\ZYT~wyadkT>k~@T>UvhJ T`XZhddXZT>kZkbwzdT>czczT>kZk[`TWJaQ\Zwzd>T %IadcRczhBfwR[YÝ\ZYTkTWXbwRadcGadUv\ZwzQaQ\ZwzhJ[ïhdiUvT>cRcRk adkadUv\ZwzdTaQXZ\ZwRUWwzMad[\Zk_wy[\ZYT[YTW\gfihdXZjUvhJV`\ZaQ\ZwzhJ[EP_®YhdXZdTW\JaQ\T>kaQXZTwR[`wz\ZwyadcRwzæWT>x2fwz\Z¯k TW\XbwRU hJkZwz\ZwRdTQadcRVYT>kWC ®ÊPtr©hdX\Z`nT `Xbk\~czhUZ j %+®OLhdX_\ZYTqkT>UvhJ[`x~czhUZ j %TW\ZUQPi|XZT>UWwykT~wRadkwR[`wR\ZwRadcRwzæ>aQ\ZwzhJ[ wRk[`hd\&UvXbwz\ZwRUWadc\ZYhJVYJK 9"hd\Z`TWXqädcyVYT>kfihdXZyj âV`kZ\qadkqfiT>cRcãP cRcÄhd\ZYTWXqfiT>wzJ]\Zk&wR[UWcRV`x`wR[`Ý\ZYThJV`\VY\ ~wyadkaQXZTwR[wz\ZwRadcRwzæWT>xXbad[`xYhJ cRwR[\Z`TXoad[YdkT «ßÞ_1Ê ×tds ÖZ1Ê ×tys ¬äP%³_`TWXZTaQXbT¢s¨¢ad5x âV`kZ\ZaQ~czT&fiT>wzJ]\Z&k %Ef`wyUb wRkUvhJ aQXbaQ~czT\h \ZYT&[V`&~TWX_V`kZT>xµ~]±Ý³¶ÉwR[kZhJczwy[Y\ZYT&}S_u lmkTWT&³áQ~czT&PROnoP T>wRJ\Ubàd[YdT>kaQXZT2 adxYT aQL\TWX%T>adUo wR[YVY\k%~@hJc`XZT>kT>[]\ZaQ\ZwzhJ[EPµ\&\ZYTÝ~TWJwR[`[`wR[`ÝhdT>adUo \Xbadwy[`wR[Y²k\XZT>adw %\Z`TµcRT>aQXb[`wR[Y²XbaQ\rT õ wRk%wy[`wz\ZwRadcRwRæWT>x fwz\Z ÊPtrPÑÂú\T>wz\ZYTWXXZT> adwR[`-k YT>xÑhdX x`T>UWa>k ~a madUv\hdXhdÊPt¤d¤ TWX\ZwRT&k\TWÁlm±G³¶ëfwz\Z õEágxYT>UWaYnoP±eT>aQXo[`wR[Y XoaQ\TxYT>UWawRkfT>cRck\ZV`x`wRT>xwR[ k\ZaQ\Zwyk\ZwRUWadcaQ`XZh>YwR aQ\ZwzhJ[ï\Z`TWhdXZ²ad[x wRkadcRkhUvhJ hJ[ºwR[ï[YT>V`XbadcÄ[YTW\gfihdXZj&k %´TdP£YP¯lmâaQXZjdT>e[ %O>¤d¤drJnoP

s¨¢

×]êëìÚGÛ_íþúTïÿIÛëíÙiðÃÙÄ¥ Ú_ØvÅØíc¥ÛÚ³Æ×ØÙÚ]ðòÙò¦ëÿ{ìíÛñð>×_Ú$ð>ØÙ¸Rð>Ú]êáÿ´ÜeÚ# czdhdXowz\Z` hJcyVY\ZwzhJ[`k uhhx²hJcãP ³IXbaP±´³¶Éfwz\ZµTv\TWXb[àdcIXZT>kTW\ B¯ ¢ läB¢d¢Òn Ê JÞ ³Xbadx`wR\ZwzhJ[àdcE±Ý³¶ Ê l©áòn O ÃOdO>{d{ ±G³¶Åfwz\Z\ZaQ\T%âT>UWaºlmÊPt¤Jn Ê l©áòn Ê JÞ ±´³¶Åfwz\Z®`hdXZdTW\uaQ\T>k O>¡Ãl O>¡d¡d¡d¤Jn sd¤ ãd¤ OO ±´³¶Åfwz\Z®`hdXZdTW\uaQ\T>k ad[`xkZT>]VYT>[]\ZwRadGc õåxYT>UWa {dsÃl Ov¢JÊJ¡Jdn { ã{d¡¨¢ {¨¢ V

V

[Z x`xYT>x STW~TWXuXoad aQXlä}S_uqnoìhJcRV` [ Þ hJcRV`\ZwzhJ[`kZßG|´TWXbUvT>[\ZaQdT hdÞÃ@TWXb©T>Uv\ZßkhJcRV`\ZwzhJ[`klmUvhdXZXZT>Uv\`XbT>x`wRUv\ZwzhJ[ÝhdOWÊk\XbT>ad khdOWÊdÊ1% ÊdÊdÊÝk%~hJcRkT>adUbMn%wy[ÝMaQXZT>[\Z`T>kZwRk \ZYT_[V`%~TWXhdM\XoadwR[`wR[Yk\XbT>ad k`XbT>kT>[\T>x V`[\ZwycYkhJcRVY\ZwzhJ[fadkXbT>adUbYT>xEPhJcRV` [ÞbuhhxÝhJcãP ßG|´TWXá UvT>[]\ZaQdThdIkhJcRV`\ZwzhJ[`kfwz\Z2ad[2adTWXbaQdTqk\XZT>adócRT>[Yd\Z4 ²ËOWÊdÊdÊlmT>ad[µczT>[`d\ZÝhdITWXZXZhdX©XbTWT`XbT>x`wRUv\ZwzhJ[ wRkiJwzdT>[wR[ad[YJczT~`XoadUZjdTW\ZkonoPihJcRV` [ ÞST>k\Zß}TWXbUvT>[]\ZaQdTqhdÞÃ~adx`ßkhJcRV`\ZwzhJ[`kfwz\ZµadTWXbaQdTkZ\XZT>ad czT>[`d\Z OWÊdÊdÊ²lmT>ad[¯cRT>[Yd\ZhdGTWXZXZhdX©XZTWT&`XZT>x`wRUv\ZwRhJ[wRkJwzdT>[¯wR[ad[YJczTq~XbadUZjdTW\ZkonoP³_`TqXZT>kZV`cR\Zk_aQXZT adTWXbaQdT>khBdTWX_OWÊdÊwR[`xYTWT>[`xYT>[]\Zcz\XbadwR[`T>x%[YTW\úfhdXbjkWP\ZYTWXÄadczdhdXbwz\Z` kcRwRjdTiÇ|}³³åaQXbT[`hd\Gwy[ÙWcRV`xYT>x wR[µ\Z`TqUvhJaQXbwRkZhJe[ %Y~T>UWadV`kT\ZYTWµ\T>[`x\h LadwRcTWdT>[hJ[\ZYTqT>adkbwzTWX %[YhJ[YágUvhJ[\ZwR[VàdcESuP bV

dc

TïXZTWhdXZ\2XZT>kZV`cR\Zk hdqTvhJ[YT>[]\ZwRadcõeágxYT>UWa/lmadkµkT>UWw'T>x"aQ~hdTBn%~VY\µadcRkhå\T>k\T>xçkTWdTWXoadchd\ZYTWX QaQXbwRad[]\ZklmcRwR[`T>aQX%O s%@O n%]ad[x©hJV[`x\ZYT> adcycY\hqfihdXZjadkfiT>cRcfwz\ZYhJV`\ÄTv\T>[`kbwzdT_hd`\ZwR wzæ>aQ\ZwzhJ[ hdáQXbadTW\TWXokWP OeOf

Oeg

f

T T :T ?AUWÎRUH+BJ=C´$B ³XbadwR[`wy[Yfiadkk\hd`T>xaQL\TWXaQ\hJk\QÊdÊdÊdÊ2\XbadwR[`wR[` k\XZT>ad k&%MT>adUb²hdGf`wRUofadk_T>[`x`T>xfYT>[\ZYT `Xok\`XZT>x`wRUv\ZwRhJ[ÑTWXbXZhdX hdX \ZYT OWÊdÊdÊdÊdÊQ\Z)kZVUWUvT>kZkZwzdTwR[`VY\k%~hJchUWUWV`XZXZT>xEP³áQ~czTPtsUvhJMaQXZT>k Tv\T>[`xYT>x±G³¶ lLfwz\Z ad[`x fwz\ZYhJV`\cRT>aQXb[`wR[Y¯XbaQ\TxYT>UWaYn\h\Xbadx`wz\ZwzhJ[adci±G³¶ ad[`x ad[Á±´³¶ QaQXbwRad[]\fwz\ZïxYT>UWa>²hdi\ZYTwR[\TWXo[àdcGUvT>cRcÄk\ZaQ\T lLfwz\ZïakT>cz}XbT>UWVYXZXZT>[]\fT>wzJ]\ OnoP VYXXZT>kbV`cz\Zk LhdX\Xbadxwz\ZwzhJ[àdce±G³¶fwR\Z[YTW\gfihdXZjadUv\ZwRäQ\ZwzhJ[¯XZT>kTW\ZklL~ad[Tv\TWXb[àdc\T>adUbYTWXvnaQ\kT>VYT>[ÙvT%T>[`x`k aQXZTqkbcRwzJ]\Zcz ~TW\\TWX\Zad[\ZYhJkTq~MadkT>xhJ[a xßw @TWXbT>[\\hdhJczhdd @l ÄhUoYXZT>wz\TWmX *°Ub wRx`VY~TWX %O>¤d¤JdnoP \TWXb[adc`XZT>kTW\Zklm[YhJ[YágUvhJ[\ZwR[VàdcUWadkTBnadcRczhBf"±G³¶8\h [`xTvYUvT>cRczT>[]\khJcRVY\ZwzhJ[k}wR[B¢ °hdE\ZYT\XowRadcR&k % adUWUvhdXbxwR[Y2\h2hJV`XkZ\XbwR[YdT>[]\\T>kZ\ZwR[YµUvXbwz\TWXbwzhJ[eP³Xbadx`wz\ZwRhJ[àdc±G³¶madwRcR&k %YhBfTWdTWX %wR[\Z`TUvhJ[]\ZwR[Vàdc UWadkTdP)ÂÃ[\TWXo[àdc_k\ZaQ\TïxYT>UWa>Áx`h]T>kÝ[Yhd\ YT>cz"%VÙoÁT>wz\Z`TWX²lLfiT\XbwzT>xÁäQXbwRhJV`k kT>cz©áäXZT>UWVYXbXZT>[\ fT>wzJ]\ QadcRVYT>kad[`xXZTWhdXZ\hJ[`czµ\ZYT%~@T>kZ\XZT>kZVcz\onoP_Ä\T>[`xYT>x²±´³¶Åfwz\ZLhdXZdTW\JaQ\T>&k %EYhfiTWdTWX %EUWad[ïkhJczdT \ZYTqUvhJ[]\ZwR[VàdcE`Xbhd~czT>P UvhJ[\ZwR[VàdcRcRïx`T>UvXZT>adkZwR[YïczT>aQXb[wR[YXbaQ\TµczT>xÑ\h¯TWdT>[ ~TW\\TWXXZT>kZVcz\Zkq~V`\%àdxå[Yh²/T @T>Uv\hJ[å\ZYT hd\ZYTWXadczdhdXowz\Z` kWPâßw TWXZT>[]\\hdhJczhdJwRT>k a>¯`XZhwRxYT~TW\\TWXXZT>kZV`cz\Zk %M\hJh 9"fiT x`wRx²[Yhd\aQ\\T>`\\h hd`\Zwy wzæWT\hdhJczhdddP ad[\ZYT[YTW\gfihdXZjczT>aQXb[\hÝXbT>UvhdJ[`wzæWT%aQ`XZhd`XbwyaQ\Tq\ZwRT>k©hdXhdT>[`wR[Y ÜWczhJkbwR[Y wz\ZkJaQ\T>kfwz\ZYhJVY\ V`kbwR[Y2\ZYTwR[YLhdXb aQ\ZwzhJ[ïUvhJ[]dTWdT>xº~\ZYT aQXZjdTWX&k&~hJcRk ø ad[`x õ Ç³h\T>k\q\ZwRkfTXZTWMcRadUvT>xadcRc S_uËkZVY~[`TW\Zkhd´\Z`Tq\gT Þ Þ Þ Þ ~ Þ Þ P ³`wRk aQjdT>k\ZYTÄ\ZadkjhdXZT}x'w UWV`cz\adke\Z`T[YTW\I[YhBfº[YTWT>x`kE\hjdTWTW%\XbadUbjhdkZT>]VYT>[UvT>kEhdY[]V` TWXZhJV`k hd\T>[]\ZwRadcRcz UvhJ[`LV`kbwR[Y ¦ ad[x ¾ k&~hJcRkWPÇVY\_±G³¶Éfwz\ZµLhdXZdTW\_JaQ\T>k%lmkbadTq\hdhJczhdd`nifiadkk\ZwRcRc aQ~cRT\yh [`x2TWXZLT>Uv\khJcRVY\ZwRhJ[`&k %Yadcz\ZYhJVYJczT>kZkLXZT>]V`T>[\ZczlmkT>VYT>[]\ZwRadGc õåxYT>UWa>µfadk[Yhd\aQ`cRwzT>x@noP

ih

V

j

km ?l oq n ps +r oqpu ?t oqp.k> +oln

k> +os ln pkÄTv\T>[`xYT>x ±´³¶ khJczdT_\Z`T\Zadkj%hJ[ f`wRUo\Xbadx`wz\ZwzhJ[adc±G³¶ LadwycRkÇT>Uv\ZwzhJ[ÝPROQPRO_adczXZT>adxY T>[]\ZwzhJ[YT>x±G³¶{ k`XZhd~cRT> hdGV`[ÙvhJ[]\XZhJcRcRT>xµdXZhf_\Zhd\ZYTqwy[\TWXb[adc@kZ\ZaQ\T>kWP®GwzJVYXbTP£¢ÝkZYhfk\ZYT TWdhJcRV`\ZwzhJ[hdG\ZYTwR[\TWXo[àdcek\ZaQ\T>kcyx`VYXbwR[`\ZYTq`XZT>kZT>[\ZaQ\ZwzhJ[¯hdÄa \T>k\k\XZT>adP_³YT&wR[]\TWXb[àdcekZ\ZaQ\T>k \T>[`x2\hdXZhBf cRwR[`T>aQXbczdPG\\ZYTk\ZaQXZ\Zkhd´kZVÙWUvT>kZkZwRdTS_uËk\XbwR[`J&k %]\Z`Tq[YTW\gfihdXZjÝwykiwy[2ad[wy[ÙvXZT>adkZwR[`Jcz adUv\ZwzdTk\ZaQ\TdP\khJT&hJwR[]\lm`TWXZT%aQL\TWXO>ÝkbVÙWUvT>kZkZwzdTk\XbwR[`Jkon %M\Z`T%`wzJ¯czTWdT>chd}k\ZaQ\TadUv\ZwzQaQ\ZwzhJ[ czT>adxke\hkZaQ\ZVYXbaQ\ZwzhJ[%hd`\ZYTiUvT>cRc]hJVY\VY\Z&k %dad[`x&TWXZ©hdXo ad[ÙvT}~XZT>aQjk´xYhBf[EPGÄ\T>[`xYT>x±´³{ ¶ %YhBfTWdTW(X % czT>aQXo[`k´\h&V`kT\ZYTLhdXZdTW\JaQ\T>k}©hdXXbT>kTW\\ZwR[Yqwz\ZkÄkZ\ZaQ\T_fYT>[[YT>UvT>kbkZaQXZdPÄ®GwzJVYXZTPtr lL\hdÝàdczonkZYhBfka \ú]wyUWadcwy[\TWXb[adckZ\ZaQ\TTWdhJcRVY\ZwzhJ[aQL\TWX}czT>aQXb[`wy[YYPeT_kTWT\ZàQ\G\ZYT\Z`wRXbx% T>hdXZ&~MczhUbjqXZT>kTW\Zk}wz\ZkGUvT>cRcRk wR[¯k[ÙbYXbhJ[fwz\Z\ZYTk\ZaQXZ\Zkhd}S_u k\XbwR[YJklL\ZYTdTWXZ\ZwRUWadcGcRwy[YT>kwR[¯®GwzJVYXZTPtr2wR[`x`wRUWaQ\T%\ZYT\Z`wzXbx k%~@hJcGhd_ak\XbwR[Y]noPµ³_YT wy[\TWXb[adck\ZaQ\T>k%hJkZUWwRcRcRaQ\TÝaQXZhJV`[`xæWTWXbUvhJ T>k_UWczT>aQX_YhBfç\ZYTXZT>czTWQad[\wR[`©hdXb aQ\ZwzhJ[2dTW\Zk k\hdXbT>xE\Z`T kT>UvhJ[`xUvT>cRcGhdi\ZYT\Z`wzXbx²~cRhUbj¯k\Zakq[`TWJaQ\ZwzdT f`wyczT&\ZYT kZ%~hJc ¾ àdk\h~TkZ\hdXZT>ex % f`TWXZT>adk%a ¦ wRk%XZTW`XZT>kZT>[\T>xÑ~] a¯hJkZwz\ZwzdTÝädcyVYTdP³_YTÝ\Z`wzXbxº~czhUZ j k&LhdXZdTW\JaQ\TadUv\ZwzQaQ\ZwzhJ[`kaQXZT cRhd\\T>xÝwR[2®GwzJVYXZTPtrµlL~@hd\\hJ2noP¶hJkZ\hde\Z`T\ZwR T\ZYTWÝaQXZTT>VàdcM\hOQP 1Ê %\ZV`kczTW\\ZwR[`&\ZYTT>hdXZ UvT>cRcykXZTW\ZadwR[\ZYT>wzXwR[]\TWXb[àdceädcRV`T>kWPi\\ZYTqT>[xhdÄad[¯S_u/k\Xbwy[Y \ZYTqLhdXZdTW\JaQ\>T kadUv\ZwzäQ\ZwRhJ[dhT>k \h æWTWXbkTW\\ZwR[` UvT>cRcEk\ZaQ\T>k\h æWTWXZhYP [àdczæ>wR[YÝ\ZYT~@T>a>wzhdXhd\ZYThd\ZYTWXqT>hdXZ~czhUZj&k %MfiT [`x\ZàQ\hJ[`cz\ZYT\Z`wzXoxwRkx`wzXZT>Uv\ZcR XZT>kZ@hJ[kZwz~czTLhdX~`XbwRx`JwR[YSu kczhJ[YdT>k\\ZwRTµcRaQÁlLf`wRUoÑwyk%kZCV UWwzT>[]\adkhJ[YsT âZV`k\~wz\àdk\h²~T k\hdXbT>xMnoP®GwzJVYXZTqPt{czhd\ZkiQadcRVYT>kad[àdcRhddhJV`k\h\ZYhJkTwR[µ®GwzJVYXZTqPtrLhdX\ZYmT `Xbk\T>hdXb ~czhUZj2ad[`x wz\Zfk `Xbk\UvT>cRcãP³_YnT `XbkZ\~czhUZ j k_UvT>cRcead[`x©hdXZdTW\JaQ\T%kZYhBf kZ`hdXZ\Ãáä\TWXb~T>àwRhdXhJ[`czlm[YT>UvT>kZkZaQXZµLhdX `XbT>x`wRUv\ZwR[Y\Z`T2[V`TWXZhJVk%kZ`hdXZ\&\ZwR T2cRaQ²TWdT>[]\Zkhd_\ZYTµS_TW~TWX%dXoad aQX noPï³_YT2kZadTµwRk&\XoVYT LhdX adcRc´hd\ZYTWX~MczhUbjkTvYUvTW`\\ZYT\ZwzXbxEPhJ hJ[ï\hadcRcGT>hdXb~czhUZjkwRk\ZàQ\\ZYTWïczT>aQXb[`T>x¯\hXbT>kTW\ \ZYT> kT>czdT>k_wR[ad[aQ``XZhd`XowRaQ\TLadkb`wzhJ[EP Ä

T T ?A"!%$& ! =#' Vþ"&>By-/Â 7 'cok 6(3G W "C+ \T>[`x`T>xq±´³¶°khJcRdT>kE\ZYTiS_u `XZhd~MczT>)f`wRcRT´\Xoadx`wz\ZwzhJ[àdcd±G³¶ xYhT>ke[Yhd\WPGÇiV`\IUWad[q\Xbadxwz\ZwzhJ[àdc ±G³¶,khJczdT`Xbhd~czT> kf`wRUo Tv\T>[`xYT>x±G³¶ÈUWad[[Yhd\Ç T\T>kZ\T>xµTv\T>[`x`T>x±´³¶ÈhJ[ÝhJ[`Thd\ZYT ,x

×]êëìÚGÛ_íþúTïÿIÛëíÙiðÃÙÄ¥ Ú_ØvÅØíc¥ÛÚ³Æ×ØÙÚ]ðòÙò¦ëÿ{ìíÛñð>×_Ú$ð>ØÙ¸Rð>Ú]êáÿ´ÜeÚ# -12- -20- -11- -15- -11--10- -15- -14- -9- -19- -10- -9- -9-

Internal Cell State

sd{

20 10 0 3.Block, 1.Cell 3.Block, 2.Cell

-10

Forget Gate Activ.

680

T

P

P

T

T P

P

T T

T

T T 850

T

P

P

T

T P

P

T T

T

T T 850

1 0.5 0 680

Symbol

®GwzJVYXbTPtr³hdeÂÃ[]\TWXb[àdcGk\ZaQ\T>k2yhd\ZYT\gfihUvT>cRcRkhd}\Z`TkZT>czyáäXbT>kTW\\ZwR[Yµ\Z`wzXox¯T> hdXZ~cRhUbjwR[ ad[2Tv\T>[`xYT>x±Ý³¶É[YTW\úfhdXbjÝx`VYXowR[Y%a\T>k\_kZ\XZT>adó`XZT>kZT>[\ZaQ\ZwzhJ[eP³_YT `JVYXZTkZYhBfkO¨Ê kZVÙWUvT>kbkZwzdT k%~@hJcyk\ZaQjdT>[©XbhJó\ZYTqczhJ[YdTWX_kZT>]VYT>[UvTq`XZT>kT>[]\T>x\h a[`TW\gfihdXZjµ\ZàQ\_czT>aQXb[YT>xµ\Z`T}S_uPM\ZaQXZ\Zk hdI[`TWf-SuàkZ\XbwR[YJkiaQXZTwR[`x`wRUWaQ\T>xÝ~ dTWXZ\ZwRUWadccRwR[`T>k}cRaQ~T>czT>xÝ~] \ZYTkZ%~hJcRkl ¾ hdX ¦ nÄ\h~TkZ\hdXZT>x V`[]\ZwRc\ZYT[YTv\kZ\XbwR[Yµk\ZaQXZ\WPÇhd\\hJkbwR%Vcz\Zad[YTWhJV`kLhdXZdTW\JaQ\T adUv\ZwzQaQ\ZwzhJ[`khd\ZYTkZadTT>hdXZ ~cRhUbjMP hJkZ\&x`w' UWVcz\[YhJ[`cRwR[`T>aQXqczhJ[Y\ZwR T cRaQ\ZadkjkqTWdTWXkhJczdT>xº~]ad[ÑS^^&Þ^hJwRkZï³T>hdXbadcXbxYTWXbß lm^³qnlL\Zadkj{¨~¯\ZaQjdT>[LXZhJ ÄhUoYXZT>wz\TWmX *°Ub wRx`VY~TWXO>¤d¤JÝnoP Y²¦`Ò S ³_YTdhJadcMwyk\h%UWcyadkZkZwzLkZT>]VYT>[UvT>k}hdeczhUWadcRczXZTW`XbT>kT>[\T>x2k&~hJcRk>PádUbµkT>VYT>[ÙvTkZ\ZaQXZ\Zk fwR\Zad{ [ ã~%´T>[`x`k&fwz\Za lL\ZYTºÞÃ\XbwzddTWXk&~hJcRßn %ád[`x hd\Z`TWXZfwRkT UvhJ[kZwRk\Zk&hdiXoad[`xYhJ cz²UbYhJkZT>[ k%~@hJcyk©XbhJ \ZYT²kTWk\ çD:Ö hÖãhÖ ê TvUvTW`\2LhdX \ZYXZTWT¯T>czT>T>[]\ZkÝaQ\ÝhJkZwz\ZwzhJ[`k Ô Ö3Ô ad[`x Ô µ\ZàQ\µaQXZT T>wz\Z`TWX hdX ëlm®GwzJVYXZT P dnoP³_YTkT>VYT>[ÙvT czT>[Yd\ZïwRkXbad[xYhJ czUoYhJkT>[ï~TW\gfiä TWT>æ [ÁOWÊdÊad[`x OdOW1Ê %¦Ô ä wRkXoad[`xYhJ czÑUbYhJkT>[Á~@TW\úfTWT>[ OWÊ ad[`xÁsQ1Ê %"Ô wykXbad[`xYhJ cRºUbYhJkZT>[~TW\gfiTWT>["dad[`x ¢d %ad[`x Ô µwRk Xbad[xYhJ czUoYhJkT>[ï~TW\gfiTWT>[ {d{µad[`x Q{P&³_YTWXbæ TaQXZT ¡2kT>VYT>[ÙvT UWcRadkbkT>k Ö Ö¨$Ö "Ö !Ö oÖ Ö f`wyUb xYTWT>[`xÁhJ[\ZYT²\T>hdXbadchdXbxYTWXÝhdq\ZYT ²kµad[`x kWP ³YTXbVczT>k aQXZTlL\T>hdXbadchdXbxYTWX UWcyadkZkono RÖ gÖ § gÖ RÖ § RÖ _Ö ¨"§ RÖ _Ö -§ Ö gÖ ~§ Ö gÖ o§ _Ö Ö § Ö _Ö P³áQXZdTW\ÝkbwzJ[àdcRkhUWUWVYX hJ[`cz aQ\\ZYTT>[x hdakZT>]VYT>[UvTdP³_YT `Xbhd~czT>w k wR[wR adcM\ZwRTqcRaQ kZwRæWTqwRk_¡QÊl noP}®YhdXbdTW\\ZwR[YÝwykhJ[`czÝaQXbmV`cMadkadcRc@XZT>czTWQad[\wR[YLhdXb aQ\ZwzhJ[ àdk_\h ~TjdTW`\V`[]\ZwRc@\ZYTqT>[`xµhdÄa kZT>]VYT>[Uv>T %aQ©\TWXf`wRUoµ\ZYTq[YTW\úfhdXZjwykXZT>kTW\ad[]]fa>dP TV`kZTÝ\ZYT[YTW\úfhdXbjº\hdhJczhddÑx`T>kZUvXbwz~T>xºwy[åkZT>Uv\ZwzhJ[PtPts²fwz\Z ¡¯wy[YVY\ad[`x¡hJVY\MVY\V[`wz\ZkWP kZwR[YåaÑcyaQXZdT¯~MwRadk²lãrP Ên LhdXµ\ZYT¯LhdXZdTW\µJaQ\T>& k %Tv\T>[xYT>x)±´³¶ kZhJczdT>x\Z`T²\ZadkZjadkµ]V`wyUZjczÁadk \Xbadxwz\ZwzhJ[àdc±G³¶plLXZT>UWadcyc}\ZàQ\a²`wzJ LhdXZdTW\JaQ\Tµ~wRadk& aQjdT>k%Tv\T>[`xYT>x ±G³¶(xYTWdT>[YTWXbaQ\TwR[]\h \Xbadxwz\ZwzhJ[àdc´±´³¶²noP kbwR[Yµa2 hxYTWXoaQ\T~MwRadkcRwzjdT\ZYT%hJ[`TVkT>x²©hdX%S_u,lÃOQP Ên %eTv\T>[xYT>x±´³¶ _-

0y

hdXZÝ~MczhUbjMP 0

B

33 - 43

10 - 20

noise a.b,c,d

{XY}

noise a.b,c,d

{XY}

66 - 76 noise a.b,c,d

{XY}

100-110 noise a.b,c,d

E

recurrent connection for continual version

®GwzJVYXZT&P ]i^³àad[`x¯^³à\ZadkjkWPiTWT\Tv\©hdXx`TW\ZadwRcRkWP \hhdj aQ~hJVY\i\ZYXZTWT\ZwRT>kiczhJ[YdTWXhJ[µa>dTWXbaQdT>%`~MVY\x`wyx khJczdT\ZYT`XZhd~MczT>PG³_YTkZczhfiTWXczT>aQXb[`wy[Y&kTWT>x XZT>kbV`cz\ZkLXZhJó\ZYTq[YTW\à>wR[Y\h cRT>aQXb[µ\h XZT>T>%~TWX_TWdTWXZ\Z`wR[Y ad[x[Yhd\_\h LhdXZdTW\WP uT>[`TWXbadcRczkT>aQjwR[Y[UvhJV`[\TWXbT>xa `Xbhd~czT> \ZàQ\±´³¶ kZhJczdT>kf`wRcRTqTv\T>[xYT>x ±G³¶Éx`h]T>k_[`hd\WP ½ Y¯¦ Ò S ^hBf fiTq\ZaQjdT&\Z`Tq[YTv\hd~]wzhJV`kk\TW²ad[`xµ\Xbad[`kZ©hdXb\ZYTq^³ËwR[]\h a UvhJ[\Zwy[]Vàdce`XZhd~Yá czT> \ZaQ\xYhT>kXZT>V`wzXZTLhdXZdTW\\ZwR[YâV`kZ\_adkwR[kT>Uv\ZwRhJ[¯PtPR5O %@~]2dT>[YTWXbaQ\ZwR[` UvhJ[\Zwy[]VàdcIwR[YVY\k\XbT>ad k UvhJ[`kbwRk\ZwR[Y&hdUvhJ[ÙWaQ\T>[àQ\T>x^³ kT>VYT>[ÙvT>kqlm®GwzJVYXbTP dnoP}|}XZhUvT>kZkZwR[YkZVÙo2kZ\XZT>ad kifwz\ZYhJV`\wR[]\TWXá T>xwRaQ\TXZT>kTW\Z&k %`\Z`T[`TW\gfihdXZjwRkXZT>V`wzXZT>xÝ\h czT>aQXb[µ\h UWcyadkZkZwzL ^³àkT>VYT>[ÙvT>k_wR[ad[µhJ[cRwR[YTLadkb`wzhJ[EP adUbwR[YV`\ek\XZT>adËwRk´k\hd`T>x&hJ[UvT\ZYT[YTW\úfhdXZj% aQjdT>kÄad[%wy[ÙvhdXZXZT>Uv\GUWcRadkZkZ¤w UWaQ\ZwzhJ[qhdXOWÊdÊkZVÙWUvT>kbkZwzdT ^³ËkT>VYT>[ÙvT>kàdT&~TWT>[UWcRadkbkZ'w `T>xUvhdXZXbT>Uv\ZczdP±eT>aQXb[`wy[Y ad[`x\T>kZ\ZwR[YÝadcR\TWXb[àQ\>T §\Z`T&TWXZLhdXb ad[ÙvT T>adkbVYXZTwyk}\ZYTa>dTWXoaQdT&kZwzæWThd}OWÊ%\T>k\k\XZT>ad &k %YT>adkbVYXZT>xÝ~] \Z`T[V`%~@TWXhdI\ZYT>wRX^³ kT>]V`T>[ÙvT>k lLT>adUoåUvhJ[]\ZadwR[`wy[Y~TW\gfiTWT>["OWÊdÊ²ad[`xÁOdOWÊ²wR[YMVY\qk&~hJcRkonoPµ³XbadwR[`wR[`µwyk&k\hd`T>xÑaQ©\TWXaQ\%hJkZ\ÝOWÊ X

sd¡

×]êëìÚGÛ_íþúTïÿIÛëíÙiðÃÙÄ¥ Ú_ØvÅØíc¥ÛÚ³Æ×ØÙÚ]ðòÙò¦ëÿ{ìíÛñð>×_Ú$ð>ØÙ¸Rð>Ú]êáÿ´ÜeÚ# cRdhdXbwz\Z` |´TWXZLT>Uv\hJcäP |áQXb\ZwRadchJcãP ³IXbadxwz\ZwzhJ[àdcE±G³¶ Ê l©áòn WO ÊdÊ L¢k s¨¢²Ãl O>¡QÊddn Q{ ÃO>sd×ts ±´³¶ÉfwR\Z®YhdXZdTW\uaQ\T>k ad[`xkT>]V`T>[\ZwRad¦c õÑx`T>UWa> Jãl sdsd{dr¨¢]n {d ÃOdO5×t¡ V

V

LZ

^\

_Z

^\

_Z

^\

³áQ~czTPtiihJ[]\ZwR[]V`hJV`k^hJwRkÝ³IT>hdXbadceXoxYTWXlä^³&noYihJcyV` [åÞ |´TWXZ©T>Uv\hJcäP ß|TWXoUvT>[\ZaQdTqhd ÞÃTWXZLT>Uv\ZßkhJcRVY\ZwzhJ[klmUvhdXbXZT>Uv\_UWcRadkZkZ¤w UWaQ\ZwzhJ[2hdOWÊdÊdÊkZVUWUvT>kZkZwzdTq^³ kT>VYT>[ÙvT>kwR[ïOWÊ\T>k\_k\XZT>ad kon § wR[aQXZT>[]\ZYT>kT>kW[]V&~TWXhd\XbadwR[`wy[YÝk\XbT>ad k`XZT>kT>[]\T>xEP&ihJcRV` [Þ |GaQXZ\ZwRadchJcäP ß_TWXbUvT>[\ZaQdThd khJcyVY\ZwzhJ[`kád[`xa>dTWXbaQdTk\XZT>ad/kZwRæWTlLQadcRVYTwR[ad[YJVcRaQXG~`XbadUZjdTW\Zk n ÕOWÊdÊPÄcyc]XbT>kZV`cz\ZkGaQXZTa>dTWXoaQdT>k}hBdTWX OWÊdÊwR[xYTWT>[`xYT>[]\Zcz%\XoadwR[YT>xµ[YTW\gfihdXZjkWPi\Z`TWXadcRdhdXbwz\Z` klmÇ|}³n³ %YS´³S±ïTW\ZUQP naQXbT[`hd\wy[ÙWcRV`xYT>xÝwR[ \ZYTqUvhJ aQXbwRkhJP[ %Y~@T>UWadVkT\ZYTWµLadwycTWdT>[hJ[\ZYT&T>adkZwzTWX %`[YhJ[YágUvhJ[\ZwR[Vàdce^³P bV

bV

c

\Xbadwy[`wR[Yk\XZT>ad k>P ö2¸ÔFª©?Ô S ³áQ~czTPt2kZV` aQXbwzæWT>k\ZYTXZT>kZV`cz\Zk>PThd~kZTWXZdT\ZàQ\\Xbadx`wz\ZwzhJ[adcI±Ý³¶ÌaQJadwR[²madwRcRk \hïkhJczdTÝ\ZYT2UvhJ[]\ZwR[Vàdc`XZhd~czT>PÄ\T>[`xYT>xº±G³¶(fwR\Z LhdXZdTW\%JaQ\T>k%}YhfiTWdTWX%}UWad[åkhJczdTµwz\WP UvhJ[]\ZwR[VàdcRczxYT>UvXbT>adkZwR[YczT>aQXb[wR[Y XoaQ\TAl õxYT>UWawy[Yµ~]²aÝLXbadUv\ZwzhJ[ïhdiÊPt¤µaQ©\TWXT>adUo ^³ÆkT>VYT>[ÙvT wR[akZ\XZT>adÝnczT>adxk\h kZcRwRJ\ZczÝ~TW\\TWX_XZT>kZV`cR\Zk~VY\wRk_[`hd\_[YT>UvT>kZkZaQXZdP

N

@ÿ ihJ[]\ZwR[VàdcwR[`VY\qk\XZT>ad k%dT>[`TWXbadcRczïXZT>V`wzXZT hUWUWadkZwRhJ[àdc}XbT>kTW\Zk&hd\Z`TÝk\XZT>adáä`XZhUvT>kZkZwR[Y²[YTW\gfihdXZj@P |GaQXZ\ZwRadcXZT>kTW\Zk2aQXZT¯adcykhÑxYT>kbwzXbaQ~czTLhdX \ZadkZjk2fwz\Z`wzTWXbaQXbUo`wRUWadc_x`T>UvhJhJkZwz\ZwzhJ[EPç®YhdXµwR[`kZ\Zad[Ùv>T %XZTvá hUWUWVYXZXbwy[YÝkZV`~`\ZadkjkkZYhJV`cRx¯~TkhJczdT>xï~\ZYTkZadT [`TW\gfihdXZj² hx`Vcz>T %f`wRUo²kbYhJV`cRx¯~TXbT>kTW\hJ[ÙvT \ZYT kbVY~`\Zadkj)wRkkhJcRdT>xEP8wR[UvT²\ú]MwRUWadcXZT>adc áäfihdXbcRx wR[YVY\µk\XZT>ad kaQXZTº[`hd[ \ 29 0?A750|?x`T>UvhJhJkT>x wR[]\h \Xbadwy[`wR[Y kZV`~kT>VYT>[ÙvT>&k %ad[xkZwR[ÙvT&\gwRUWadcIkZT>]VYT>[]\ZwRadcI\ZadkjkaQXZT[Yhdo\ 2s9:0|?@7>0?Ix`T>UvhJhJkT>xwR[]\h aQ`XZhd`XbwRaQ\TkZVY~`XZhd~czT> k %S^^kkbYhJV`cRx ~@TaQ~czT\Ãh 4',250|B \hadUb`wzTWdTqaQ`XZhd`XbwyaQ\TxYT>UvhJhJkZwR\ZwzhJ[`kWP ³_`T_[YhdT>cM©hdXbdTW\}JaQ\T>k[àQ\ZVYXbadcycz%TWXb wz\±´³¶\hczT>aQXb[ cRhUWadc`kZT>czyáäXbT>kTW\Zk}hdET> hdXZUvhJ[\T>[]\Zki\ZàQ\ àdTq~T>UvhJT&wzXbXZT>czTWQad[\WP ±G³¶,Tv\T>[`xYT>xµfwz\Z2©hdXZdTW\JaQ\T>kYhJcRx`k`XZhJ wRkT©hdX_ad[]ÝkT>VYT>[]\ZwRadc`XZhUvT>kZkZwy[Y%\ZadkZjÝwR[2f`wyUb fiT²kbV`kT>Uv\Ý\ZaQ\µa `wRTWXbaQXbUbwRUWadcx`T>UvhJhJkZwz\ZwzhJ[- a>"Tvwyk&\ %~V`\2xYh [Yhd\µj[YhfÈwR[-adxYQad[ÙvTïfàQ\ \Z`wyk&xYT>UvhJhJkZwR\ZwzhJ[ºwRk>P³_YT2hx`T>c}àdk%~TWT>[ÑkZVUWUvT>kZkmV`cRczïaQ`cRwRT>x\hï\ZYT \Zadkjºhdx`wRkZUvXowR wR[àQ\Zwy[Y cRad[`JVàQdT>k´©XZhJàdTWXZ%cRwR wz\T>x&`XZhJkhx`wRU}wR[YLhdXb aQ\ZwzhJ[µläV` wR[`&k %duTWXb&k %T*çUo` wRx`VY~TWX %O>¤d¤d¤JnGfYTWXZT \ZYTWXbTqwRk_[Yh UWczT>aQXcRwR[YJV`wyk\ZwRU\ZYTWhdXZ2hdGwzTWXbaQXbUo`wRUWadcEk\XbVUv\ZVYXZTdP Æ

¨

©«ªX¬®~¯Z°

Ä6¿

³Û¸

²

5

6 ¶

À

7Æ6¹¸q·

7

6¥³)7¸

È6 ³

4(5 :

4(5

¹¸

3 4(5(6¹¸5 ¶

:

Eÿ ¶¯ad[\ú]MwRUWadcIXbT>adcefihdXbcRxkZT>]VYT>[UvT&`XZhUvT>kZkZwy[Y \Zadkjkwy[dhJczdTUvhJ[\Zwy[]VàdcwR[YV`\_k\XZT>ad k&%x`wRkZ\Xbwz~VY\T>x wR[`VY\_XZTWXZT>kT>[]\ZaQ\ZwzhJ[`&k %EUvhJ[]\ZwR[]V`hJV`kÃáäQadcRVYT>x²\ZaQXZdTW\Zkqad[`xïwR[`VY\Zkad[x²wy[\TWXb[adck\ZaQ\T>k %ead[`xïczhJ[Y2\ZwRT cRaQJk´~TW\gfiTWT>[XZT>cRTWäd[]\GTWdT>[\ZkWPhfiT_xYT>kZwzJ[`T>x%kTWdTWXoadcàQXZ\Z'w UWwyadc][YhJ[cRwR[YT>aQX\Zadkjk\ZàQ\ÄUvhJ&~wy[YT\ZYT>kT madUv\hdXbkWP âVYT\hwR\ZkaQXbUo`wz\T>Uv\ZVYXbT\Xbadxwz\ZwzhJ[àdc`±G³¶,wRk}fT>cyckZV`wR\T>x LhdX\ZadkjkwR[]dhJczwy[Y&adx`x`wR\ZwzhJe[ %kZV`~`\XbadU á \ZwzhJ[µad[`xµwR[]\TWdXbaQ\ZwzhJ[@l ÄhUoYXZT>wz\TWiX * YUb` wyx`]V`~@TW(X %O>¤d¤JdnoPVUb2hdTWXbaQ\ZwzhJ[`kaQXZTT>kZkT>[]\ZwRadc@LhdX_ ad[ XZT>adczáäfhdXbcyx\ZadkjkWPÇVY\ad[Yhd\ZYTWXT>kZkT>[]\ZwRadcÄaQXbwR\Z`TW\ZwRU%hdTWXbaQ\ZwzhJe[ %e[àd T>czN 8.14û;@?Ð9:4û?@K2>;@?@7>BC%´x`h]T>khJkT `Xbhd~czT> kWP ®`hdXZdTW\&JaQ\T>&k %ÄYhBfTWdTW(X %´hdXowzJwR[àdcRcRwR[]\XZhxVÙvT>xï\hXZT>czT>adkZTÝwzXZXbT>czTWäd[]\T>hdXZUvhJ[]\T>[\Zk % dXZT>aQ\ZcR2wR `XZhdT%±´³{ ¶ kTWXZLhdXb ad[ÙvThJ[\ZadkZjkwR[dhJcRwR[`V`cz\ZwRcRwRUWaQ\ZwzhJP[ %adkfwRcRc@~TqkTWT>[~T>czhf&P -ÆÅ

m e ~$

eÿ TLhUWVkhJ[ï\Zadkjk&wR[dhJcRwR[`2aQXbwR\Z`TW\ZwRUhdTWXbaQ\ZwzhJ[`khJ[ wy[YVY\k\XZT>ad k\ZaQ\qkhµLaQX&a>dT~TWT>[adxá xYXbT>kZkT>xÁhJ[czÁwR[[`hJ[ágUvhJ[\Zwy[]VàdckTW\\ZwR[`Jk²lm³_kbV`[YI * ihd\\XZT>cycA%qO>¤d¡d¤d§ÄhUb`XZT>wz\TWXr*YUb` wyx`]V`~@TWX(% O>¤d¤JdnoP ÷µ¸=¸AB§ªMÔ>¸]?WðoF3Ó S T©TWT>x&\ZYT}[YTW\UvhJ[\ZwR[Vàdc]k\XZT>ad kIhd¢QágxwRT>[`kZwRhJ[àdcdwR[YV`\@dT>Uv\hdXokIdT>[YTWXoaQ\T>x wR[ad[&hJ[`cRwR[`TÄLadkZwzhJ[EPeTxY&T [Y!T Ô Õ)Ê&lmkZ\XZT>ad kZ\ZaQXZ\onEad[`sx Ô Õ¥Ô ® c®ÝJl ÞqO>n LhdGX ØRÕË5O Öbds Ö&×&×&×V% f`TWXZT æjç1Ê Ö>5O Ö&×&×&×$Ö ê wRkUb`hJkT>[ïXbad[`xYhJ cz> %ead[`xïwR[]\TWdTWX °wRk ¯\ZYä T wR[wR adce\ZwyTcRaQYP³YT `Xbk\ UvhJhJ[YT>[]\hd_T>adUbwR[`VY\qdT>Uv\hdX wRka²Xbad[`xYhJ[V`&~TWX%©XZhJ\ZYTµwR[]\TWXZQadc «ßÞq5O Ö®O ¬äP ³_YTÝkZT>UvhJ[`x ad[`x2\Z`wRXbxÝkZTWXZdTqadkÞ aQXZjdTWXbkbß\ZYTWÝaQXbTqadczfiakÊP Ê TvUvTW`\_aQ\\ZwR T>qk Ô Ï LhdX Ñ«ÕË5O Öbds Ö&×&×&×V%fYT>[ T>wz\Z`TWX\ZYTqkT>UvhJ[`xUvhJ @hJ[`T>[\wykOQP Êfwz\Zµ`Xbhd~aQ~wRcRwR\g G%YhdX_\Z`T\Z`wzXbxæ wRk¯OQP ä Ê fwz\Zµ`XZhd~MaQ~wRcRwz\ú1O Þ IP ³_`TLhJVYXZ\ZµUvhJhJ[YT>[]\iwRkiadczfa>kÊ&TvYUvTW`\_aQ\\ZwRT>k Ô æ Ï fYT>[Ý\ZaQXZdTW\Zk_aQXZTJwzdT>[ad[`x2wz\ZkadUv\ZwzQaQ\ZwzhJ[ wRkOQP ÊPq³_YT%\ZaQXZdTW\qaQm\ Ô wRkÊPÂò\ZYTsQ[`xïUvhJhJ[YT>[]\fadkadUv\ZwzdTaQm\ Ô æ Ï ¯ä \Z`T>[¯\Z`T%\ZaQXZdTW\qaQ\ Ô æ Ï wRk\Z`T kZV`Èhdi\ZYT `XbTWwzhJVk\ZaQXZdTW\%aQ\ Ô æ Ï ¯ æ ad[xï\ZYTºÞ aQXZjdT>xwß `Xbk\qwy[YVY\UvhJhJ[YT>[]\&aQ\ Ô æ Ï ¯ä P \Z`TWXZfwRkTwz\_wyk\ZYT`XZhx`VÙv\hd´\ZYT>kTq\úfhÝädcyVYT>kWP sd¤ -ÆJø

RTS

i

_p

f

X

.

5

f

×]êëìÚGÛ_íz{dï

random value

marker

QÊ

>Ú )ÛÚ$ðW×çØì}Û_íëÚ$ð>ØÙÜØÙ ×_ØÙÚ$ðÃÙò¦ëÿNðÃÙìoòÚÜeÚ]íÛë ï Ü

ëífð ]ê

[T-T/5, T] [T, T+T/5] ? 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 + 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 * 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 +1 +/

*

+/*

?

0 -1

... [T, T+T/5] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 ... 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 ... [T-T/5, T]

t1

t2

? t4

t3

time

®GwzJVYXZT¢YPROQ}ÂòcRcyV`k\XbaQ\ZwzhJ[2hdG\ZYTqUvhJ[]\ZwR[Vàdceadx`x`wR\ZwzhJ[ lmad[`xV`cz\ZwzMcRwRUWaQ\ZwzhJ[Mn\ZadkjkWP ÄT>[ÙvT[YhJ[ágwR[wz\ZwRadcY\ZaQXZdTW\Zkx`TW@T>[xhJ[2TWdT>[\Zk\ZàQ\àQ`T>[YT>xÝaQ\icRT>adk\s çk\TWkiaQdhYP^hd\T\ZàQ\ hUWUWVYXZXZT>[UvT>kqhd ÞÃQadcRVYT aQXbjdTWXbkZßad[x \ZaQXZdTW\ZkhJkZUWwRcycRaQ\TdPTWTÝ®GwzJVYXZT ¢YPRO LhdX%ad[åwRcRcRV`kZ\XbaQ\ZwzhJ[²hd\ZYT \Zadkj@P cRck\XZT>ad k}aQXZTk\hd`T>xhJ[ÙvT\ZYTaQ~khJcRV`\TihJV`\VY\´TWXZXZhdXÄTvYUvTWT>x`kÊP ÊQ¢YP³T>k\Äk\XZT>ad kaQXbTadcyhJk\ V`[cRwR wz\T>x lm a¨@PczT>[Yd\Zé ÕÆOWÊdÊdÊ2\ZaQXZdTW\hUWUWVYXZXZT>[ÙvT>k n %@~MVY\\Xbadwy[`wR[Y kZ\XZT>ad kT>[`x²aQ©\TWXaQ\hJk\OWÊ \ZaQXZdTW\ hUWUWVYXbXZT>[ÙvT>kWPå±eT>aQXb[`wy[Yad[`x \T>k\ZwR[Yadcz\TWXo[àQ\TdaQ©\TWX T>adUb\Xbadwy[`wR[Yïk\XZT>adÌfiTµ©XbTWTWæWT\ZYT fiT>wzJ]\Zk_ad[`xLTWT>x OWÊdÊ \T>kZ\k\XZT>ad k>PiVYXTWXZLhdXb ad[ÙvTq T>adkZVYXZTqwRk_\ZYTqa>dTWXbaQdT\T>k\k\XZT>adkZwRæWTdP ¦i§`Ô>ü;¼½%CM=@?BKm=EF§ª´§`D´D´K©?BKLCM3= S ÃÕË5O × Êµlm[Yh%V`cz\ZwzMcRwRUWaQ\ZwzhJ[MnoP XÕçsQÊP³ádkjO_T>kbkT>[\ZwyadcRczXZT>]VwzXZT>k \h jdTWTW²adx`x`wR[`lLhJkZkbwz~cz [`TWJaQ\ZwzdTBnQadcRVYT>k\h \Z`T&adczXZT>adxY2TvYwRk\ZwR[YwR[]\TWXb[àdcEkZ\ZaQ\TdP ¦i§`Ô>ü¥E¼ó½%CM=@?¨KL=EF§ªÝ§`D´D´K©?BKLCM= §=D ÐF´ª©?BKLÓ´ªmKLHJ§Y?BKLC@#= S »Õ 1Ê ×tdr % Õ sQÊP/Âòq\ZYT¨Xbx)wR[YVY\ UvhJhJ[YT>[]\wRkadUv\ZwzdTaQ\ Ô æ Ï ¯ä ad[`x \ZYTOWk\ wRk[YTWJaQ\ZwRdT\ZYT>[ \ZYTcRaQ\\TWXfwRcyc}dTW\XZTWMcRadUvT>x ~]åwz\Zk aQ~kZhJcRVY\TQadcRVYTdP ¦i§`Ô>üÁ·e¼÷µªmKLDGKL=¹º§DDGK©?BKLCM3= S ±IwzjdTq³ádkj5O %`~VY\\ZaQXZdTW\ZkaQ\_\ZwR T>k Ô Ï T>Vàdc\ZYT&kZVóhd\Z`Tq\gfih hJkZ\XZT>UvT>[]\_ aQXZjdT>x2QadcRVYT>kaQ\\ZwRT>"k Ô æ Ï ä ad[`x Ô æ Ï ¯ä lL\ZYmT `Xok\}\ZaQXZæ dTW\_æ aQq\ Ô æ T>VàdcRk\ZYmT `XbkZ\QadcRVYT aQm\ Ô ä noP þÕOWÊP³ádkjïµwRkàQXbxYTWX\Zàd[ï\Zadkj O~T>UWadV`kTwz\XZT>V`wzXZT>kkT>czT>Uv\ZwzdTMaQXZ\ZwRadc´XZT>kTW\Zkhd\ZYT UWVYXbXZT>[\_wy[\TWXb[adcEk\ZaQ\TdP 5f

mf

f

}f

# Vþ.$DÌ@ 6ÑÂM" "1- '!3(Í'I '7.$EG 6B w

_³ `T¢&wR[YV`\GV[`wz\ZkÄaQXZT_mV`cRcR%UvhJ[`[YT>Uv\T>xÝ\h&aqwRx`xYT>[cRadTWXUvhJ[`kbwRk\ZwR[Yhde T>hdXZ~czhUZjkÄfwz\ZO_UvT>cRc T>adUolLXZhJVYJ`cRMczT>kZk~cRhUbjkxYT>UvXbT>adkT>x TWXZLhdXb ad[ÙvTLhdXq±´³¶ ad[`xhdXbT~czhUZjkx`wRx²[Yhd\qwR`XbhdT TWXZLhdXb ad[ÙvTqkbwzJ[`w'UWad[]\Zcz`noPi³YT&UvT>cRcIhJVY\VY\ZkaQXZT&mV`cRcz2UvhJ[`[YT>Uv\T>x²\h \Z`T%UvT>cRcwR[YVY\Zk%Y\hÝadcRcEJaQ\T>k% ad[`x\hµ\ZYT&hJVY\MVY\V`[wz\WPcRceJaQ\T>kad[`xhJVY\VY\V`[`wz\ZkaQXZT~wRadkT>xEPÇiwyadkfiT>wzJ]\Zk\hµwR[áad[`xhJV`\VY\ JaQ\T>kaQXZTwy[`wz\ZwRadcRwRæWT>x~czhUZj]áäfwRkTdq Þq5O × ÊqLhdX}\ZìT `XbkZ\~czhUZ j %:Þds × Ê&©hdX\ZYTkZT>UvhJ[èx %ad[`x kh&©hdXZ\ZePlm³`wRk wRka k\Zad[`xàQXoxwR[`wR\ZwRadcRwzæ>aQ\ZwzhJ[2`XbhUvT>x`V`XZTd~czhUZjk_fwz\Z`wzJ`TWX_~wRadk\T>[x\h dTW\XZT>czT>adkT>x¯cRaQ\TWXxVYXbwR[Y \ZYTÝczT>aQXb[`wR[`2MàdkTdP n®`hdXZdTW\%JaQ\T>kaQXZT wy[`wz\ZwRadcRwRæWT>x¯fwz\Z k TW\XbwyU@hJkbwz\ZwzdTQadcRVYT>kW ®OQP Ê©hdX%\ZYT `Xok\I~cRhUb: j %[®sP ÊLhdXG\ZYTikZT>UvhJ[èx %Jad[`xkhLhdXZ\ZEPG³_YTkb]Vàdkb`wR[YLV`[Uv\ZwzhJ[`"k W%|ad[`x à ä aQXZT\ZYTwRxYT>[]\Zwz\ú mV`[Ùv\ZwzhJ[eP

# : UH+BJ=C´$B w

TWT2³aQ~MczT ¢YPROQP²³IT>k\k\XZT>ad kZwRæWT>k&aQXZTµT>adkZVYXbT>xº~ []V&~TWX&hd\ZaQXbdTW\XZT>kT>[]\ZaQ\ZwzhJ[`k%~@TWLhdXZTy`Xbk\ madwRcRVYXbTdP kZ\XZT>adÈkZwzæWT~T>czhf 2UvhJV`[]\Zkadkad[V[`kZVÙWUvT>kZkZLV`cI\XbwRadcãPTXZTWhdXZ\\ZYT~T>k\\T>kZ\TWXZLhdXá ad[UvTqx`VYXbwR[à\Xbadwy[`wR[YàdkZTwy[dhJczwR[Y JOWÊ \Xbadwy[`wR[Yk\XZT>ad k%a>dTWXbaQdT>x¯hdTWXOWÊ wR[`x`TW@T>[xYT>[\ ØÙ

O

dïûúTï

cRdhdXbwz\Z` ³adkjïO ³adkZjµs ³ádkjµ ³IXbadxwz\ZwzhJ[àdcE±G³¶ Q¯lÃOWÊdÊ n álmÊ n álmÊ n ±G³¶ ®Ë®YhdXZdTW\uaQ\T>k ¢s¯lÃOWÊdÊ n ¢JÊlã{QÊ n s¨¢ÒÝlãrQÊ n ³áQ~czT2¢YPROQdTWXbaQdT\T>k\ k\XZT>adÌkZwRæWTlLTWXbUvT>[]\ZaQdThdkZVÙWUvT>kZkmV`ci\XbwRadcyk&JwzdT>[wy[ÑMaQXZT>[\Z`T>kZwRkonoP ÂÃ[ ³ádkjhJ[YT&[YTW\úfhdXbj2fwz\Zµ©hdXbdTW\_JaQ\T>kTvYUvTWT>xYT>x\ZYT&cRwR wz\hdOWÊdÊdÊ \ZaQXZdTW\hUWUWVYXZXZT>[UvT>kWP aV aV

aV

aV

aV

aV

[YTW\úfhdXbjkWP ¦i§`Ô>üÆG; S Çhd\Zå\Xbadxwz\ZwzhJ[àdc±´³¶Íad[xÑ±G³¶9fwz\ZÑ©hdXbdTW\%JaQ\T>kczT>aQXo[º\ZYT2\Zadkj@PhdXbkZT2TWXZLhdXá ad[UvThd´±´³¶,fwz\ZÝ©hdXbdTW\JaQ\T>k_wRkUWadV`kT>x2~ÝkZczhBfTWX_UvhJ[]dTWXZdT>[ÙvT>%~@T>UWadVkT\ZYT[YTW\àdki\h czT>aQXb[ \h XbT>T>&~TWXTWdTWXZ\Z`wR[` ad[`x[Yhd\_\h ©hdXZdTW\WP ¦i§`Ô>ü I¥ S ±Ý³¶,fwz\ZÝLhdXZdTW\JaQ\T>k_khJczdT>ki\ZYT`XZhd~cRT>°TWdT>[µf`T>[Ýadx`xwz\ZwzhJ[2ad[`xÝV`cz\ZwRcRwRUWaQ\ZwzhJ[ÝaQXZT UvhJ%~wR[YT>Px %fYTWXbT>adk\Xbadxwz\ZwzhJ[àdce±G³{ ¶ kkhJcyVY\ZwzhJ[`kaQXZT[`hd\kZ[\ZcRµadUWUWV`XbaQ\TdP³_wRkkZ`hfk\ZàQ\ LhdXZdTW\JaQ\T>kadx`x¯adcRdhdXbwz\Z` wyULV[Ùv\ZwzhJ[àdcRwR\gµ\hT>hdXZ~cRhUbjk~T>kZwRxYT>kXZT>czT>adkZwR[` XZT>khJV`XbUvT>kxVYXbwR[Y XbV[\ZwR TlL\Z`T>wzXhdXbwzJwR[adcVYXZhJkTf`wyUbwRk[Yhd\T>kZkT>[]\ZwRadceYTWXZTBnoP ¦i§`Ô>üç· S ³IXbadxwz\ZwzhJ[àdc´±´³¶ÌUWad[`[Yhd\qkhJcRdT\Z`TXZhd~czT>,aQ\adcRAc %fYTWXbT>adk±´³¶Ìfwz\Z²©hdXbdTW\JaQ\T>k xYhT>k [`xdhhx²ad[xTWdT>[)ÞÃTWXZLT>Uv\Zß2kZhJcRVY\ZwzhJ[`k>P_-]G Ç ³_YT%©hdXZdTW\JaQ\T>kczT>aQXo[\h2`XZTWdT>[]\±´³{ ¶ k V`[UvhJ[\XZhJcyczT>x&wR[]\TWXb[àdck\ZaQ\TdXZhBf_\ZlmkTWTT>Uv\ZwzhJ[PtP£¢]n %J~qXZT>kTW\\ZwR[`k\ZaQ\T>kGhJ[UvTkZ\hdXZT>xwR[YLhdXb aQ\ZwzhJ[ ~T>UvhJT>khd~MkhJczTW\TdP ³YTXZT>kZV`cR\Zk UvhJk2aQXbT ad[`xàQ\hdXb©hdX2±´³¶ LT>xÁfwz\ZUvhJ[\ZwR[VàdcwR[YVY\ k\XbT>ad kläàQ`\TWXJno'P %EfYTWXZT%hd~khJczTW\TT>hdXowzT>k[YTWT>x\hµ~T%xwRkZUWaQXbxYT>x¯aQ\khJT@hJwy[\lmkTWT²Þ³ádkj %ucywRx`wR[Yµadx`x`wR\ZwzhJ[`ßnoPÄTWXbwRT>[]\qsµkZYhfk\ZàQ\qLhdXZdTW\qJaQ\T>kadcRkhdXZT>aQ\ZcR²madUWwRcRwR\ZaQ\Thd@TWXoaQ\ZwzhJ[`k wR[]dhJczwR[Y%Vcz\ZwzcRwyUWaQ\ZwzhJ[EP

5N

@ÿ ÂÃ[\Z`wRkIUbàQ`\TWXfTixYT>hJ[`kZ\XbaQ\T>xq\ZàQ\©hdXbdTW\IJaQ\T>k´xYh[Yhd\hJ[`czkTWXZdTLhdXI\ZYT}`XZhUvT>kZkZwR[YhdYUvhJ[]\ZwR[Vàdc wR[`VY\k\XZT>ad k~VY\adcRkh adV`JT>[\±´³{ ¶ k_aQXbwR\Z`TW\ZwRUUWaQaQ~wycRwz\ZwzT>kWP T`XZT>kZT>[\T>x\ZadkjkhJ[¯UvhJ[\ZwR[Vàdc´wR[YV`\_k\XZT>ad kfwz\ZaÝczTWdT>chdaQXbwz\Z`TW\ZwyUqUvhJczTvYwz\úfYTWXZT \Xbadxwz\ZwzhJ[àdcE±G³¶ëLadwycRk~VY\±´³¶ëfwR\Z©hdXbdTW\JaQ\T>kkhJczdT>k\ZYT&\ZadkZjkwR[ad[¯T>czTWJad[]\fa>dP[\ZYT hd\ZYTWXàd[`xfiTàdT[Yhd\LhJV`[`xaq\ZadkjdTW\}\ZàQ\\Xoadx`wz\ZwzhJ[àdcY±G³¶ UWad[ khJcRdT_~VY\±G³¶ fwz\ZLhdXZdTW\ JaQ\T>kUWad[`[Yhd\WP -Æ

ds

×]êëìÚGÛ_íz{dï

>Ú )ÛÚ$ðW×çØì}Û_íëÚ$ð>ØÙÜØÙ ×_ØÙÚ$ðÃÙò¦ëÿNðÃÙìoòÚÜeÚ]íÛë ï Ü

ëífð ]ê

¨

©«ªX¬®~¯Z°

¸7

³

½

nÆÅ

77

À

5 ¸ 5

¿

¹ ½

4 7

)7«·¾¸:7 ³

¸

5 ¸

¹ ¾

Ä6¿ ¸

Eÿ

m e ~$

V` ad[`k]VwRUZjczczT>aQXb[\h XZT>UvhdJ[`wRæWT%Xb]\Z` wRUMaQ\\TWXb[¯kZT>]VYT>[UvT>k&%MfYhJkT%x`T&[`wR[Y adkZ@T>Uv\ZkaQXZT\ZYT \T>hdXbadcwR[]\TWXZädcyk~TW\gfiTWT>[-kbVY~YáäaQ\\TWXb[kWPçhJ[dTWXbkZT>cz>%xYXbV TWXbkad[`xhd\ZYTWXbk2aQXZT¯adcykhÑaQ~cRT\h dT>[YTWXoaQ\T%`XbT>UWwRkT>czµ\ZwRT>x¯Xb]]\Z wRUqkT>VYT>[ÙvT>khdhd\hdXUvhJ ad[`x`kWP³`wRk hd\ZwzäQ\T>k\ZYTk\ZV`xYhd aQXZ\Z¤w UWwRadcGkk\T> k\ZàQ\qcRT>aQXb[ï\hkTWaQXbaQ\T hdXqdT>[YTWXbaQ\T MaQ\\TWXb[`k\ZàQ\qUvhJ[]dTW wR[YLhdXb aQ\ZwRhJ[¯\Z`XZhJVYJ \ZYTqcRT>[Yd\ZhdGwR[\TWXbädcRk_~@TW\úfTWT>[TWdT>[\Zk>P )wRxYT>cR¯V`kT>xaQ`XZhJadUb`T>kq\hkT>VYT>[ÙvT `XZhUvT>kZkZwR[Y[ ¶aQXbjdhº¶hxYT>cyk @l Ä¶²¶k n % \ú]wyUWadcRczµx`wRkZUWaQXbx¯kZVÙowR[YLhdXb aQ\ZwRhJ[EP³_YTWaQXZT%kbVÙWUvT>kZkmV`cIwy[kTWT>Ub¯XZT>UvhdJ[`wz\ZwRhJ[`XbT>UWwRkT>cz2~T>UWadV`kT \ZYTWxYh%[Yhd\UWaQXZT©hdX\ZYTxßw @TWXbT>[ÙvT_~TW\gfiTWT>[Ýkbczhf-ad[`xmadk\}dTWXbkbwzhJ[`khdEa&JwzdT>[ khdjdT>[ fhdXbxeP\ZYTWX \ZadkjkkZVÙbåadk%Xb\Z`xYTW\T>Uv\ZwzhJe[ %ÄV`kZwRU `XZhUvT>kZkZwR[`V`wzXbTTvYadUv\q\ZwR T T>adkZVYXZT> T>[\ZkWPÝcz\Z`hJVYJ adN [ Ä¶¯¶(UvhJV`cyx xYT>adcÄfwR\ZÃa [`wz\T kTW\hdwR[]\TWXZQadcRk ~TW\gfiTWT>[)JwzdT>[)TWdT>[]\Zk~"xYTWdhd\ZwR[` akZTWaQXbaQ\T wR[]\TWXb[àdck\ZaQ\T LhdXµT>adUbÕwR[]\TWXZQadAc %\ZwRk2fhJV`cyx"~T UWV`%~TWXbkhJTad[`xwy[Y&T UWwzT>[]&\ %ead[`xfihJV`cRxï[Yhd\qV`kZT\ZYTdTWXZïk\XZT>[Yd\Zïhdq Ä¶²¶¯kq\h~TwR[QaQXbwRad[]\\h [YhJ[YágcRwR[YT>aQX\T>hdXbadcekZ\XZTW\ZUb`wy[YYP S^^kYhJcRxµhdXZT`XbhJ wRkT©hdXXZT>UvhdJ[wzæ>wR[YaQ\\TWXb[`k\ZaQ\aQXbTx`&T [YT>xÝ~]Ý\T>hdXbadcx`wRk\Zad[ÙvTdPÄÂÃ[ madUv&\ %fwRczT Ä¶²¶¯k}ad[`x\Xoadx`wz\ZwzhJ[àdcx`wykZUvXZTW\Tk%~@hJcywRU}dXoad aQXcRT>aQXb[`wR[YxYTWwRUvT>kaQXbTcRwR wR\T>x%\h%x`wRkÃá UvXZTW\Tk\ZaQ\TkadUvT>k %S^^k}aQXZTwR[`Xbwy[ÙWwzczTikZV`wz\T>xLhdXadcRcYkT>VYT>[ÙvT_cRT>aQXb[`wR[Y\ZadkjkÄ~T>UWadV`kT\Z`TWàdT ³´VYXbwR[`%UWaQaQ~wycRwz\ZwzT>kläwzTWdT>cy ad[`w[ * hJ[]\ZaQ¤d¤OnoPi³wRUWadcS^^ cRT>aQXb[`wR[YadczdhdXbwR\Z` klm|´T>aQXbcRVY\Ãá \TWX %O>¤d¤drJnTWXZLhdXb8dXbadx`wzT>[]\qxYT>kZUvT>[]\qwR[aµdTWXZ²dT>[YTWXbadckZadUvThdhd\T>[]\ZwRadcRcz²[YhJwRkTváäXbT>kZwRk\Zad[]\adczdhQá XbwR\Z` keV`kbwR[Yx`wyk\Xbwz~V`\T>ex %BUvhJ[]\ZwR[VYhJV`kÃáäQadcRVYT>xwR[]\TWXb[àdckZ\ZaQ\T>kG\h aQXZT>adc áäQadcRVYT>xwR[YMVY\ekT>VYT>[ÙvT>k´\h XZT>adczáäädcRV`T>xÝhJV`\VY\_kT>VYT>[ÙvT>kWP Ä~`XbwRx Ä¶¯¶ágS^^/aQ``XZhJadUoYT>k&lmÇiT>[YJwzh * ®YXbadkZUvhJ[`=w %eO>¤d¤drJn wzJ]\ ~T aQ~czT\h²UvhJ&~MwR[YT \ZYT wzXb\ZVYT>khd~hd\Z TW\ZYhxYhJczhdJwzT>k %~V`\\hhJVYXqj[YhBfczT>xYdT a>dTÝ[`TWdTWX%~TWT>[ aQ`McRwzT>x2\h \ZYT`Xbhd~czT> hdG`XZT>UWwRkZTTWdT>[]\\Zwy wR[YadkxwRkZUWV`kZkZT>x2`TWXZTdP |}XZTWwzhJV`k\Zadkjk adcRXZT>adxY XZT>V`wzXZT>x \ZYT²±´³¶ [YTW\úfhdXbj\hÑadUv\ÝV`@hJ[ TWdT>[]\Zk2\ZàQ\ hUWUWVYXZXbT>xrQÊ x`wykZUvXZTW\T\ZwyT¯k\TWMk2aQd[`x`T>[\ZczåhdfàQ\µàQ`T>[YT>xhdTWX\ZYT²wR[]\TWXZdT>[`wR[YÑ¢¤åk\TWkïlmkTWT aQ`\TWX&µad[`é x ÄhUoYXZT>wz\TWoX * Ub` wRx`VY~TWX %Ó>¤d¤JdnoPÝSwzJ]\~@TWLhdXZT\ZYT UvXowz\ZwRUWadcÄhJT>[]&\ %IYhBfTWdTW(X % d Ä

¨¢

×]êëìÚGÛí

Tï

Û Ù òÙÄ¥XìíÛ×]ðÜÛàÚ$ð gðÃÙ¥ØRð>Ú]ê ì}Û}Ûì_êØqÿIÛÿ´ÜeÚ ÿ _ëí ð

y output gating sy ∗ c

cell output

y

out

c

peephole connection

net ϕ

wϕ

y

forget gate

ϕ

∆t

memorizing

out

wout

netout

win

netin

output gate peephole connections

∗

and forgetting

gy input squashing g input gating

cell input

s

CEC

cell

c state

in

∗

y

∆t in

input gate

wc netc

®GwzJVYXZT&rPROQi±´³¶ÉT>hdXb2~MczhUbj fwR\ZhJ[YTqUvT>cRcA§YTWTW`hJczTUvhJ[`[YT>Uv\ZwzhJ[`kUvhJ[`[YT>Uv\M \h \ZYTJaQ\T>k>P \ZYTWXbTfadka%YT>cR`LVcÞ aQXZjdTWXbßwR[`VY\}wR[`©hdXb wR[Y\ZYT[YTW\gfihdXZjÝ\ZàQ\iwR\Zk[YTv\adUv\ZwzhJ[2fhJV`cyx ~TUvXoVÙWwRadcãP ³_V`k\ZYT[YTW\gfihdXZjÝxwRx [Yhd\iXZT>adcRcR à>dT\hczT>aQXb[Ý\hT>adkZVYXbTa%\ZwRTwR[]\TWXZädcMhdrQÊk\TWk §wRP\ âZV`k\àdx \h2czT>aQXb[\hµk\hdXZT%XZT>czTWQad[\wR[YLhdXb aQ\ZwzhJ[µLhdXrQÊÝk\TW&k %@ad[`xVkT&wz\_hJ[UvT&\ZYT aQXZjdTWXfiadkhd~kTWXZdT>`x 9 khJ TW\Z`wR[Y\ZàQ\wyk_wRhJkZkZwR~czTLhdX_\Xbadxwz\ZwzhJ[àdcES^^k~VY\_UvhJ aQXbaQ\ZwzdT>cz2T>adkµLhdX±´³¶ P ÇVY\fàQ\wz\ZYTWXbT2aQXZT[YhïkZVÙoÑ aQXZjdTWXbkaQ\adcR:c ÇÆ)àQ\wz\ZYT[YTW\gfihdXZjåwz\ZkT>czàdk\hczT>aQXb[Ñ\h T>adkbVYXZTad[`xÝwR[]\TWXb[àdcRcz%XZTW`XZT>kZT>[\\Z`Tx`VYXoaQ\ZwzhJ[hdE\Zadkj]ágkT>UW'w U_wy[\TWXZQadcR&k %hdX\h },/Bp,/032>;,kT>]V`T>[ÙvT>k hdaQ\\TWXb[kkTWaQXbaQ\T>xÝ~%TvYadUv\iwy[\TWXZQadcRk ÇwÄTWXZTfT_fwRcRcYk\ZV`x`%\hqfàQ\Tv\T>[\\Z`wRkwRk@hJkbkZwz~czTdP´³_YT `wRJ`czï[YhJ[`cRwR[`T>aQXq\Zadkjkwy[ \ZYTÝ`XZT>kT>[]\UoàQ`\TWXxYh²[Yhd\wy[dhJczdT2ad[º\ZwRTÝ aQXZjdTWXwR[YVY\Zk §Gwy[`k\T>adx \ZYTW XZT>]VwzXZT\ZYT[YTW\gfihdXZj \h\ZwyTXZT>UWwRkT>cz ad[xµXZhd~V`k\ZcR adUvXZhJkZkczhJ[Y\ZwRTcRaQJkwR[µUvhJ[]\ZwR[VàdcwR[YVY\ k\XbT>ad kWP ÇiTWLhdXZT&fiT&xYT>kbUvXbwz~ThJVYX[YTWf \ZwR wR[`TvTWXbwR T>[\ZkfiTqfwycRpc `Xbk\_wyxYT>[\ZwR©µafiT>aQj[YT>kbkwR[±´³{ ¶ k UvhJ[`[`T>Uv\ZwzhJ[ kZUoYT>>T %ad[`x wy[\XZhx`VÙvT>k@TWTWMYhJczTUvhJ[`[`T>Uv\ZwzhJ[`k}adkiaXZT> T>xYläT>Uv\ZwzhJ[µrPtsJnoPiT>Uv\ZwzhJ[`kirPt ad[`xrP£¢2xYT>kZUvXbwR~@T\ZYTq hx`¤w `T>x2©hdXZfaQXbxad[x~adUZjfaQXbxadkbk©hdXÝÞÃ@TWTWMYhJczTq±Ý³¶ P ß ÿ µ ý7¶ àÿ3þ O _þ $Eÿ TaQXbT_~V`wRcyx`wR[Y_hJ[Ý±´³¶ófwz\ZLhdXZdTW\JaQ\T>kläaQ`\TWXJn%]kbwRcz%UWadcRczT>x Þ±G³¶¯ßwR[fàQ\LhJcRczhBfkWP Ï ªmKmÐÑK©?§`?BKLCM=CMù¯«¬e¦%m S adUbçJaQ\TXbT>UvT>wzdT>kUvhJ[`[YT>Uv\ZwRhJ[`k2©XZhJÍ\Z`Tïwy[YVY\ V[`wz\Zkµad[`x-\ZYT hJVY\MVY\ZkhdiadcRcÚvT>cRcRkWPÇVY\\ZYTWXZTwRk[Yhx`wzXZT>Uv\UvhJ[`[YT>Uv\ZwzhJ[ïLXZhJ8\ZYT Õwz\wRkkZVY@hJkZT>x\hUvhJ[\XbhJcãP cRc´wz\UWad[ïhd~kTWXZdT x`wRXZT>Uv\ZczwRk\ZYTUvT>cRc´hJVY\MVY&\ %Ef`wRUoïwRkUWczhJkT\hæWTWXZhadkczhJ[Yadk\ZYThJV`\VY\JaQ\T wRkUWczhJkT>xEP³YTXZT>kbV`cz\ZwR[Y%cRadUZjÝhdeT>kZkT>[]\ZwRadc@wR[YLhdXb aQ\ZwzhJ[2 a>ÝàQXbÆ[YTW\gfihdXZj TWXZLhdXb ad[Uv>T %]T>kT>UWwRadcycz wR[UWadkZTqhdG\ZYT\Zadkjk_fiT&aQXZTqdhJwR[`\h k\ZV`xYµYTWXZTdP ¾¸¸]ÓGÀIC@ªL¸&HJC@==¸]HJ?BKLCM=EÔ S VYXIkZwR czT´~VY\EdTWXZ/T T>Uv\ZwzdT}XbT>T>xYwRkE\hadxxfT>wRJ\T>x¯ÞÃ@TWTWMYhJczT>ß UvhJ[`[`T>Uv\ZwzhJ[`k©XZhJ \ZYTqiÁ\h\ZYTJaQ\T>khdI\ZYTkbadT T>hdXZ ~cRhUbjlm®´wRJVYXZTrPROnoPi³_YTJaQ\T>kczT>aQXb[ \hkb`wzT>cRxï\ZYTµiàLXZhJÉV[fad[\T>xÑwR[YV`\Zk lL©hdXZfaQXbxºadkZkonhdXV`[fad[]\T>x TWXZXZhdXkZwzJ[àdcyk lL~adUZjfiaQXox adkbkonoPÄ³IhqjdTWTWÝ\ZYTkZwzT>cRxwR[]\ZadUv&\ %x`V`XbwR[YczT>aQXb[`wy[Y[YhqTWXZXZhdXkZwzJ[àdcRk}aQXZT_`XZhdMaQJaQ\T>x ~adUbjLXZhJ°JaQ\T>k wRaµTWTWYhJczT UvhJ[[YT>Uv\ZwzhJ[`k%\h\ZYTiólmkZTWT ~adUbj]faQXbxÑadkZ&k %ÄT>Uv\ZwzhJ[ÑrP£¢]noP|´TWTW`hJczT UvhJ[`[YT>Uv\ZwRhJ[`k aQXZT\XZT>aQ\T>xµcRwzjdTXZTWJV`cyaQXUvhJ[`[YT>Uv\ZwRhJ[`k\h%JaQ\T>klLTdP£Y'P %LXZhJ/\ZYTwR[YV`\onTvUvTW`\©hdXiVYExàQ\T\ZwR wy[YYP´®`hdX nÆJø

RTS

p_

?

ÏÅØ 8

TïûúTï

ÜÜ

dr

qí ëíiñþìpë

vU hJ[]dT>[\ZwRhJ[àdc±G³¶\ZYT%hJ[`czkhJVYXoUvT&hdXZT>UWV`XZXZT>[]\UvhJ[`[YT>Uv\ZwzhJ[kwyk_\ZYTUvT>cRcehJVY\MVY\iÓ %@khÝ\ZYThdXbxYTWX hdiVYExàQ\T>kfwz\Z`wR[²aµcRa>dTWX%wRkaQXZ~wz\XbaQXbdP%|´TWTWYhJczTUvhJ[`[`T>Uv\ZwzhJ[`kLXZhJ8fwz\Z`wy[\ZYTUvT>cRc=%EhdXXZT>UWVYXZXZT>[]\ UvhJ[`[`T>Uv\ZwzhJ[`kLXZhJóJaQ\T>k&%@YhfiTWdTWX%MXZT>]VwzXZTaXZT&[`T>T>[\hdÄ±G³¶{ kVYExàQ\TqkZUb`T>TdP Ó}D§Y?B¸]Ô²ùgCMA Ó¸¸]ÓGÀIC@ªL¸Á«_¬e¦ S }adUb T> hdXZ UvT>cRc}UvhJ @hJ[`T>[\kZYhJV`cyx~TÝVYExàQ\T>x~MadkT>x hJ[µ\ZYTq hJk\_XZT>UvT>[]\_adUv\ZwzQaQ\ZwzhJ[`khdXkZ\ZaQ\T>k_hdGUvhJ[`[YT>Uv\T>xkZhJVYXbUvT>kWPÂÃ[µ\ZYTqkZwyczT>k\UWadkTq\ZwRkiXZT>]VwzXZT>k a\úfhQáäMàdkT%V`xaQ\T&kZUb`T>>T §f`T>[XZT>UWVYXZXbT>[\UvhJ[[YT>Uv\ZwzhJ[`k©XZhJJaQ\T>kaQXbTq`XZT>kT>[]&\ %\Z`T `Xbk\MàdkT V`k\~TmVYXZ\ZYTWX_kZV`~xwzwyxYT>x wR[]\h\ZYXZTWT%k\TWk&lm1a %£P~ % Uno OQPÉlmanÂò[YMVY\JaQ\T%adUv\ZwzQaQ\ZwzhJr[ Ó C% lL~@nµ©hdXZdTW\JaQ\T%adUv\ZwzQaQ\ZwzhJ[rÓ Ê % lmUnUvT>cRcEwR[YV`\ad[xUvT>cRcEk\ZaQ\T % sPhJVY\VY\JaQ\TadUv\ZwzäQ\ZwRhJw[ Ó 5¡l¢ ad[xUvT>cRcEhJVY\MVY\ Ó P ³_V`k´\ZYThJVY\V`\´JaQ\T_wyk´V`xaQ\T>xaQL\TWX}UvT>cRck\ZaQ\T %kTWT>wy[YwRawz\ZkGTWTWYhJcRTiUvhJ[`[`T>Uv\ZwzhJ[\ZYTUWVYXZXZT>[]\ QadcRVYThd cl ÔZnlmadczXZT>adxY2Va @T>Uv\T>x¯~ÝLhdXZdTW\_JaQ\T&ad[xµXZT>UvT>[\wR[YVY\on %Yad[`xµhJkZkbwz~cz\ZYTqUWV`XZXZT>[]\_wR[YVY\ ad[`xµLhdXZdTW\JaQ\T%adUv\ZwzQaQ\ZwzhJ[`kWP

5N

nÆ

O

p¥ e

:

ÇiTWLhdXZT kT>UWwzLwR[` \ZYTT>VàQ\ZwzhJ[`kLhdX\ZYT ±G³¶ hxYT>c´fwR\Z¯TWTW`hJczTUvhJ[[YT>Uv\ZwzhJ[`k&%IfT wR[]\XZhx`VÙvT a wy[YhdX_kZwRMcR'w UWaQ\ZwzhJ[µV`[`XZT>cRaQ\T>xµ\h \ZYT%UvT>[\XbadcwRxYT>ahdG\Z`wRkUoàQ`\TWX>PhmaQX±Ý³¶ÉT> hdXZUvT>cRcRk wR[UvhdXZhdXbaQ\T>xad[wR[YVY\´kZVàdkZ`wR[`mV`[Ùv\ZwzhJ_[ 2>B:ad[hJVY\V`\Gkb]Vàdkb`wR[YLV[Ùv\ZwzhJ[lmUWadcRczT>xÄ wy[%T>aQXocRwzTWX ±G³¶ VY~cywRUWaQ\ZwzhJ[`konoPTXbT>hdT\ZYTcyaQ\\TWXLhdXcRadUbjhdET>wzXowRUWadcYTWwRxYT>[ÙvT_\ZaQ\iwz\wRkXZT>adcycz[`TWT>xYT>x lmwR[madUv&\ %\ZYTdTWXZ `Xbk\±G³¶VY~cRwyUWaQ\ZwzhJ[@l ÄhUbYXbT>wz\TW"X * YUb` wyx`]V`~@TW(X %O>¤d¤JdnadczXZT>adxYhJ wz\\T>x \ZYT hJVY\MVY\_kZVàdkZ`wR[`%mV`[Ùv\ZwzhJ[µLhdXkhJ TqTvTWXbwyT> [\Zk noP ¬?¸Ó ;]§ hã;Cî S ³_YT%wR[YVY\JaQ\TadUv\ZwzäQ\ZwRhJr[ Ó µad[`x\Z`Tq©hdXbdTW\JaQ\TadUv\ZwzäQ\ZwRhJr[ Ó.ÊaQXZT&UvhJMVY\T>x adkW £ Ï ØGÙÔ> £Jcl Ôn Õ Ú Í £ Ï Ó cl Ô5Þ On ® Ú û Í £ £± £± cl Ô5Þ On ÖÓ £ cl Ô!n Õ¹àJ £Jcl ØGÙ&Ô5 £]cl Ônmn Ö lãrPROn

ª ò£ ä Ê £Jl n Ê £ l On® ª û ò ä Ê £ £± £± l On Ê £ l Zn Ê £Jl Ê £l Znn lãrPtsJn ³_`TiTWTW`hJczTUvhJ[`[YT>Uv\ZwzhJ[`k©hdX\Z`TwR[YV`\JaQ\Tad[`x\ZYT_LhdXZdTW\JaQ\TaQXbTwR[ÙvhdXZhdXbaQ\T>xwy[T>]VàQ\ZwRhJ[ rPRO ad[`xrPts ~]µwR[ÙWcyV`x`wR[Y%\ZYTkqlmUvhJ[]\ZadwR[`wR[Y\Z`T&UvT>cRcEk\ZaQ\T>konhdÄT>hdXZ2~czhUZjÄ ¦ adkkhJVYXbUvTqV[`wz\ZkWP ¬?¸Ó,;H.S \ ëÊ i\Z`Tk\ZaQ\TH l nhd T>hdXZUvT>cRc8 wRkwy[`wz\ZwRadcRwRæWT>xÑ\hÑæWTWXZh kZV`~kT>VYT>[\ZcR l ]²ËÊnwz\qwRkUWadcyUWV`cRaQ\T>xï~²adx`xwR[Yµ\ZYT kZVàdkZYT>x EJaQ\T>xºwR[YMVY\\h\Z`T k\ZaQ\T £ aQ\q\ZYTXZTWwRhJV`k\ZwRT k\TW l On `fwRUbwRkV`cz\ZwRcRwzT>xïlLJaQ\T>xMn~µ\ZYTLhdXZdTW\JaQ\TadUv\ZwzäQ\ZwRhJ[ ,Ê l Zno £± l n £± l On £± l n Ê £ l Zn £± l On"® £ l ZnÏ@ l £± l nn lãrPtJn ¬?¸Ó-¥S ³_YThJVY\V`\JaQ\T%adUv\ZwzQaQ\ZwzhJ[ 5¡l¢ wRk_UvhJMVY\T>xadkW £ 5¡l¢|£ l Zn 85¡l¢|£ l On"® û 85¡l¢|£ £± £± l Zn 5¡l¢|£ l Zn >¡D¢|£ l 5¡l¢|£ l Znn ª ò ä lãrP£¢]n Ï

ØGÙÔ

cÔ

Õ

Ú

Ï

Í

Ï

yÔyÕ

Ó

Ï

cÔ5Þ

Ú

1%

Í

cÔVÞ

nÖÓ

cÔ Õ¹à

cØGÙÔ

cÔ

Ú]ê ì}Û}Ûì_êØqÿIÛÿ´ÜeÚ

×]êëìÚGÛí

Tï

ÿ _ëí ð

]VàQ\ZwRhJ[¯rP£¢µwR[ÙWcRV`x`T>k\ZYT%TWTW`hJczTqUvhJ[`[YT>Uv\ZwzhJ[kLhdX\Z`T&hJVY\V`\JaQ\T©XZhJ \Z YTikhdT>hdXZ ~cRhUbj¦ fwz\Zµ\ZYT&UvT>cycek\ZaQ\T>kM lcÔZn%adk_VYExàQ\T>xwR[kZ\TWÑOWUQP}³_`TqUvT>cRcEhJVY\V`\Ó wRkUvhJ VY\T>xadkW £± cl Ôn ÕXÓ 5¡l¢|£ cl Ôn £± cl ÔZán × Ó lãrPtrJn ³YTT>]VaQ\ZwzhJ[`kLhdX_\ZYThJVY\MVY\_V`[`wz\Zfk åXZT> adwR[adk_kZ@T>UW¤w `T>xwR[µT>VàQ\ZwzhJ[`ksPt¤P

ÿ_ PÉnX e O : ³_`TXZTWwRkT>xÝVYExàQ\TkZUoYT>TLhdXiT> hdXZ ~czhUZjkiadcRczhfk}LhdX\XbT>aQ\ZwR[Y@TWTWMYhJczTUvhJ[[YT>Uv\ZwzhJ[`kicRwzjdTXZTWQá V`cyaQXUvhJ[`[YT>Uv\ZwRhJ[`klmkTWT&T>Uv\ZwzhJ[`krPtsad[`x2rPtJn %àd[x2kZh%XZT>V`wzXbT>khJ[`cz wR[YhdXUoàd[YdT>k\h\ZYT~adUZjfaQXbx adkbk läàQ`\TWX&JnoPT fwRcycI`XbT>kT>[\wz\~T>czhBfà~MVY\[Yhd\LVcRczXZTvágxYTWXowzdTwz\WP&T fwRcRAc %YhfiTWdTWX %@hJwy[\ hJVY\}\ZYTx`'w @TWXZT>[UvT>k\h&\Z`T`XZTWwzhJV`kT>]VàQ\ZwRhJ[`k}wR[µT>Uv\ZwzhJ[2PtsPtsP}@T>[x`w ÇÁJwzdT>k}MkT>V`xYhQágUvhxYT_LhdX \ZYTT>[]\ZwzXZTqadcRdhdXbwz\Z`P ÂÃ[µfàQ\LhJcRczhBfkfiTfwRcRcM`XZT>kT>[]\T>VàQ\ZwzhJ[`kLhdX_¢ÔÓ ±Ý³¶ÈfwR\Zµ©hdXZdTW\JaQ\T>kad[`xµTWTWYhJcRTUvhJ[[YT>U á \ZwzhJ[&k %Y~VY\fwz\Z`hJVY\hJVY\VY\_kb]Vàdkb`wR[YYPÄ³_YTkZwRJ[ ÕàfwRcyc@wy[`x`wRUWaQ\TfYTWXbTfT&VkTTWXZXZhdX_\XoV`[ÙWaQ\ZwzhJ[EP âVYXowR[Y T>adUo k\TWwR[¯\ZYT©hdXbfiaQXbx²adkb&k %E[Yh aQ\\TWXfYTW\ZYTWXa2\ZaQXZdTW\qwRkJwzdT>[ïhdX[Yhd&\ %EfiT[YTWT>x \h2VYExàQ\Tq\ZYT%aQXZ\ZwRadcxYTWXbwzQaQ\ZwzdT>k ö £± ö ÍfÎÜÏ/ad[`x ö £± ö Í Î £± Ö LhdXfiT>wzJ]\Zk\h2\ZYTUvT>cRc@l Ò]Õ©ª n %M\h \ZYTqwy[YVY\JaQ\T2@l ÒPÕ7éØn %ad[`x\h \Z`T©hdXZdTW\JaQ\T2@l ÒPÕèGno £± cl ÔZn Õ¢ÔÓ ö £± cl ÔVÞ On Ó Ê £ lcÔZn®r å lcØGÙ&Ô ± lcÔZnn"Ó £ lcÔZnÓ Ï lcÔ5Þ OnÖ ö lãrPt{Jn nÆ

ù

e

¡

¢¡

e

ö

£± l Zn ¢ÔÓ £ £± l Zn ¢ÔÓ £ £± Ö y £± l Zn ¢ÔÓ Ê £± £l Zn ¢ÔÓ Ê £ £± Ö Õ

Ï

Í

ö

ö

Ï

cÔ

ö

ö

£±

Í

cÔ

Í

cÔ

ö

ö

cÔ

Í

Õ

Ï

Í

ö

ö

Õ

Õ

£±

Í

ö

e

£

Ï

£± l On Ê £ l Zn®r@l £± l nn £ £± l On Ê £ l Zn®r@l £± l nn £ £± Ö D £± l On Ê £ l n® £± l On £ Ê £± l On Ê £ l n® £± l On Ê £ £± Ö £± lmÊn £± Ö °Ê ©hdX 8æÏçlé cÔVÞ

ö

ö

ö

Ó

Ï

Í

cÔVÞ

Ó

ö

Í

cÔ5Þ

ö

ö

ö

ö

cÔ5Þ

Í

cØGÙÔ

cÔ à

cÔ

cØGÙÔ

cÔ à

Ó

cÔ

cÔ5Þ

à

Ó

cÔ

cÔ5Þ

à

Ï

Í

cÔ

å £ l å £ l

5 £]l nn l On 5 £]l nn8 £± Ö l On

cØGÙ&Ô

cÔ qÓ

cØGÙ&Ô

cÔ

Ï

cÔ5Þ

n§

cÔ5Þ

Ö

lãrP än lãrP B~@n

lãrPt¡Qan Ê å £ l Ê £l Znn l On lãrPt¡¨~@n Ê å £ l Ê £l Znn £± Ö l On è ã©Gª ê P}VàQ\ZwzhJ[ rP B~ºad[xºrPt¡¨~ºaQXZTLhdX cØGÙ&Ô

cÔ "Ó

cØGÙ&Ô

cÔ

Ï

cÔ5Þ

cÔVÞ

§

mÖ

fwR\Z ö £± lmÊn ö ÍfÎÜÏvÕ ö Í Î Õ vÒ =Ø!Ö _Ö ö \ZYTTWTW`hJczTUvhJ[`[YT>Uv\ZwzhJ[fT>wzJ]\ZkWP ®`hJcRczhfwR[Y`XZTWwzhJV`k}[Yhd\ZaQ\ZwzhJe[ %]fT wR[`wR wzæWT\Z`Thd~1âT>Uv\ZwzdTmV`[Ùv\ZwRhJ[ã ~dXbadx`wzT>[]\}xYT>kbUvT>[\lmkZVY~Yá âT>Uv\\hTWXZXbhdX\XbV[ÙWaQ\ZwzhJ[M n %YUbad[YJwR[Y%\ZYTfT>wzJ]\Zk"ÍfÎÜÏ lLLXZhJ V`[`wz\Ñ8\hV`[`wz\"Òãn~ ad[µadhJV`[]\qóÍfÎÐÏ JwzdT>[¯~\ZYTczT>aQXb[`wR[` XbaQ\~T õ \ZwRT>k\ZYT[YTWJaQ\ZwzdTdXbadxwzT>[\hd_ ã P®YhdX\Z`T%hJVY\MVY\V`[wz\Zk_fiThd~`\ZadwR[ \ZYTqkZ\Zad[`xàQXbx~MadUZj]áä`XZhdaQJaQ\ZwzhJ[fT>wRJ\Uoàd[YdT>kW óÍ Ï%cl ÔZ! n Õ¹õtÕ ä cl ÔZ"n Ó Ï cl Ô5Þ O n Ö Õ ä cl ÔZn Õ öØGãÝÙ&Ô cl ÔZlcn ÔZn × lãrPt¤Jn ä 0e

be

ö

ä

TWXZTfiT&V`kT\ZYT%UWV`k\hJ aQXZkZVàQXZT>xµTWXbXZhdX_hd~1âT>Uv\ZwzdT&mV`[Ùv\ZwRhJ[µ~adkT>xhJ[\ZaQXbdTW\ZkiÔ ä %`wzT>cRxwR[YY Õ ä cl ÔZn Õ¹à ä å cl ØGÙ&Ô ä cl ÔZnn Ù ä cl ÔZn Ö lãrPROWÊn Ä


Tï Tï

J

32

f`TWXZTmÙ ä lcÔZnièÕ¥Ô ä lcÔn¦ÞRÓ ä lcÔniwyk\Z`TTv\TWXb[adcRcz wR[5âÃT>Uv\T>xµTWXZXbhdX>P³_YTfiT>wzJ\_Uoàd[YdT>kLhdX_UvhJ[`[YT>Uv\ZwRhJ[`k \hq\Z`ThJVY\VY\ÄJaQ\TlLhd@\Z`TC¦Qáä\ZÝT>hdXZ~czhUZj`n´LXZhJ/\ZYT_khJV`XbUvTV`[`wR\Zk_lmadk}kT>UWw'`T>x~\ZYT_[YTW\gfihdXZj \hdhJczhdd`"n Í 5¡l¢|£ Ï ad[`xµLhdX_\ZYTqTWTW`hJczTUvhJ[`[YT>Uv\ZwzhJ[`kiÍ 5¡l¢|£ £± aQXZTd óÍ85¡l¢ £Ï cl ÔZ n Õ õtÕð5¡l¢|£cl ÔZ"n Ó Ï cl ÔZn ÖÝóÍ85¡l¢ £ £± cl ÔZn Õ¹õtÕð5¡l¢|£cl ÔZn £± cl ÔZ:n Ö lãrPROdOWan

Õð>¡D¢|£]l Zn cÔ

Õ

¢ÔÓ 5å l¡ ¢|£ l >¡D¢|£l Znn ùú û £ £± l Zn ªò ä à

cØGÙ&Ô

Ú

cÔ

cÔ

Ú Í

£± Õ l n ýü

ä

cÔ

ä

ä

lãrPROdOv~n ×

V`\VY\kb]Vàdkb`wR[YlLXZT>hdT>x YTWXZTBnIfhJVcRx&XbT>]V`wRXZT\Z`Twy[ÙvhdXZhdXbaQ\ZwzhJ[%hd\ZYTxYTWXbwRäQ\ZwzdThdM\ZYThJV`\VY\ Zk VàdkZwR[Y&mV`[Ùv\ZwRhJ[ÝwR[ïlãrPROdOv~@noPi³Ih UWadcRUWVcRaQ\TfT>wzJ]\Ubàd[`dT>kióÍfÎÜÏÕad[xÃóÍ Î £± Ö lLTWTW`hJczTUvhJ[[YT>U á \ZwzhJ[ïfiT>wzJ]\ZkonLhdX&UvhJ[`[YT>Uv\ZwzhJ[k\h\ZYT UvT>cRc_l@ÒqÕ ©ª n%\ZYT wR[YMVY\JaQ\T²l@Ò"Õ é=Øn%Iad[`x \ZYT©hdXbdTW\JaQ\T @l ÒPÕèGnifiT&V`kT\Z`TqaQXZ\ZwRadcyk©XZhJ }VàQ\ZwzhJ[`k_rPtd { %rP BP~ %ad[`xrPt¡¨~I óÍ ± Ï lcÔZn Õ õ Ù lcÔn ö £± cl ÔZn lãrPRO>sJn

£

£

£±

Í

£±

Ï

£± l n £ £± l n Ê£

£

£ £± Ö l Zn û £± l n £± £ l Z£± n Ö ± l Zn £ ª ò£ä ª £ò ä £Ê l Zn û £± l Zn £Ê £± Ö l Zn û £± l n £± £ l Z£± n Ö Ê ªò ä ªò ä f`TWXZT\ZYT&wR[]\TWXb[àdcEk\ZaQ\T%TWXZXZhdX £± wRk_kZTWaQXbaQ\T>czµUWadcRUWV`cRaQ\T>x©hdXT>adUb T>hdXZUvT>cRcã £± l Zn ¢ÔÓ 5¡l¢|£ l Zn ë £± Õ l Zn5ì óÍ

£

Ï

Ï

óÍ

lcÔZn

Õ

cÔ

Õ

õ

û

õ

Ú

ö

Ú

Ù

cÔ

Ù

ö

Í

cÔ

ö

ÖÝóÍ

Ï

cÔ

ö

cÔ

Ù

Ù

ö

Ï

Í

cÔ Õ¹õ

ÖÝóÍ

cÔ Õ¹õ

Ú

Ú

Ù

Ù

ö

cÔ

ö

ö

cÔ

ö

cÔ

Í

Í

cÔ

lãrPRO>Jn lãrPROv¢]n

cÔ

Õ

Ó

cÔ

Ú

Í

ä

ä

cÔ

×

lãrPRO>rJn

ä

± wzjdTµ\Xbadx`wz\ZwRhJ[àdc±G³¶ ±´³¶ fwz\Zå©hdXbdTW\JaQ\T>k ad[`xåTWTWYhJczTµUvhJ[`[YT>Uv\ZwRhJ[`k wRkk\ZwRcycczhUWadcwR[ kMadUvT25B 2\ZwRTdPÄ³_YTwR[ÙvXZT>adkZTwR[2UvhJcRTvwz\ú x`VYT\h@TWTWMYhJczTUvhJ[[YT>Uv\ZwzhJ[`kwRkikZ adcRcäG%fiT>wzJ\ZkiTWX UvT>cRcäP nÆ

RTS

eÿ

_p

Tik\ZV`xY±´³¶N kETWXZLhdXb ad[UvTÄhJ[q\ZYXZTWT}\ZadkjkI\ZàQ\IXbT>]V`wRXZTG\ZYT`XZT>UWwykT}T>adkZV`XZT>T>[]\ehdXIdT>[YTWXoaQ\ZwzhJ[ hd}x`T>cRa>kWPTUvhJaQXZTUvhJ[]dT>[\ZwRhJ[àdcÄ\hµTWTWYhJczT±´³{ ¶ %ad[àdczæWT\Z`TkhJcyVY\ZwzhJ[`&k %@hdXTvcRadwR[f [YhJ[`TfiadkLhJV`[`xeP à¸§`ÔFA¨Km=¹Ô>ÓGK©üM¸D¸ªL6§ ZeÄÔ \Z°¬ ^HaDS TWTT>Uv\ZwzhJ[rPtrPtsP}³_YTidhJadcYwRk´\hUWcRadkZkZwR©&wy[YVY\´kT>]V`T>[ÙvT>k UvhJ[`kbwRk\ZwR[Y hd}kZàQXZïkwRjdT>kWP³_`T%UWcRadkZkxYTWT>[`xk_hJ[¯\ZYTwy[\TWXZQadc~@TW\úfTWT>[kZwzjdT>kWPTUvhJ[kZwRxYTWX\gfih dTWXbkbwzhJ[`khd´\ZYTq\Zadkj@UvhJ[]\ZwR[Vàdclã¶ïâqnad[`x[YhJ[ágUvhJ[]\ZwR[]Vadc}lm^¶ïâqnoP^¶ïâ/kT>VYT>[ÙvT>kk\hdaQL\TWX \ZYT2kT>UvhJ[`x kwRjd>T %´fYTWXZT>adk¶²Yâ kT>VYT>[ÙvT>kaQXZTµUvhJ[\Zwy[]VàdcikwRjdT \XbadwR[kWPÇihd\Z^¶ïâ ad[`xå¶ïâ XZT>V`wzXbT&\ZYT [YTW\úfhdXbj¯\hT>adkbVYXZTwR[]\TWXZädcyk~TW\úfTWT>[ kZwzjdT>&k §I¶ïâ adcRkhµXZT>V`wzXZT>k\ZYT`Xbhx`VUv\ZwzhJ[hd k\ZaQ~MczTXZT>kZVcz\Zk wR[`XZT>kT>[UvThdqUvhJ[]\ZwR[]VadcRczk\XZT>ad wy[YºwR[YMVY\Z&k %ifwz\ZYhJVY\Ýad[Tv\TWXb[àdcXZT>kTW\2hd\ZYT [YTW\úfhdXb:j kÝk\ZaQ\TdP-ad[Á±G³¶!czT>aQXb[ \ZYT¯x`ßw TWXZT>[ÙvTµ~TW\gfiTWT>[adcRhJkZ\ wRxYT>[]\ZwRUWadcaQ\\TWXb["kT>]V`T>[ÙvT>k \ZàQ\x`ßw TWXihJ[`cz~ akZ adcRc@czT>[Yd\ZYT>[wR[Yqhd\ZYTwy[\TWXZQadcGlLTdP£Y'P %`©XbhJz Ø²\h Ø[wR[YVY\ kMwzjdT>k ÇwÄhfàxYhT>k\ZYT&x`¤w UWV`cz\ú hd´\Z`wRk`Xbhd~czT> xYTWT>[`xµhJw[ Ø%Ç

d¡

×]êëìÚGÛí

Tï


µ÷ ¸=¸AB§Y?¨KL=¹¯?BKmÐº¸DºÔ>ÓGKLü¸Ô@\v÷µ¦ ¬%aDS TWTT>Uv\ZwRhJ[2rPtrPtP}³YTu³ \Zadkj UWad[2~T_hd~`\ZadwR[`T>x LXZhJ \ZYT¶ïâ°\ZadkZj~TvUoàd[YJwR[`µwy[YVY\Zkad[`x²\ZaQXZdTW\ZkWPqÂò\XZT>V`wzXZT>k\ZYT`XZhx`VÙv\ZwzhJ[¯hdUvhJ[]\ZwR[VàdcGkwzjdT \Xbadwy[`k&%Jf`TWXZT\ZYTwR[]\TWXZQadcY~TW\gfiTWT>[ÝkZwzjdT>kV`k\XbT&`T>Uv\}\ZYT aQJ[`wz\ZV`x`Thdad[ÝwR[YV`\GkbwzJ[àdcY\ZàQ\ a> Uoàd[YdT&aQL\TWXTWdTWXZkMwzjdTdP u³"wRkÝa kT>UWwRadcUWadkThdqTWXbwRhx`wyULV[Ùv\ZwzhJ["dT>[YTWXbaQ\ZwzhJ[ lm|®-u %_kTWTï~T>czhfnoP/Âò[-UvhJ[\Xoadk\\h `XbTWwzhJVkZcz k\ZV`xwzT>x |}®u8\ZadkZjklL)wRcRcywRad k * ÷wzkZTWX %_O>¤d¡dd¤ §âhJI a * ÈhJkZ`wzæ>afi1a %O>¤d¡dd¤ §³kZV`[Yé * ihd\\XbT>cRAc %eO>¤d¤drJn %eu³µwRk`wzJ`czÝ[YhJ[`cRwR[`T>aQX_ad[`xwR[]dhJczdT>k_czhJ[` \ZwRTqcRaQJk~TW\gfiTWT>[kZwzJ['w UWad[]\ihJV`\VY\ Uoàd[YdT>&k %f`wyUbUWad[[Yhd\~@T%czT>aQXb[YT>x¯~]¯UvhJ[dT>[]\ZwzhJ[àdcGS^^kWP|}XZTWwzhJV`k_fihdXZjadcykhÝx`wyx[Yhd\©hUWV`khJ[ k\ZaQ~MwRcRwz\ú wRkZkZV`T>kW!P ÄTWXZ>T %Y~µUvhJ[\XbadkZ&\ %`fTqx`T> ad[`x2\ZàQ\_\ZYTdT>[YTWXoaQ\ZwzhJ[~Tk\ZaQ~czTLhdXqOWÊdÊdÊ kZVÙWUvT>kbkZwzdT kMwzjdT>kWPÄT&kk\T> aQ\ZwyUWadcRczµwR[dT>kZ\ZwzJaQ\T&\ZYT/T T>Uv\hdÄ wy[`wR adcM\ZwRTqcRaQ hJ[\Zadkjx`'w UWVcz\gdP Ï2D´D´K©?BKLC@=I§ªÓ¸AQK©CEDGKLHÑùòF=Hd?BKLC@= ¹M¸=¸AB§Y?¨K©C@= ?§`Ô>üEX Ô \¾q»F ÷ a PTWTµT>Uv\ZwzhJ[ºrPtrP£¢YPÝTÝk\ZV`x` \ZYT`XZhd~czT> hddT>[YTWXoaQ\ZwR[YºTWXbwRhx`wyU mV`[Ùv\ZwzhJ[khd\ZYTWX \Zad[Á\ZYTkZwzjdT\XbadwR[k aQ~hdTdPç³_YTUWcyadkZkZwRU TvYadczT>kaQXbT kZhhd\Z`czhJkbUWwRcRcRaQ\ZwR[`ÑhJVY\MVY\ZkkZVÙoçadkkZwR[`T²fa>dT>k %f`wRUoçaQXZT czT>aQXb[àQ~MczTï~]"LV`cycz UvhJ[`[`T>Uv\T>x\T>adUbYTWXZáä©hdXbUvT>x2S^^kGfYhJkT_V`[wz\ZkGaQXZT_adcRchJVY\VY\ÄV`[wz\Zk´fwz\Z\T>adUb`TWXágxY&T [YT>x adUv\ZwzäQ\ZwRhJ[`k lL)wRcRcRwyad k *®÷wzMkTWX %O>¤d¡d¤JnoP[ïadcz\TWXo[àQ\ZwzdTaQ``XZhJadUo²\XbadwR[`kad[ïS^^/\hµ`XZT>x`wyUv\\ZYT[YTv\wR[YVY&\ § aQL\TWX\XbadwR[`wR[`hJVY\VY\ZkaQXZT©T>x¯~adUZj¯x`wzXZT>Uv\Zczµ\h2\ZYTwR[YVY\khÝadk\h2dT>[YTWXbaQ\T\ZYTfiadTW©hdXbÌlmâhJa *ÈhJkb`wzæ>a>f1 a %O>¤d¡dd¤ §@³_kZV`[` * ihd\\XZT>cRAc %eO>¤d¤ddr §@T>wRkZk %eO>¤d¤dd¤ §@³hf[czTWÝTW\adcã'P %eO>¤d¤d¤JnoP ÄTWXZTfTLhUWV`khJ[µhdXZTx`'w UWV`cR& \ %]`wzJcz[`hJ[`cRwR[YT>aQ(X %\XbwRad[`JV`cRaQXad[x XZT>Uv\Zad[`JV`cRaQXifiadTW©hdXb &k %Y\ZYT cRaQ\\TWX ©T>aQ\ZVYXowR[YczhJ[Y\ZwR TµcyaQJk~TW\gfiTWT>[kZwzJ['w UWad[]\%hJVY\MVY\ Ubàd[`dT>kWPÁJadwye[ %\Xoadx`wz\ZwzhJ[àdcS^^k UWad[`[`hd\iczT>aQXo[ \ZadkjkwR[dhJcRwR[`&czhJ[Y%\ZwRTcyaQJk@l ÄhUbYXZT>wR\TWX %EO>¤d¤5O §`ÇT>[`Jwzh%TW\adcã'P %O>¤d¤¨¢]n %ad[`xÝ`XZTWwzhJV`k fihdXZjïx`wRxï[Yhd\qLhUWVkhJ[ k\ZaQ~wycRwz\g²wRkZkZV`T>kWPÇïUvhJ[]\Xbadk&\ %GfT x`T> ad[`x\ZàQ\&\ZYTdT>[YTWXbaQ\ZwzhJ[º~TkZ\ZaQ~czT LhdX&OWÊdÊdÊ kbVÙWUvT>kZkZwzdT&@TWXowzhxk}hdG\ZYTqfa>dTWLhdXbP

Vþ.$DÌ@ 6ÑÂM" "1- '!3(Í G 6&>Ï+!"$E'8'I '7.$EG 6B x

x

TLhJV`[`x\ZàQ\UvhJMaQXbaQ\ZwzdT>czkZ adcRc±Ý³¶ó[YTW\ZkÄUWad[ adcRXZT>adxY&khJcRdT\ZYT\Zadkjk}aQ~@hBdTdPkZwy[YJczTwR[YVY\ V`[wz\qlmV`kT>xhJ[czÝLhdX\ZadkZjkfYTWXZT\Z`TWXZT%wRkwR[YVY\onwRkLVcRcz UvhJ[`[`T>Uv\T>x\hÝ\ZYT%`wyx`xYT>[µcRa>dTWXUvhJ[`kZwyk\ZwR[Y hdakbwR[YJczT T> hdXZï~czhUZj²fwz\ZhJ[YT2UvT>cRcãPµ³_YT UvT>cRc}hJVY\VY\%wRkqUvhJ[`[YT>Uv\T>xÑ\h\Z`TÝUvT>cRcwy[YVY&\ %e\h²adcRc \ZYXbTWTJaQ\T>k %Iad[`xï\hakZwR[YJcRT&hJVY\V`\V[`wz\lm®´wRJVYXZTrPtsJnoPcRc´JaQ\T>&k %I\Z`TUvT>cyc´wR\ZkT>czç %ad[x¯\Z`ThJV`\VY\ V`[wz\aQXZTUvhJ[`[YT>Uv\T>x2\h%a%~wRadkV`[`wz\lmaV`[`wz\}fwz\Z UvhJ[k\Zad[\adUv\ZwzQaQ\ZwzhJ[ÝhJ[YTBnadkfiT>cRcãPÄ³_YT~wRadkfiT>wzJ]\Zk \hwR[`VY\JaQ\>T %LhdXZdTW\qJaQ\>T %Gad[`xhJV`\VY\JaQ\TÝaQXZTÝwR[`wz\ZwRadcywzæWT>x\h1Ê × 1Ê %qÞds × Êad[r x ®ds × 1Ê %IXbT>kT>Uv\ZwzdT>czdP lmcz\ZYhJVYJ¯[Yhd\UvXowz\ZwRUWadAc %M\ZYT>kTädcRV`T>kàdT%~TWT>[¯©hJV`[xT>wRXbwRUWadcRczÝ\hÝfihdXZjfiT>cRAc §MfTV`kT&\Z`T> LhdX adcRchJVYXqTv@TWXowRT>[]\ZkWP nÁcRcÄhd\Z`TWX%fiT>wzJ]\Zk&aQXZT2wR[`wz\ZwRadcywzæWT>x²\h¯V`[`wR©hdXb,Xoad[`xYhJÉQadcRVYT>k&wR[\ZYTÝXbad[YdT «ßÞ_1Ê ×R5O ÖZ1Ê ×RO ¬äPåÂÃ[åadxx`wz\ZwzhJ[Ñ\hï\ZYTµ\ZYXZTWTµTWTWYhJczTµUvhJ[`[YT>Uv\ZwzhJ[k\ZYTWXZTaQXZTOv¢adVx âVk\ZaQ~czT2fT>wRJ\ZkW¤ ÞV`[wz\Ãáä\hQágV`[`wz\Zß UvhJ[`[YT>Uv\ZwzhJ[`kad[`x¯r ~wRadk_UvhJ[[YT>Uv\ZwzhJ[`kWP_³YT&UvT>cRAc kwR[YV`\kb]Vàdkb`wR[YmV`[Ùv\ZwzhJ`[ µwRk\ZYT wRx`T>[\Zwz\úïLV`[Uv\ZwzhJ[EP³_YTÝkZ]VadkZ`wR[YLV`[Uv\ZwzhJ[hd_\ZYTÝhJVY\VY\V`[`wz\qwyk%a¯czhdJwRk\ZwRU kZwzJhJwRxºfwz\ZºXbad[YdT « 1Ê Ö>O ¬©hdX¶ïâçad[`xu³lLTvYUvTW`\ifYTWXZTTvcywRUWwz\Zczk\ZaQ\T>x2hd\ZYTWXZfwRkZTBn %ad[`x \ZYTwyxYT>[\ZwR\gLV[Ùv\ZwzhJ[LhdX |®uPilmókbwzJhJwRxïmV`[Ùv\ZwzhJ[ fhJV`cyxfhdXbj adkqfiT>cRAc %I~MVY\qfiT LhUWV`kqhJ[º\ZYTÝkZwRcRT>k\kkZ\T>Å\ZàQ\UWad[ khJcRdT\ZYTq\Zadkj@P n VYX%[YTW\úfhdXbjk`XZhUvT>kZkUvhJ[\Zwy[]Vàdcik\XZT>ad k%hdwR[YVY\Zk&ad[`xº\ZaQXbdTW\Z&k §hJ[czaQ\%\ZYT2~TWJwR[`[`wy[Yhda k\XbT>ad°aQXZT\ZYTWXZT>kTW\WP³_`TWV`k\iczT>aQXb[\hadczfiak`XZT>xwRUv\Ä\ZYT\ZaQXbdTW\ Ô ä cl ÔZn %]XZhxVÙWwR[Ya&kZ\XZT>ad°hd hJVY\MVY\QadcRVYT>klL`XZT>x`wRUv\ZwRhJ[`ko$n Ó ä cl ÔnoP)`XbT>x`wRUv\ZwzhJ[ÝwRk}UvhJ[`kbwRxYTWXZT>xÝUvhdXZXZT>Uv\wz\ZYTaQ~MkhJcRVY\ThJVY\VY\TWXbXZhdX Ù cl ÔZn Õ Ô ä cl ÔZ n Þ[Ó ä cl Ôn wRk~T>czhBf 1Ê ×£¢¤µLhdXq~wR[àQXb\ZaQXZdTW\ZkÝlã¶ï-â %I^¶²YâÆad[xÑu³¯\Zadkjkon %~@T>cRhf Ê ×tä²hd\ZYTWXbfwRkT lm|}®ú8\ZadkjkonoP\XZT>ad kaQXZTkZ\hd`T>xåadk kh]hJ[adk \ZYT[YTW\gfihdXZj aQjdT>k ad[wR[UvhdXZXZT>Uv\ 1 `XbT>x`wRUv\ZwzhJe[ %]hdXaQ©\TWX_aJwzdT>[ aŸwR adc@[V`&~TWXihdkZVUWUvT>kZkZwzdTTWXbwzhx`klmkZwzjdT>konoOWÊdÊx`VYXowR[Yq\Xbadwy[`wR[Yk\Zwy[YYP ±IT>aQXb[`wR[àd[`xq\T>kZ\ZwR[YadcR\TWXb[àQ\TdaQL\TWXGT>adUb\XbadwR[wR[Y_k\XZT>adr %¨fiTi©XZTWTWæWTi\ZYTfT>wzJ]\Zkád[`xqdT>[YTWXoaQ\T £

£

£

£


Tï Tï

d¤

32

Out


Output Gate Forget Gate Input Gate

In

®GwzJVYXbTrPts³YXZTWTvágcRadTWXÝ±Ý³¶ \hdhJczhddåfwz\ZåhJ[YTwR[`VY\ad[`x hJ[YThJVY\V`\ V`[`wz\WPåS_T>UWVYXZXbT>[ÙvTwRk cRwy wz\T>x¯\h\ZYT`wRx`xYT>[²cRadTWX %ef`wyUbïUvhJ[`kZwRkZ\ZkhdakZwR[`JczT±G³¶Ì T>hdXZ~MczhUbjfwR\Z²akbwR[YJczTUvT>cRcãP cRce¤ïÞV`[`wR\Ãáä\hQágV`[`wz\ZßUvhJ[`[`T>Uv\ZwzhJ[`kaQXbTqkZYhf[e%Y~VY\~wyadkad[`xTWTWYhJczTUvhJ[`[`T>Uv\ZwzhJ[`kaQXbT&[Yhd\WP a\T>k\k\XZT>adPVYXTWXZLhdXb ad[UvTT>adkZV`XZTwRkI\Z`TadUo`wzTWdT>x\T>k\kZ\XZT>adËkZwzæWTdOWÊdÊdÊ&kZVÙWUvT>kZkZwRdTTWXbwzhx`k aQXZT%xYTWT>T>x²aºÞÃTWXZLT>Uv\Zß khJcRV`\ZwzhJ[EP³IXbadwy[`wR[YwRkk\hd`T>xhJ[ÙvTa \Zadkjwyk_czT>aQXb[YT>xhdXaQL\TWXaÝ aŸwR adc [V`%~@TWXhdOWÊ \XbadwR[`wy[YkZ\XZT>ad k&lÃOWÊ LhdX\ZYT&¶ïâËad[`x^¶ïâà\ZadkjkonoPT>wzJ]\Uoàd[YdT>kaQXbTq adxYT aQL\TWXT>adUbÕ\ZaQXZdTW\`XZT>kZT>[\ZaQ\ZwzhJ[ePÆ³_`TïcRT>aQXb[`wR[YåXbaQ\[ T õ°wRkkTW\\hÕOWÊ ¯ §&fT V`kZT²\Z`T hJT>[]\ZV` adczdhdXowz\Z` lm|cRadVY&\ %^hBfcRade[ %C*ÝÄwy[\hJe[ %O>¤d¡d{Jnfwz\ZÝhJT>[]\ZV` aQXbadTW\TWXÊPt¤d¤d¤LhdX\ZYTqu³ \Zadk j % ÊPt¤d¤©hdX\ZYT2|}®u ad[`xÑ^¶ïâ\ZadkZ:j %Äad[`xåÊPt¤d¤d¤d¤¯©hdX\ZYT2¶ïâó\ZadkZjMPT2XZhJVYJczïhd`\ZwR wzæWT>xº\ZYT hJ T>[\ZVàQXbadTW\TWX~]2\XZwR[YhJVY\x`'w @TWXZT>[]\hdXbxYTWXokhd´ aQJ[`wz\ZV`xYTdP ®`hdX\Zadkjku³2ad[`x¶²Yoâ %\ZYTk\hUbadk\ZwRUwR[YMVY\k\XbT>ad kaQXZTdT>[YTWXbaQ\T>xhJ[cRwR[YTdPGÕTWXZLT>Uv\khJcRVá \ZwzhJ[UvhdXZXZT>Uv\Zczµ`XZhUvT>kZkT>k%OWÊ \T>k\k\XZT>ad k %`\hÝ aQjdTkZVYXZT\Z`T&[YTW\gfihdXZj`XbhwRxYT>k_k\ZaQ~MczTqTWXZLhdXb ad[ÙvT wR[xYTWT>[`xYT>[]\hd_\ZYTµk\XZT>ad~TWJwR[`[`wR[`kZV`cz\Zk%aQXZTa>dTWXbaQdT>k hBdTWX%OWÊ wR[xYTWT>[`xYT>[]\Zcz \XbadwR[`T>x[YTW\gfihdXZjkWP ^¤

¥

X

: ÃØG'Bb=# 6&>!#1/;OWÊ ê lmkZwzæWT ds %¶¯¼ â éòn %adkqkbYhf[wy[¯\Z`T~@hd\\hJÅàdcz}hdi³aQ~MczT rPtsP kiTvT>Uv\T>ex %Y\Xbadwy[`wR[Y%\ZwRTxYT>UvXZT>adkT>k_fwz\Z2wR[ÙvXZT>adkZwy[Y¶¯âP` cRaQXbdTWXkZTW\hdIhJkZkZwz~MczTxYT>cRakkbYhJV`cRx aQjdTÝ\ZYT \Zadkj aQXbxYTWX>PVYXZ`XowRkZwR[YJcR> %MLhdX @cl Øn æ7ç1Ê Ö'×'×'Ödé ê lmkZwzæW T éy®²5O %Ä¶¯â éònfwR\Zp éXbad[`JwR[YLXZhJ O\hrÝ\ZYT\Zadkj¯aQ`T>aQXbk\hµ~T>UvhJTT>adkZwzTWX lmx`VYT\hµ\ZYTkZwRV`cz\Zad[`TWhJV`kwy[ÙvXZT>adkThdi¶¯âqn~@TWLhdXZT\ZYT 3¬

T/

i/

/

E/

_¬

¶/

L¬

·/

H/


Tï Tï

¢Ò

32

MlcØnæ çÊ1Ö>O ç1Ê Ö>5O Öbsê ê ç1Ê Ö'×'×'Öb ê ç1Ê Ö'×'×'Ö¢ ê ç1Ê Ö'×'×'Öbr ê ç1Ê Ö'×'×'Öb{ ê ç1Ê Ö'×'×'Öo ê ç1Ê Ö'×'×'Öb¡ ê ç1Ê Ö'×'×'Öb¤ ç1Ê Ö'×'×'Ö>OWÊ ê ê ç1Ê Öbs ê ç1Ê Öb ê ç1Ê Ö¢ ê ç1Ê Öbr ê ç1Ê Öb{ ê ç1Ê Öo ê ç1Ê Öb¡ ê ç1Ê Öb¤ ê ç1Ê Ö>OWÊ ê /

V

hJcãP OWÊdÊ OWÊdÊ OWÊdÊ OWÊdÊ OWÊdÊ OWÊdÊ OWÊdÊ OWÊdÊ ¤QÊ {QÊ OWÊdÊ OWÊdÊ OWÊdÊ OWÊdÊ OWÊdÊ OWÊdÊ OWÊdÊ OWÊdÊ OWÊdÊ

±G³¶ ³IXoadwR[`wR[Y \X>PM« OWÊ ¬ ¢¡ çO>s sdr å¢ O>ds ×t ds ×£¢ ¡ ×tr ç5O ×t d ¢ ×tr 1Ê ×£¢ < { ×RO ç5O × Ê d ¡ ×tr Áds ×t¤ d OvP« OWÊ ¬ OWÊdÊ ¢{ )Ov¢ OWÊdÊ OW1Ê ×t d ×t OWÊdÊ } ×£¢ ds ×ts OWÊdÊ d ×t{ 1Ê ×£¢ OWÊdÊ d{ × Ê )5O ×£¢ OWÊdÊ } ×RO ds ×t¡ ¨Ê O>r d{ ×tr rQÊ sds ¤ rQÊ d )O sQÊ d¤dr )O>{J OWÊdÊ O>¡ r OWÊdÊ sd { OWÊdÊ O>d × ds × OWÊdÊ OW1Ê ×£¢ ds × Ê OWÊdÊ O>ds × d ×RO OWÊdÊ Ov[\ZwRhJ[àdcEdTWXbkZVk}TWTWYhJcRT±G³¶8hJ[2\ZYT^¶ïâ \ZadkjµfwR\Z RÕïOWÊ ad[`xäQXbwRhJV`kUbYhJwyUvT>k_LhdX_\ZYT&kTW\hd´xYT>cya>k Mcl ØnoP J¬

¸/

x`¤w UWV`cz\ú²wR[UvXZT>adkT>kqXoaQwRx`cz²LhdXcRaQXbdTW2X éZP³]V`k&\ZYT \ZadkZ:j kx`'w UWVcz\gxYhT>k&[Yhd\%dXZhfócRwy[YT>aQXbcz¯fwz\Z \ZYT[V`%~@TWXhdhJkZkZwz~cRT%xYT>cRak%IUvhdXZXZT>khJ[`xwR[Y \hädcRV`T>k lmk\ZaQ\T>konwR[kZwRxYTaUvT>cRcG\ZYT [YTW\úfhdXZjïV`k\ czT>aQXo[Ý\h xwRk\ZwR[YJVwRkZEPÂò[`kZ\T>adxµfThd~kZTWXZdTq\ZàQ\_±G³¶,maQXZT>k~T>k\aQ\_x`wRkZ\ZwR[YJV`wRkb`wR[Y&{hdXxßw @TWXbT>[\ xYT>cya>kWPï[`TÝwRk&\T>`\T>xå\h²x`Xba>fóaUvhJ[[YT>Uv\ZwzhJ[å\h¯\Z`TåÞ aQJwyUÝ[V`%~@TWXoßhd çs¯wz\T> k%\ZàQ\ad[ adTWXbaQdT V` ad[ïUWad[ k\hdXZTwR[ `hdXZ\³TWXbÉ¶T>hdXZlä³¶ïnlã¶wRcyczTWX %GO>¤drd{Jn %I~VY\kZVUbïacRwR[YjkZTWT> k XbaQ\Z`TWX_LaQXZáä©TW\ZUoYT>x\h VkWP Tadcykhhd~kTWXZdT\ZaQ\\ZYTXZT>kZV`cR\Zk%LhdX @cl Øn æzç1Ê Ö>O ê aQXZT~TW\\TWX \Zad[ \ZYhJkThd~`\ZadwR[`T>x fwz\Za kZwRJhJwRxLV[Ùv\ZwzhJ[lmUvhJMaQXZT³áQ~czTrPROnoP}®GcRVÙv\ZVaQ\ZwzhJ[`kÄwR[\ZYTk\hUoàdk\ZwRUwR[YVY\ÄUWad[ UWadVkT_\T>hdXbaQXZ kZaQ\ZV`XbaQ\ZwzhJ[hdGkZwzJhJwyxV`[`wz\Z&k §Y\Z`TXZT>kZV`cz\Zwy[Y\ZwR[µxYTWXbwRäQ\ZwzdT>k_©hdX_\Z`Tq~adUZjfaQXbxadkZkfwRcRc@kZczhBf xYhBf[ czT>aQXo[`wR[Ylm±eTBVe[ %`Çihd\\hJeV %XbX %W* ¶ V`cRczTWX %EO>¤d¤d¡JnoP °¬ ^ Ïµ=§:ª ZeÔKLEÔ S ±G³¶ÍczT>aQXo[YT>xº\hïT>adkZV`XZT \ZwR T2wR[º\gfihï`XbwR[ÙWwRczT>xïfa>kWP²³YT `Xok\%wRk&\h kZcywzJ\ZcRÝwR[ÙvXbT>adkT\ZYT%UvT>cyceUvhJ[\T>[]\Z)k aQ\T>adUb¯\ZwRT&k\TWG %kh \ZàQ\\ZYTqT>cRaQMkT>x\ZwRTqUWad[¯~TXZT>adx5h \ZYTQadcRVYThd P&³_`wRkjwR[`x¯hdkhJcRV`\ZwzhJ[ïwRkkZYhf[¯hJ[ï\Z`TcRTW©\Ãágàd[x²kbwRxYT&hdi®GwzJVYXZT rP£¢YP2lm³YTk\ZaQ\T XZT>kZTW\&TWXZLhdXbT>x~\ZYT LhdXZdTW\JaQ\TwRkqT>kZkZT>[\ZwRadc}hJ[`czïLhdXUvhJ[]\ZwR[VàdchJ[`cywR[YT `XbT>x`wRUv\ZwzhJ[hBdTWX ad[ TWXbwzhx`kWP n ³_YT kT>UvhJ[xfiawRk\hT>k\ZaQ~cywRkZïwR[]\TWXb[àdcGhJkZUWwRcycRaQ\hdXbkad[`x x`TWXbwzdT\ZYTT>cRaQkZT>x\ZwRTLXZhJ \ZYT>wRXMàdkT>klLXbwzJ]\Ãágàd[`xkbwRxYT2hd®GwzJVYXbTrP£¢]noPÇihd\ZjwR[`x`k%hdkhJcRVY\ZwzhJ[kUWad[Á~@TµczT>aQXb[`T>x fwz\ZåhdX fwR\ZYhJVY\TWTWYhJczTÝUvhJ[`[YT>Uv\ZwzhJ[`k %ádkwz\&wRk%[YTWdTWX[YT>UvT>kZkZaQXZ `TWXZT \h²UWczhJkT \Z`T hJVY\V`\qJaQ\TÝLhdXhdXZT \Zàd[hJ[YT\ZwRTqk\TW lmkTWTq~hd\\hJXbhfÕhdÄ®GwzJVYXZT&rP£¢]noP ³

]/

±¹

¢s

×]êëìÚGÛí

Tï


1

1

0.5

0.5

in k tk y

0

0 20

1 10 sc c y

0

0

1

1

y

y

in

out

yϕ

0

0 0

10

20 Time

30

0

10

20

30

Time

®GwzJVYXbTrP£¢Y ¦&¿C²¿§6ZeÔ%?Cï?BKmÐÑ¸S ³IT>kZ\iXoV`[ fwz\Z \XoadwR[YT>x2±´³¶É[YTW\gfihdXZjk©hdX\ZYT¶²YâÕ\ZadkjÝfwz\Z RÕOWÊ&ad[`x MlcØnæWçÊ1Ö>O ê Pi³Ihde´\ZaQXZdTW\iädcyVYT>kÔ ad[`x [YTW\úfhdXbjhJVY\VY\Ó § wRx`x`czTdÚvT>cRc`k\ZaQ\Tc ad[`x UvT>cRchJV`\VY\ Ó §`~hd\\hJ}adUv\ZwzQaQ\ZwzhJ[hd\ZYT%wR[YVYä \iJaQ\TsÓE %Y©hdXZdTW\JaQ\TsÓ ä Ê %ad[`xhJVY\VY\JaQ\ToÓ>¡D¢òP ¬

´/

)]Ñ aÑ\ZYTµhJVY\MVY\JaQ\T~@TczTWL\hdT>+[ Ç ³aQXZdTW\Zk hUWUWVYXXoaQXZT>cz> %}YT>[UvT2\Z`Tµ[`TW\gfihdXZjÑhJV`\VY\ WU ad[µ~TwzJ[YhdXZT>xµhJk\hd\ZYT\ZwRTdPwy[ÙvT\Z`TWXZTwRkihJ[`cz hJ[YTq T>hdXZ ~MczhUbj:%YVY\ZVàdc@TWXZ\ZVYXZ~MaQ\ZwzhJ[ hd ~cRhUbjk´wRkG[Yhd\GhJkZkZwz~cRTdPI³_`wRk\gThdMXZT>adkhJ[`wy[Ywyk´wy[QadcRwRx&\ZYhJVYJ©hdXÄhdXbTUvhJ czTv%T>adkbVYXbwR[Y\Zadkjk wR[]dhJczwR[YcRaQXZdTWX[YTW\Zk_hdXhdXZTqLXZT>VYT>[]\\ZaQXZdTW\ZkWP®GwzJVYXbT&rPtr kZ`hfk\Z`T&~T>à>wzhdX_hdÄ±G³¶wR[kZVÙo aXZTWJwR TdPÄ-wz\ZµTWTW`hJczT±G³¶É\ZYThJVY\VY\JaQ\ThdT>[`khJ[`cR fYT>[a\ZaQXZdTW\_wyk`XZhwRxYT>Px %]f`TWXZT>adk UvhJ[]dT>[\ZwRhJ[àdcq±Ý³¶pxYhT>k[Yhd\czT>aQXb[-\Z`wykÝ~T>àwRhdX>P ^hd\T \ZàQ\wR[-khJT UWadkZT>k\ZYT>kTÕÞUWczT>ad[`TWXbß khJcyVY\ZwzhJ[`kfwR\ZTWTWYhJczTUvhJ[[YT>Uv\ZwzhJ[`k \h]hdjÁczhJ[`dTWX \hº~TcRT>aQXb[YT>x lmUvhJMaQXZT²³áQ~czT>kÝrPRO¯ad[`x-rPtds % ~T>UWadV`kT\Z`TWÝXZT>V`wzXbTqhdXZTqUvhJcRTv2~T>àwzhdXP


Tï Tï

¢

32

3 2

in tk yk

1 0

2 1 0

4

0

in tk k y

sc y

c

-2

0

sc c

y 1

1 yin

yin

out

y

yout yϕ

ϕ

y

0

0 0

10

20

30

40

Time

50

60

70

Time

G® wzJVYXbTirPtrÄÇiT>à>wzhdXGhdTWTWYhJcRT±Ý³¶ÈlmczTWL\onedTWXokZV`kG±´³¶ÈlLXbwzJ]\oneLhdXG\ZYT¶²â\Zadkj&fwz\Z ÕïOWÊ ad[`x @lcØn³æWçÊ1Ö>O5Öbs ê P³IhdeÄ\ZaQXbdTW\_ädcyVYT>k"Ô ad[`xµ[YTW\gfihdXZj2hJVY\VY\qÓ §` wRx`x`czTdGUvT>cRck\ZaQ\T_ ad[`xUvT>cRc hJVY\MVY\Ó §Y~@hd\\hJ}adUv\ZwzQaQ\ZwzhJ[hdG\ZYTqwR[`VYä \JaQ\TsÓE %`LhdXZdTW\_JaQ\ToÓ ä Ê %àd[`xhJVY\MVY\JaQ\TsÓ5¡l¢ÃP H¬

º/

:T ÎW!3G 'Ò$& !317Â@& 7( ;k\ZYTXZhJczT>khdwy[YVY\Zkad[`xï\ZaQXZdTW\Zkhd\ZYT ¶²â \Zadkj@\Z`TkZwzjdT\XbadwR[ lcØn% xY&T M[YT>xadkLhdXq\ZYTÝ¶ïâÆ\Zadk j %´[YhBf°wRk\ZYTÝ[YTW\gfihdXZ j kq\ZaQXZdTW&\ %Gf`wyczT%\Z`T xYT>cRa> @lcØnwRk`XZhBwyxYT>xadk wR[`VY\WP x B:7>;~T&cRT>aQXb[YT>x~[YTW\gfihdXZjkfwR\ZYhJVY\TWTWYhJczTqUvhJ[[YT>U á ÷¦¬ö2¸]ÔF´ª©?EÔ S ³_YTu³2\ZadkjUvhJV`cyé \ZwzhJ[&k §\ZV`k&fTÝXZTWhdXZ\qXZT>kZVcz\Zkfwz\Z @TWTWMYhJczT ±Ý³¶9hJ[`czdPµS_T>kZV`cR\Zkqfwz\Z äQXbwRhJV`kq wR[wR%VÈkwzjdT wR[]\TWXZQadcRk ,lm®GwzJVYXZTrPt{Jn_kZV`ddT>k\\ZaQ\\ZYT%XZT>]VwzXZT>x\Xbadwy[`wR[Y\ZwR T&wR[ÙvXZT>adkZT>kxYXoad aQ\ZwRUWadcRcz2fwz\Z % adkfwz\Z \ZYT^¶²Yâç\Zadkj¯läT>Uv\ZwRhJ[2rPtrPtsJnoPi³_YT[YTW\úfhdXbj hJVY\VY\ix`VYXbwy[Ya%kZVUWUvT>kZkmV`c`\T>k\XbV`[ ©hdXi\ZYT u³¯\Zadkj¯fwz\Z ZÕOWÊwRkkZ`hf[ïhJ[\Z`T\hdºczTWL\hd®GwzJVYXZT rP ]P |´TWTWYhJcRT ±´³¶adcRkhkhJcRdT>k\ZYT \ZadkjµLhdX RÕïOWÊ ad[`x Mcl Øn æpç1Ê Ö>O ê hdcX ç1Ê Ö>5O Öbs ê %Eadk_kbYhf[wR[®GwzJVYXZT&rPt{lmcRTW©\onoP _f

·/

¸¬

_¬

7¬

¬

`/

¢d¢

Û Ù òÙÄ¥XìíÛ×]ðÜÛàÚ$ð gðÃÙ¥ØRð>Ú]ê ì}Û}Ûì_êØqÿIÛÿ´ÜeÚ

×]êëìÚGÛí

Tï

ÿ _ëí ð

/

WO Ê sQÊ QÊ ¢JÊ rQÊ OWÊ OWÊ

6

Training Streams [10 ]

MlcØn³æ |´TWhJTWcãP YhJ³cRTqXbad±´wR[EP8³« ¶ OWÊ ¬ çÊ ê OWdÊ Ê ¢Ò å¢ çÊ ê OWdÊ Ê {J ¡ çÊ ê Q¡ Ê ¡¨¢r ¡ds çÊ ê OWÊdÊ OdO>rds )OWÊYO çÊ ê OWÊdÊ sdrdd¡ ¨¢ ç1Ê Ö>O rQÊ O>{¨¢] å¢{ ç1Ê Ö>5O Öbsê ê QÊ ¤dr¨¢ d¤d

¬

z

V

B³

¸³

[\ZaQdThdeTWXZLT>Uv\ikZhJcRVY\ZwzhJ[`kLhJV`[`xe%ad[`x2\ZYT T>ad[2ad[xÝk\Zad[xàQXbx2xYTWXbwzQaQ\ZwzhJ[ hde\Z`T[V`%~TWXihd\XbadwR[`wy[Y%k\XbT>ad kXZT>V`wzXZT>xEPuXbaQMïlLXowzJ\on}cRhd\Zk\Z`T[V`%~TWXihd\XbadwR[`wy[Y%k\XbT>ad k aQJadwR[k\\ZYT& wy[`wRV` kZwzjdTqwR[]\TWXZQadc %YLhdX MlcØn³æpçÊ ê P @¬

/

µ¬

/

÷¦¬)Ï2=§:ª ZeÔKLEÔ S ®GwzJVYXbTrP ÝkZ`hfk\T>k\XbV[`kfwz\Z²\XbadwR[YT>x²[YTW\gfihdXZjkLhdX\Z`T u³\ZadkjMP³_YT hJVY\MVY\JaQ\T>khd@T>[hJ[`czµaQ\\ZYT%hJ[`kTW\hdGaÝkwzjdT%ad[`xUWczhJkZT&aQJadwR[²wR T>x`wyaQ\T>czÝaQL\TWXZfaQXbx`kWPÄT>[ÙvT>% x`V`XbwR[Y a2kwzjd>T %\ZYThJVY\V`\hd\ZYTUvT>cRcT>]VadcRkwz\ZkkZ\ZaQ\Tlm wRx`x`czT&XZhfËhd}®GwzJVYXbTrP dnoPq³_YT%hdT>[`wR[Y hd\ZYT hJV`\VY\qJaQ\TµwRk\XbwRddTWXZT>x ~ï\ZYTÝUvT>cRc}k\ZaQ\@T %wz\%k\ZaQXZ\Zkq\h¯hdT>[ hJ[ÙvTÝ\ZYT wR[YMVY\©XZhJÅ\ZYT TWTWYhJcRTµUvhJ[[YT>Uv\ZwzhJ[hJVY\úfT>wRJ`k a[YTWJaQ\ZwRdT~wRadk>P ³YThdT>[`wR[YïkT>cz©áäXZT>wR[YLhdXbUvT>kwRaa UvhJ[`[`T>Uv\ZwzhJ[ LXZhJ8\ZYT UvT>cRc´hJVY\V`&\ %Ef`wRUo¯XZhxVÙvT>k\ZYT `wzJï[YhJ[`cywR[YT>aQXbwz\ú[YT>UvT>kZkZaQXbLhdXdT>[YTWXbaQ\ZwR[Y\Z`TkZwzjdTdP ³_wRk`XZhUvT>kZkwRk_\TWXb wR[àQ\T>x¯~]µ\ZYTUWczhJkZwR[` hdÄ\ZYT&LhdXZdTW\JaQ\>T %\XbwzddTWXZT>x¯~]\Z`T%UvT>cRcIhJVY\VY\kZwzjdTdP wy%V`cR\Zad[YTWhJV`kZcz\Z`T&wR[YV`\JaQ\TUWczhJkT>&k %kZh \ZàQM\ wRkXZT>kTW\WP ÂÃ[ \Z`TÝaQXb\ZwRUWV`cRaQX&kZhJcRVY\ZwzhJ[ÑkZYhf[ºhJ[Ñ\ZYT2XbwzJ]\Ãágàd[`xºkbwRxYT hd_®GwzJVYXbT2rP ©hdX ¥Õ"rQ1Ê %Ä\ZYTÝXZhJczT hd_\Z`T2LhdXZdTW\JaQ\TwR[Ñ\Z`wRk`XbhUvT>kZkwRk%\ZaQjdT>[hBdTWX ~]åa¯[`TWJaQ\ZwzdTkT>cz©áäXZT>UWVYXZXbT>[\UvhJ[`[YT>Uv\ZwzhJ[ hd\ZYT UvT>cRc@wR[2UvhJV[ âZV`[Ùv\ZwzhJ[Ýfwz\Z2akZwRV`cz\Zad[YTWhJV`khdT>[`wR[Y%hde\ZYThd\ZYTWX\gfihJaQ\T>kWPT\T>[]\ZaQ\ZwzdT>czÝXZT>hBdT>x \ZYTLhdXZdTW\_JaQ\TÝlL~2wR[`[`wy[Y&wz\ZkadUv\ZwzQaQ\ZwzhJ[\hOQP Ênfwz\ZYhJVY\Uoàd[YJwR[Y\ZYTfiT>wzJ]\Zk_czT>aQXb[`T>R x ^!?@;c\ZYT LhdXZdTW\JaQ\>T k`T>czeP³_YT&[YTW\úfhdXbj\ZYT>[V`wRUbjcR2czT>aQXo[YT>xa @TWXb©T>Uv\khJcRVY\ZwzhJ[ePi±IT>aQXb[`wR[Y ©XZhJ kZUvXbaQ\ZUo ^!?@;cd75. T %iYhfiTWdTWX %}[YTWdTWXwzT>cRxk&aïkhJcRVY\ZwRhJ[ ¯³_`TÝLhdXZdTW\JaQ\TwRk%T>kZkT>[]\ZwRadcx`VYXbwR[`\ZYT czT>aQXo[`wR[Yàdk>T %fYTWXZTwz\`XbTWdT>[\Zk\ZYT&adUWUWV`V`cRaQ\ZwzhJ[hd´wzXbXZT>czTWQad[\TWXZXbhdXbkWP ³YTTvYadUv\µ\ZwR wR[YÑhd&aåkwzjdT²wRkÝx`TW\TWXb wR[YT>x"~\ZYT¯dXbhf_\Z)hd_ %_f`wyUb-wRk \ZV[YT>x\Z`XZhJVYJ UvhJ[`[`T>Uv\ZwzhJ[`k%\h²wR[`VY\qJaQ\>T %LhdXZdTW\JaQ\>T %}ad[`xÑ\ZYTµUvT>cRciwz\ZkT>czòP³h²khJcRdTu³LhdX @cl ØÄn æ ç1Ê Ö>O ê hdX @cl Øn æWç1Ê Ö>5O Öbs ê %M\Z`Tq[YTW\gfihdXZj2T>kZkT>[]\ZwRadcRczÝ\Xbad[`kZcRaQ\T>k\Z`TqwR[YVY\wR[]\hakZUWadcRwR[YmadUv\hdX_LhdX_\ZYTdXZhBf_\Z hd lm®GwzJV`XZT&rPt¡JnoP I¬

@

»/

/

6&(3&>*pË#=3!C*K$&"! Îp!#G 6'+$& "! @Ë)Î x

x

w

¼¨

ª©

T[YhBf \Xbadwy[¯±G³¶\hµdT>[YTWXbaQ\TXZT>adc áäQadcRVYT>x¯TWXbwzhx`wRUmV`[Ùv\ZwzhJ[k&%Madkhd`hJkT>x¯\hµ\ZYTkZwzjdT&\XoadwR[`k hd\ZYT²u³å\Zadkj@P-\ÝT>adUb-x`wRkZUvXbTW\T\ZwRTkZ\TWfiT`XZhBwRx`TaXZT>adc áäQadcRVYT>x\ZaQXbdTW\&%kZadcRT>x fwz\Z LXZT>VYT>[Ùv °©XbhJa\ZaQXZdTW\mV`[Ùv\ZwRhJ[ràÄlcÔnoP}^h wR[`VY\wRkJwzdT>[\h \ZYT&[YTW\úfhdXZj@P ³YT\Zadk j kxYTWdXZTWThdex`'w ÝUWV`cz\g wyk}wR[ÙvT>x ~\ZYTkZàQThd$ àad[`xÝ\ZYTkZadcRwy[YqLXZT>]V`T>[Ùv P ³_`T_©hdXoTWXiUWad[Ý~T_aQXZ\ZwyadcRczUbàQXbadUv\TWXowzæWT>xµ~]\ZYTaQ~khJcyVY\T aŸwR adcädcRV`T>k}hdewz\Z_k `Xbk\iad[`x kZT>UvhJ[`x xYTWXowzäQ\ZwRdT>&k %@ a¨ à å ad[`xï a¨ à å P%wR[ÙvT%fTfihdXZjwR[²x`wRkbUvXZTW\T%\Zwy>T %Ead[`x²fwz\Z²[YhJ[ágx`ßw TWXZT>[]\ZwRaQ~czT ´¬

£

²¬

£

£

£


Tï Tï

¢r

32

1

1 k

tk yk

t yk

0

0

0 2 -4

sc yc

sc c

y

0

1

1 yin

y

yin

0

out ϕ

y

yout yϕ 0

0 10

20

30

0

50

Time

100

150

Time

®GwzJVYXbTrP ]q³T>k\XbV[¯hdaµ\XbadwR[`T>xï@TWTWMYhJczT±´³¶[YTW\gfihdXZj²©hdX&\ZYT u³¯\Zadkjïfwz\Z @lcØn]æ çÊ ê % ad[`xåa² wR[`wRV`ëkwzjdT2wR[\TWXbädchd Õ OWÊ lmcRTW©\onqk>P Õ-rQÊålLXowzJ\onoP³hde \ZaQXZdTW\QadcRVYT>koÔ ä ad[`x [YTW\úfhdXbjhJVY\VY\ Ó ä § wRx`x`cRTdUvT>cRcMk\ZaQ\)T ad[`xÝUvT>cRc`hJV`\VY_\ Ó §~hd\\hJÄadUv\ZwzäQ\ZwRhJ[ÝhdE\Z`TwR[YMVY\GJaQ\T ÓE %`LhdXZdTW\_JaQ\o T Ó Ê %àd[`xhJVY\MVY\JaQ\sT Ó5¡l¢ÃP 7/

¬

>¬

k\TWLV`[Uv\ZwzhJ[`k&%fiT&xYT&[YTd à

å l Zni Äl l®µOn cÔ

èÕ¹à cÔ

Äl Zn

]ÞNà cÔ Ö

a¨ £

à

å

£½

¢ a¨ £

à

å l Zn cÔ

£

Ö

a¨ £

à

å

¢ a¨

£^½

£

à

å l w®On cÔ

Äl Zn å

]Þ{à cÔ

£

×

uT>[`TWXbadcRczkZ@T>aQjwR[YkTQadcRVYT>k %E\ZYT àQXbxYTWX\Z`T\ZadkjMP óxYTW\TWXo wR[YT>k\ZYT [V`%~TWXhd `x wyk\ZwR[YJV`wykZàQ~czTwR[\TWXo[àdcYk\ZaQ\T>kXbT>]V`wRXZT>x\hqXZTW`XZT>kZT>[\\Z`T@TWXowzhxwRU}mV`[Ùv\ZwRhJ[wR[wR[]\TWXb[àdcYk\ZaQ\TkZadUvTdP ³_`T&cRaQXZdTWX %M\Z`T%àQXbx`TWX\ZYT%\ZadkjMPTdT>[YTWXbaQ\TkZwy[YTqfa>dT>knà UvhJk %M\XowRad[YJV`cRaQXLV`[Uv\ZwzhJ[`kià \Xbw %ad[`x XZT>Uv\Zad[`JV`cRaQX}mV`[Ùv\ZwRhJ[`k_à XZT>Uv\ %adcRc`Xbad[`JwR[Yq~TW\gfiTWT>[2Ê1× Ê%ad[`x¯O5× Ê1%]T>adUoµkZadczT>x fwz\Z \úfhLXZT>]V`T>[ÙWwzT>k&% i¬

Ú]ê ì}Û}Ûì_êØqÿIÛÿ´ÜeÚ ÿ _ëí ð

in k t yk

2 1

k

y and t

k

Tï

Input: in

¢{

0

0

sc

2

yc

0 -2

1

y

in

yout yϕ 0 50

60

70

80

90 100

Time

G® wzJVYXbTrPt¡³IT>kZ\XbV`[ïhda\XbadwR[YT>xïTWTW`hJczT±´³¶[YTW\gfihdXZjïLhdXq\ZYT2u³¯\Zadkjïfwz\Z Õ"OWÊad[`x @lcØnÄæ/çÊ1Ö>O5Öbs ê P ³IhdI \ZaQXZdTW\ädcyVYT>ksÔ ä ad[`x [YTW\úfhdXZjºhJVY\VY\-Ó ä §} wRx`x`czTdUvT>cRcik\ZaQ\Tk ad[`xåUvT>cRc hJVY\MVY\Ó §Y~@hd\\hJ}adUv\ZwzQaQ\ZwzhJ[hdG\ZYTqwR[`VY\JaQ\TsÓE %`LhdXZdTW\_JaQ\ToÓ Ê %àd[`xhJVY\MVY\JaQ\TsÓ5¡l¢ÃP ·¬

/

OWÊ ad[`x RÕsdr O O ÞºUvhJk s NÔ à UvhJk cl Ôn × Ù8Ù s × ¢ ß wzçlcÔG hx %n æ Þ à \Xbw lcÔn s Þ æ Þ ¢ ß hd\ZYTWXZfwykT n O wz)cl ÔG hx n ² æ à XZT>Uv\ lcÔn Êóhd\ZYTWXZfwRkT

¬RÕ

`¬ ½

½ÂÁ

½ÆÅ

õ

a¨ à vU å hJk ) Õ a¨ à v U å hJk

õ

a¨ à \ å Xbw ç Õ s

õ

a¨ à ZX å T>Uv\ ) Õ a¨ à Z Õ O~× X å T>Uv\ Ë

¿¾

¬

: Ä FÃ À Ä À : FÃ Ä Ä

i¬

»f

Ä

² Ä

£

æ

£

£

£

£

£

£

êÀ¬

£ ® Õ ¾meÀ¬

a¨ à \ å Xow Õ ¢

Ö

Ö

£

£

£

£

eÀ¬Ö


Tï Tï

¢]

32

±Ý³¶ ® ³IXoadwR[`wR[Y ¨]ã hJcäP \X>P« OWÊ ¬ WO Ê Q¤ Ê s¨]¢ d ¨¢Ò Ê1×RO> Ê1× ÊJd à UvhJk sdr Ê ²ÕOWÊdÊdÊdÊ 9 W O Ê Ê Õ ² O W d Ê d Ê d Ê Ê 9 à \ Xow sdr Ê ²ÕWO ÊdÊdÊdÊ 9 W O Ê Ê Õ ² O W d Ê d Ê d Ê Ê 9 à ZX T>Uv\ ds r Ê ²ÕWO ÊdÊdÊdÊ 9 \d\WP m[EP

V

z

g

³

Ç

P« OWÊ ¬ Ov¢r ds Ê1×RO>¡ Ê1× ÊYO>{ Ov¢¤ Á Ê1×RO Ê1× ÊYO>¤ ¡d{d¤ sQÊQ¢ Ê1×RO> Ê1× ÊYOv¢ ¢JÊJ{d QÊJ 1Ê ×RO> Ê1× ÊJs¨¢ OdOWÊ ¤J 1Ê ×RO>s Ê1× ÊYOv¢ B¢¡ sJQ¡ 1Ê ×RO>s 1Ê × ÊYO>s z

g

³

Ç

Uv\khJcRVY\ZwRhJ[`kLhJV`[`xP%Mad[`x¯\ZYTT>ad[ïad[`x¯kZ\Zad[`xàQXbx²xYTWXbwzQaQ\ZwzhJ[hd\ZYT[]V`%~TWXhd \Xbadwy[`wR[Yµk\XZT>ad kqXZT>]VwzXZT>xe%EadkfiT>cRcÄadkhdi\ZYTXZhhd\qT>ad[ kZVàQXZT>xïTWXbXZhdX z¨]ãLhdXq\ZYTM[àdc´\T>k\ XbV[EP §¬

»g

Ç

¾&»_÷ìö2¸]ÔF´ª©?EÔ S VYX%TvTWXbwyT>[\ZadcXZT>kZV`cz\Zk&©hdX%\ZYTÝ|®u \Zadkj aQXZT2kZV` aQXowzæWT>x wR[ ³áQ~czTµrPtP ´| TWTWYhJcRT¯±Ý³¶!©hJV`[xTWXZ©T>Uv\&%_kZ\ZaQ~czTkhJcyVY\ZwzhJ[`k ©hdX2adcRc_\ZaQXZdTW\2mV`[Ùv\ZwzhJ[`k²lm®GwzJVYXZT¯rPt¤JnoPà±´³¶ fwR\ZYhJVY\TWTWYhJczT%UvhJ[`[YT>Uv\ZwzhJ[`kUvhJV`cRx¯khJczdT&hJ[czÃà UvhJk fwz\Z NÕºOWÊ1%XZT>V`wzXowR[Y ad[]hdXZT&\XoadwR[`wR[Y k\XbT>ad kWP )wz\ZYhJVY\&©hdXZdTW\%JaQ\T>k&%Ä±´³¶[YTWdTWXczT>aQXb[`T>x\h`XbT>x`wRUv\\ZYT fa>dTWLhdXbLhdX%hdXZT \Zàd[ \úfh kbVÙWUvT>kZkZwzdT&@TWXowzhxkWP ³YT2xVYXbaQ\ZwzhJ[Ñhd\Xbadwy[`wR[YXbhJVYJ`cz Xb&T `T>Uv\T>x hJVYXUvXbwz\TWXbwyaLhdX\ZadkjÑx`¤w UWV`cz\údP²TxwRxº[Yhd\\XZ \hïadUb`wRTWdTµ a¨wR adc}adUWUWVYXbadUvÑ©hdXT>adUbå\Zadkj@ \Xbadwy[`wR[Yfadkk\hd`T>x hJ[UvTÝ\ZYT ÞÃTWXZLT>Uv\khJcyVY\ZwzhJ[`ß UvXbwR\TWXbwRafiTWXZTqmV`'c cRcRT>xEPUWUWV`XbadUvUWad[~TqwRXZhdT>x~]xYT>UvXZT>adkbwR[Y\ZYT%\hJczTWXbaQ\T>x a¨wRV`óhJV`\VY\ TWXZXbhdqX Ù Ïä ° x`VYXbwy[Yq\XbadwR[wR[Ywz\aQ\akZwzJ['w UWad[]\}wR[ÙvXZT>adkZTwR[Ý\XbadwR[`wy[Yqx`VYXbaQ\ZwRhJ[EPÄâT>UvXbT>adkZwR[Y Ù Ïä ° ~hJ[YTàdcRÄlL\h1Ê ×RO>rJn©hdX à UvhJk fwR\Z RÕsdr%adcRkhqXbT>x`VÙvT>k\ZYTadTWXbaQdT ¨!ã hdekZhJcRVY\ZwzhJ[`k~]aQ~hJVY\ hJ[YTàdcR %LXZhJÈ1Ê ×RO 1Ê × ÊYO>¤xYhf[ï\h1Ê × ÊJ¡d{ "1Ê × ÊdÊJsP|´TWXZ©T>Uv\qkZhJcRVY\ZwzhJ[`kfiTWXZT czT>aQXb[`T>x²wy[²adcycÚWadkZT>&k % ~V`\ihJ[cz2aQL\TWXlãsJ¨ÊQ¢ å¢¤Jn JOWÊ \XbadwR[`wR[`kZ\XZT>ad &k %adkhd`hJkT>xµ\h²lÃOv¢¤ Ádn JOWÊ \XoadwR[`wR[Yk\XbT>ad k lLwzT>cRx`wy[Y{QÊ khJcRVY\ZwRhJ[`kon}~TW©hdXZTdP ¾&»_÷8Ïµ=§:ª ZeÔKLEÔ S ®YhdX}\ZYT_|®u)\ZadkZ:j %]\ZYT_[YTW\gfihdXZjk}xYhq[`hd\à>dTad[]Tv\TWXb[adc`wR[YV`&\ %JkhqVYExàQ\T>k xYTWT>[`x hJ[Ý\Z`TwR[]\TWXb[àdcMUvT>cRcMkZ\ZaQ\T>khJ[`czd!P ÄT>[Uv>T %wy[ ak\ZaQ~czTkhJcRVY\ZwzhJ[ ©hdXa&@TWXowzhxwRU\ZaQXZdTW\LV[Ùv\ZwzhJ[ Ô cl ÔZn@\ZYT}UvT>cycdk\ZaQ\T>< k adcRkh_a>dT}\h©hJcRcRhfºkhJ TTWXbwzhx`wRU\Xbha âÃT>Uv\hdXb2 ]cl ÔZnMàdkTvágcRhUbjdT>xq\ih Ô cl ÔZnoPwR[ÙvT \ZäYTUvT>cRcMhJVY\VY\wRki\ZYThJ[`cz\ZwR TváääQXZwR[YwR[YMVY\\hJaQ\T>k_ad[`xÝhJVY\VY\V`[`wR\Z&k %]wz\V`k\kZwRV`ä cz\Zad[YTWhJV`kbcz wy[`wR wzæWT%\ZYTTWXZXZhdX%aQ\\ZYT hJVY\V`\V[`wz\Zkad[`xï`XbhwRxYTadxYT>VàQ\T wR[YMVY\\h\ZYTJaQ\T>kWP [ïTvad czT hdGYhf \ZYT>kTq\úfh XZT>]VwzXZT>T>[]\_UWad[~TqUvhJ%~wR[YT>xwR[µhJ[`T&khJcRVY\ZwRhJ[wRk_kZYhBf[wR[®GwzJV`XZTqrPROWÊ LhdX à UvhJk fwR\Z RÕïOWÊP³_`wRki\ZadkjUWad[~TqkhJcRdT>xfwz\ZµhdXfwz\ZYhJVY\TWTW`hJczTqUvhJ[`[YT>Uv\ZwzhJ[k~T>UWadV`kTq\ZYThJV`\VY\ JaQ\T[YTWdTWX[YTWT>xk\h²~TµUWczhJkT>x UvhJcRTW\T>cz> %}kh²\ZàQ\ adcRcJaQ\T>k UWad[ ~adkT2\ZYT>wzXUvhJ[]\XZhJchJ[å\ZYTUvT>cRc hJVY\MVY\WP )] x`wRx ±G³¶,[YTW\úfhdXbjkfwR\ZYhJVY\TWTWYhJcRTUvhJ[`[YT>Uv\ZwRhJ[`k[YTWdTWXczT>aQXb[ \Z`T\ZaQXZdTW\mV`[Ùv\ZwzhJÃ[ à UvhJk LhdX ÕÑsddr %adcR\ZYhJVYJï\ZYTWïx`wRx²czT>aQXb[wz\LhdX ZÕÁOW6Ê ÇÕ³_YThJVY\VY\JaQ\T wykaQXZ\hdiad[V[ÙvhJ[\XbhJcRczT>x LTWT>xY~adUbj cRh]hdewR\Zk adUv\ZwzQaQ\ZwzhJ[x`wRXZT>Uv\ZczÑx`TW\TWXb wR[YT>kwR\Zkhf["wR[YV`\µlmYTWXbTdwz\Zg k 75BW4è¿wR[YMVY&\ %TvYUvTW`\ LhdX\ZYT~wyadkon&wRaï\Z`TUvhJ[`[YT>Uv\ZwzhJ[\h \ZYTUvT>cRchJVY\MVY\ 9 ~VY\[YhTWXbXZhdXbk aQXZTXZhdaQJaQ\T>xÁ~MadUZj hJ[ \Z`wyk UvhJ[`[YT>Uv\ZwzhJ[eP-³_`TkZadT²wRk\XbVYT©hdXÝ\ZYThd\ZYTWXÝJaQ\T>&k %_TvYUvTW`\ \ZaQ\ hJVY\V`\ JaQ\ZwR[YåUWad[Á~czhUZj \ZYT>wRX lL\Z]V`kwy[ÙvhJczTW\TBnLTWT>xY~adUbjczh]hdIP³_wRk aQjdT>kqad[ adxaQ`\ZwzdT ±Ý³¶ T>hdXZ²~czhUZjfwz\ZYhJVY\ TWTWYhJcRT%UvhJ[`[YT>Uv\ZwRhJ[`khdXZT x`¤w UWV`cz\\h2\ZV[YTdP&x`x`wz\ZwzhJ[adckZV``hdXZ\LhdX\Z`wRkXZT>adkhJ[wR[Yµk\T> kLXZhJ \ZYTmadUv\\ZàQ\[YTW\gfihdXZjkfwz\ZÝTWTWYhJczTUvhJ[`[YT>Uv\ZwzhJ[kczT>aQXbw[ à UvhJk fwz\Z RÕïOWÊVÙbµmadk\TWX&lmkTWT³áQ~czT rPtJnoP³_YTTWTWYhJczTfiT>wzJ\ZkqhdikhJcyVY\ZwzhJ[`kaQXZT\gwRUWadcyczhdi\ZYTkZadT aQJ[`wR\ZV`xYTadk\ZYTfT>wRJ\Zkhd `¬

BÈ

È

²¬

.³

³

N5

z

Eg

5

z

¢¡

×]êëìÚGÛí

Tï


tk yk

1

0

0

1

tk k y

1

tk k y

0

0

1

tk yk

1

k 1 tk y

tk k y

0

0 0

10

20

30

40

Time

0

25

50

75

100

Time

G® wzJVYXbTrPt¤³aQXZdTW\ädcRV`T>kqÔ ä ad[`xµ[YTW\gfihdXZjµhJVY\VY\qÓ ä x`VYXbwR[`&\T>k\XbV[`kihd\Xbadwy[YT>x2@TWTWMYhJczT±´³¶ [YTW\úfhdXbjkhJ[²\ZYT%|®u°\ZadkjLhdX\ZYT%TWXbwzhx`wRUmV`[Ùv\ZwzhJ[`kà UvhJk lL\hd@n%Pà \Xbw lm wRx`x`cRTBn%ad[`xà XZT>Uv\ lL~hd\Ãá \hJÝn%fwz\ZµTWXbwzhx`k RÕïOWÊlmcRTW©\onad[x RÕsdrlLXowzJ\onoP ¬

`¬

UvhJ[`[`T>Uv\ZwzhJ[`kLXZhJóUvT>cRchJVY\V`\i\hJaQ\T>k%`f`wRUoµkZYhfk\ZàQ\\ZYTW2aQXZTqwR[`xYTWT>xµV`kZT>xµTWdT>[\ZYhJVYJµ\ZYTW aQXZTq[`hd\ ad[`xàQ\hdXbÝLhdX_\Z`wRk_\ZadkjMP ³YT¯\ZaQXbdTW\LV`[Uv\ZwzhJ[`Ãk à \Xbw ad[`¹ x à XZT>Uv\ 0, &.1?c03,)TWTWYhJczT²UvhJ[`[YT>Uv\ZwzhJ[kµ©hdXµ~hd\Z-ädcRV`T>kÝhd P ®GwzJVYXbTÝrPROdOÝkbYhfk&\gwRUWadci[YTW\gfihdXZj khJcyVY\ZwzhJ[`kLhdX\ZYT à XZT>Uv\ \ZaQXZdTW\%mV`[Ùv\ZwRhJ[EP2³YT UvT>cRc}hJVY\VYo\ Ó T>VàdcRk}\ZYTUvT>cyck\ZaQ\]T wR[ \ZYTkT>UvhJ[x2adczhdeT>adUb2TWXbwzhxlLfYT>Ã[ à XZT>Uv\ ÕËOn}ad[`x2wRkæWTWXbhwR[ \ZYT `Xbk\ àdcR %~T>UWadV`kT\ZYThJVY\VY\JaQ\TqUWczhJkT>k\Z`TUvT>cycGlL\XowzddTWXZT>xµ~A %fwRUb2wRkadUWUvT>kZkT>xwRa%\ZYTTWTWYhJczT UvhJ[`[`T>Uv\ZwzhJ[`konoP³_`T\ZwR wR[Y%wy[Y©hdXo aQ\ZwzhJ[2wRk}XbT>adxµ5h W %YadkTvcRadwR[`T>x wR[T>Uv\ZwzhJ[rPtrPtsPi®`VYXZ\ZYTWXohdXZ>T % \ZYT2\gfih k\ZaQ\T>khd\ZYrT à XZT>Uv\ mV`[Ùv\ZwzhJ[åaQXZTx`wRkZ\ZwR[YJV`wRkbYT>xEÄ DwRkUvhJV`[\T>xV`Ñf`T>[ à XbT>Uv\ Õ Êïad[`x UvhJV`[]\T>xïxYhf[²aQJadwy[¯fYT>R [ à XZT>Uv\ Õ OQP³_wRkwRkadUbwzTWdT>x²\ZYXZhJVYJïa2[YTWJaQ\ZwzdTUvhJ[`[YT>Uv\ZwzhJ[²LXZhJ8\ZYT UvT>cRchJV`\VY\\h \ZYT\ZYT%UvT>cRcEwR[YV`&\ %]LTWT>x`wR[Y [`TWJaQ\ZwzdT&wR[YMVY\iwy[\h\ZYTqUvT>cychJ[`cz2fYT>[µ\ZYThJVY\MVY\JaQ\T 0Ê

»¬


Tï Tï

¢¤

32

1

1

tk yk

0 0

0.0

-2

-0.2

-4

-0.4 sc yc

-6

1

0

0 y

y

in

out ϕ

y

10

sc yc

-0.6

1

0

tk yk

0

20

30

Time

0

y

in

y

ϕ

out

y

10

20

30

Time

®GwzJVYXbTrPROWÊ²³T>k\ XbV[`khda\XbadwR[YT>x±´³¶![YTW\úfhdXZj fwR\Z lLXbwzJ]\onkWPÁfwz\Z`hJVY\lmczTWL\onTWTWYhJczT UvhJ[`[`T>Uv\ZwzhJ[`khJ[ï\ZYT à UvhJk |®uÆ\Zadkj¯fwz\Z [Õ OWÊP&³hde\ZaQXZdTW\ädcyVYT>kÔ ä ad[x²[`TW\gfihdXZjhJVY\MVY\mÓ ä § wyx`x`czTdÚvT>cRck\ZaQ\]T y}ad[`x2UvT>cRcMhJVY\VY\ Ó/§]~hd\\hJadUv\ZwRäQ\ZwzhJ[2hde\Z`TwR[`VY\ÄJaQ\TmÓ %©hdXZdTW\JaQ\TmÓ Ê % ad[`xµhJV`\VY\JaQ\sT Ó5¡l¢ÃP L¬

wRkhdT>e[ §hd\ZYTWXZfwykT%\Z`TwR[`VY\wykxYhJ wR[àQ\T>x¯~]¯\ZYT%hJkZwR\ZwzdT&~wyadkUvhJ[[YT>Uv\ZwzhJ[EPÍ^TW\úfhdXbjkfwz\ZYhJVY\ TWTWYhJcRTUvhJ[`[YT>Uv\ZwzhJ[kÝUWad[`[`hd\2V`kZT\Z`wRkÝT>Ubad[`wRkZw%_ad[xx`wRx"[Yhd\M[`xad[]adcz\TWXb[aQ\ZwzdT¯khJcyVY\ZwzhJ[EP ³_`XZhJVYJYhJVY\adcycYTvTWXbwyT>[\Zk}TWTWYhJczT_UvhJ[`[`T>Uv\ZwzhJ[`kfiTWXZT[YT>UvT>kZkZaQXZ\h%\XbwzddTWX\ZYThdT>[`wR[`hdEJaQ\T>k fwRczT\ZYT&hJV`\VY\_JaQ\TfiadkUWczhJkT>Px %~2dXoad[\ZwR[` V`[YXZT>kZ\XbwRUv\T>xadUWUvT>kbk\h2\ZYTq\ZwR TWXwRMczT>T>[]\T>x~] \ZYT_PB³_`TJaQ\T>kÄczT>aQXo[YT>x&\hUvhJ%~wR[YTi\Z`wRk´wR[YLhdXb aQ\ZwRhJ[qfwz\Z%\ZYT>wzX´~wRadk´khadk´\hhd@T>[%hJ[XZT>adUbwR[Y aUvTWXZ\Zadwy[\XbwzddTWX\ZYXZT>kZYhJcyxEP

rQÊ

×]êëìÚGÛí

Tï


tk

1

tk

1

yk

yk

0

0 sc

1

yc

4

sc yc

3 2 1 0

0 1

1

yin yout yϕ

yin out y ϕ y 0

0 0

10 Time

20

0

25

50

Time

®GwzJVYXbT&rPROdOQ³T>k\_XoV`[`khdG\XbadwR[`T>xTWTWYhJczTq±G³¶ë[YTW\gfihdXZjkhJ[\Z`T-à XZT>Uv\ |®u/\ZadkZj2fwz\Z Õ OWÊ lmczTWL\on}ad[`x ÕsdrµlLXbwzJ]\onoP³IhdeG\ZaQXZdTW\QadcRVYT>k Ô ad[`xÝ[`TW\gfihdXZj hJVY\MVY_\ Ó § wRx`x`czTdÚvT>cRc@k\ZaQ\T2Dad[`x UvT>cRchJV`\VY\ Ó §`~hd\\hJ}adUv\ZwzQaQ\ZwzhJ[hd\ZYT%wR[YVYä \iJaQ\sT ÓE %Y©hdXZdTW\JaQ\sT Ó ä Ê %ad[`xhJVY\VY\JaQ\ToÓ>¡D¢òP `¬

´¬

Ë×_ØÙ×]ÿiòÄÜdð>ØÙ

rO

TïÌË>ï

5

10 Cycle

15

yin

yout yϕ

0.9

20

0

5

10 Cycle

15

0.1

20

yout

yin, yϕ -4

0

0.2

1

sc

-3

0

G® wzJVYXbTrPRO>siT>cRck\ZaQ\T>k2ad[`x JaQ\T²adUv\ZwzäQ\ZwRhJ[`k aQ\ \ZYThJ[`kTW\¯lLæWTWXZhºàdkTBn%hd\ZYTw`Xok\ sQÊ UvUWczT>k x`V`XbwR[Ya \T>kZ\XbV[fwz\Z¯a \XbadwR[`T>x±Ý³¶[YTW\gfihdXZjhJ[¯\ZYT-à UvhJk |}®u°\ZadkjfwR\Z éÕ OWÊP^hd\T\ZàQ\ \ZYTqwy[`wz\ZwRadck\ZaQ\TÝlmaQ\UvUWcRT&ÊnwRkV`wz\TmaQX_LXZhJó\ZYTqT>V`wRcywz~`XbwRV k\ZaQ\TdP [¬

ÎW!3G '2okCBJG EnI'Ò$E& "!8AVþ.$DÌ@ 6Ñ & !C&´$&' & .'Ò$E& "! x

x

,x

Í

\\ZYT}~TWJwR[`[wR[YhdYT>adUbk\XZT>adàUvT>cRc]k\ZaQ\T>kGad[`xqJaQ\TadUv\ZwzQaQ\ZwzhJ[`káQXZTwR[wz\ZwRadcRwzæWT>x\hæWTWXZhYPÄ³_`wRkIwR[`wz\ZwRadc k\ZaQ\TqwykadcRhJk\adczfa>k_V`wz\TLaQX©XZhJ \Z`TUvhdXbXZT>khJ[`x`wR[`k\ZaQ\T%wR[2\ZYTkZadTàdkZThdGcRaQ\TWXTWXbwzhx`k wR[\ZYT&k\XZT>adP_®´wRJVYXZT%rPRO>sÝwRcRcRVk\XbaQ\T>k\Z`wyk©hdX\ZY-T à UvhJk \Zadkj@P_L\TWXLTWfàUvhJ[kT>UWVY\ZwzdT%@TWXowzhx&k %ÙvT>cRc k\ZaQ\T>kad[`xÑJaQ\TadUv\ZwzQaQ\ZwzhJ[`k%hdkZVÙWUvT>kZkZLV`c[YTW\gfihdXZjk\T>[`xº\hïkTW\\ZczTÝ\h¯dTWXbºk\ZaQ~cz>T %GadkTvágkT>UW'w U QadcRVYT>&k %f`wRUoaQXZT_\gwRUWadcyczV`wz\T_xßw @TWXbT>[\ÄLXZhJ \Z`TUvhdXZXZT>khJ[`xwR[YädcRV`T>k}wR[\ZYiT `Xbk\ÄTWXbwzhxEP´³_`wRk kZV`ddT>k\Zk\ZàQ\\ZYTqwy[`wz\ZwRadc@k\ZaQ\Tqhd\ZYTq[YTW\gfihdXZjkZYhJVcRx ~TczT>aQXb[`T>xadkfT>cyAc %Yadk`XZhdhJkT>x2~Ý®YhdXbUWadxa ad[`xaQXZXbadkZUvh¯lÃO>¤d¤drJn %wR[`kZ\T>adxhdáQXb~wz\XbaQXbwycz wR[`wR\ZwRadcRwzæ>wR[`&wz\\h æWTWXbhYP @ÿ |}XZTWwRhJV`k_fihdXZjhJ[²±´³¶ x`wRx[`hd\XZT>V`wzXbTq\ZYT[YTW\gfihdXZj\h2Tv\XbadUv\XZT>czTWQad[\wR[YLhdXb aQ\ZwzhJ[¯UvhJ[dTWdT>x ~ï\ZYT x`VYXoaQ\ZwzhJ[hd_wR[]\TWXZQadcRk~TW\gfiTWT>[ºTWdT>[]\ZkWPrÄTWXZT fiT kZ`hf°\ZaQ\%±Ý³¶(UWad[ºkhJczdT2kZVÙoº`wzJcz [YhJ[cRwR[YT>aQX\Zadkjk adkfiT>cRAc %~]czT>aQXo[`wR[Y²\hº`XbT>UWwRkT>czåT>adkZVYXZT\ZwyTwR[]\TWXZädcyk&%}`XbhwRxYT>x fiTLV`Xb[`wRkZ ±G³¶ëUvT>cRcRk_fwz\ZTWTWYhJczTqUvhJ[[YT>Uv\ZwzhJ[`k\ZàQ\adcRcRhf \Z`T>\hÝwy[`kT>Uv\\ZYT>wzXUWV`XZXZT>[]\wR[]\TWXb[àdcekZ\ZaQ\T>kWP Âò\wykXbT> aQXZjQaQ~czT&\ZaQ\@TWTWMYhJczT%±G³¶ÌUWad[ïczT>aQXo[¯TvYadUv\qad[`x²Tv\XZT>T>czXbhd~V`k\\ZwR wR[`ÝadczdhdXbwR\Z` k fwR\ZYhJVY\\T>adUoYTWX©hdXoUWwR[Y[2wy[ UWadkThdedTWXZ V[`wR[YLhdXb aQ\Zwzd>T %XbaQXZT>cz Uoàd[YJwR[Y%\ZaQXZdTW\kZwzJ[àdcykWPÄ³_`wRk aQjdT>kwz\a `XbhJ wRkZwR[` aQ``XZhJadUoLhdX[V`TWXbhJV`kXbT>adc áäfhdXocRx\ZadkjkfYhJkTkhJcRVY\ZwzhJ[aQXZ\Zczx`TW@T>[xhJ[ \ZYTXZT>UWwRkTqx`V`XbaQ\ZwzhJ[µhdGwR[\TWXbädcRk_~@TW\úfTWT>[XZT>czTWQad[\TWdT>[]\ZkWP nÆÎ

rds

×]êëìÚGÛí

Tï


¨

©«ªX¬®~¯Z°

¸

!785

7

Ï

3 4(596"7 !6 ÂÐ

:ü¸Ä6¹¸ Í7

À

3Ñ

Ì5 ¹

ô¶

º

¹

)77 ³

Ì5 ´

3 4(5(6"7

PÐ

6

78:

Eÿ |}XZTWwRhJV`kefihdXZj%kbYhfiT>x\ZàQ\Ä±´³¶ÆhJV`\@TWXb©hdXb kI\Xbadxwz\ZwzhJ[àdc]S^^ÁadczdhdXbwz\Z kIhJ[\ZadkjkG\ZàQ\´XZT>V`wzXZT \hczT>aQXo[¯\Z`TXbV`czT>khdiXZTWJV`cRaQXcRad[YJVaQdT>k lmS±kon %IkTWTÝaQ`\TWX&µad[`é x ÄhUoYXZT>wz\TWXqad[`xºYUb` wyx`]V`~@TWX lÃO>¤d¤JdnoP_S±IkaQXbT&xYT>kZUvXbwR~aQ~czT~µx`TW\TWXb wR[`wyk\ZwRU [`wz\TqkZ\ZaQ\TadVY\hJ aQ\Za²lmâ®n&läadkTW> %GO>¤d¤dd{ §IwzTWdT>c á ad[[e%O>¤d¤dds §qÇicyadwzX *Å|hJcycRadUZ j %&O>¤d¤J} §]}qadcRwy[YjdR T *ë±eT>` ad[e[ %O>¤d¤dd¡ §o÷@T>[Y¤d¤¨¢JßnoP [\Zwyc´[`hf %eYhBfTWdTW(X %wz\àdkXZT> adwR[YT>xïV`[ÙWczT>aQXfYTW\ZYTWX±´³N ¶ kkZV`@TWXowzhdXbwz\úUWaQXZXbwzT>khBdTWX \h\ZadkjkwR[]dhJczwy[Y%UvhJ[]\Tv\©XZTWTcRad[`JVàQdT>kqlä®G±Ikon %YkZVUbadk\ZYhJkTx`wykZUWV`kZkT>x2wR[Ý\ZYTS^^ cRwR\TWXbaQ\ZVYXZT läVe[ %EuwRczT>&k %IYT>P[ %p* ±ITW>T %O>¤d¤dd §I)wRczT>nk *Æ}cR adP[ %GO>¤d¤ddr §Ä\T>w âdTWXbnk *óuXbV`[]fiadcyex %ÄO>¤d¤dd{ §³hJ[YjdT>k * )wRczT>&k %iO>¤d¤J} §}S_hxYXbwRJVYTW>æ %)wRczT>k %$*cR ade[ %iO>¤d¤dd¤ §ShxYXowzJVYTWæ *ó)wRczT>k %iO>¤d¤d¡JnoP³YT>wzXXZT>UvhdJ[`wR\ZwzhJ[ XZT>V`wzXbT>kÄ\ZYTmV`[Ùv\ZwRhJ[àdc`T>V`wzQadczT>[]\hdea%kZ\ZadUZj@PÂú\iwRk}UvhJ[UvT>wzäQ~MczT\ZàQ\±Ý³¶8àdek âZV`k\}\ZYTXbwRJ\~wRadk LhdXS±k~VY\ wRJ\madwRchJ[²®´±kvP ÄTWXZTµfiT2fwRcRc©hUWV`khJ[ \ZYThJk\UvhJ hJ[®´±k~T>[Ùb` aQXZjk%LhJV`[`x wR[Ñ\ZYTS^^ cRwR\TWXbaQ\ZVYXZTd [KÝad[kx 6, Ï Ï MPT%kZ\ZV`xYÝVYT>k\ZwRhJ[`k_kZVÙoadkW ad[±Ý³¶ÉczT>aQXo[\ZYTmV`[Ùv\ZwzhJ[àdcT>V`wzQadczT>[]\hdÄaV`kZ`x`hf[adVY\hJ aQ\hJ[+Ç uwzdT>[\Xbadwy[`wR[YkT>VYT>[ÙvT>k_V`\h kZwzæWnT Ø!%ÙWad[¯wz\dT>[YTWXbadcywzæWT\~h ØA®)5O Ö3Ø@®Áds Ö&×&×&×)Ç ZÄhBf k\ZaQ~cRTqaQXZTq\ZYTqkhJcyVY\ZwzhJ[` kÇ âhT>k±G³¶ÉhJVY\TWXZ©hdXoó`XZTWwzhJV`kaQ`XZhJadUbYT>kÇ ®GwR[adcRczfiTfwRcRcIaQcz±G³¶ \haUvhJ[]\Tv\ a,/BP²TfwRcRcLhUWV`khJ[Ñ\Z`TµUWcyadkZkZwRUÝTvad czT %fwRUbåwRka± ~V`\[Yhd\a®´±ÁläT>Uv\ZwzhJ[{PtsJnoP}ÂÃ[ dT>[YTWXoadAc %_±¯XZT>UvhdJ[`wz\ZwzhJ[2XZT>V`wzXbT>kacRwR[YT>aQXZáä~@hJV[`xYT>x adVY\hJ aQ\hJe[ % aµkT>UWwRadcG³´VYXbwR[Yµ adUo`wR[YTfYhJkZT\ZaQTczT>[Yd\ZwykaQ\q hJk\qcRwR[YT>aQXwR[²\ZYT wR[`VY\kbwzæWTdP%³_ÀT çD[KK ê cRad[`JVàQdT¯wRk hJ[YT²hd\ZYT¯kZwyczT>k\Ý±I&k §wz\ÝUWad["~TdT>[YTWXoaQ\T>x"~ÁaÑ\XZTWTvágadVx âhJwR[YT>xÁdXoad aQXad[`x XZT>UvhdJ[wzæWT>x¯VkZwR[Y aÝkhQágUWadcRczT>x¯T>&~T>x`x`T>xV`kbágxYhf[adVY\hJ aQ\hJ[cl bw âaáòYàd[YjdTWX %GO>¤d¤dsJnhdXa [`wz\T rd ÎÆÅ

m e ~$

p

p

p

p

-

ÜdðdgìÿIÛ ×_ØÙÚGÛ ÚÅ$íÛ}ÛüëÙñ ×_ØÙÚGÛ ÚÜÛÙÜdð>Ú$ðVý´ÛÿGëÙ¥ò$ë2¥ÛÜ

r¨¢×]êëìÚÄÛ_í

ÒË>ï

«2

32

k \ZaQ\TadVY\hJ aQ\hJ[¯fwz\Z¯adUWUvT>kbk\hÝ\úfhUvhJV`[]\TWXbk\ZàQ\UWad[¯~TwR[ÙvXZT>T>[]\T>x²hdXx`T>UvXZT>T>[]\T>xEP³hµhJVYX j[YhfczT>xYdTq[Yh S^^Ëàdk~TWT>[aQ~MczT\hÝczT>aQXb[a2_±P TaQXZT%VkZwR[Y ±G³¶Åfwz\ZLhdXZdTW\JaQ\T>kad[xTWTWYhJczT&UvhJ[[YT>Uv\ZwzhJ[`kwR[\Xbhx`VUvT>xwR[\ZYT%`XZTWwzhJV`k UoàQ`\TWXbkWP eÿ ³_`T[YTW\úfhdXbjÝkT>VYT>[]\ZwRadcRczhd~kZTWXZdT>kTvT> cRaQXZ kZ%~hJc@kZ\XbwR[YJkhdaJwzdT>[cRad[`JVàQdT>%`XZT>kT>[]\T>xµhJ[YT wR[`VY\kZ%~hJcMaQ\_a\ZwR TdP®YhJcRczhBfwR[Y\ZYT\Xbadx`wz\ZwRhJ[àdcaQ``XZhJadUowR[2\ZYTS^^ cRwz\TWXoaQ\ZVYXZTfTLhdXbV`cRaQ\T \ZYT%\Zadkjadka `XbT>x`wRUv\ZwzhJ[\ZadkZjMP\ad[]JwzdT>[¯\ZwRT&k\TWï\ZYT%\ZaQXZdTW\wRk_\h2`XZT>x`wyUv\_\ZYT&hJkZkbwz~czTq[YTv\ k%~@hJcy&k %]wR[UWcRV`x`wR[`\Z`TßQT>[`x2hdIkZ\XbwR[YJßk&~hJc %PÄ-YT>[µhdXbT\Zàd[ÝhJ[YTk&~hJcMUWad[ÝhUWUWVYXwR[Ý\ZYT [YTv\_k\TW 2544´hJkZkZwz~MczTkZ%~hJcRkà>dT&\h ~T`XZT>x`wRUv\T>Px %àd[`x[YhJ[YTqhd´\ZYTqhd\Z`TWXbkWP ³_`T[YTW\úfhdXbjÝkT>VYT>[]\ZwRadcRczhd~kZTWXZdT>kTvT> cRaQXZ kZ%~hJc@kZ\XbwR[YJkhdaJwzdT>[cRad[`JVàQd>T %`XZT>kT>[]\T>xµhJ[YT wR[`VY\_k&~hJcaQ\a \ZwR>T %adcRkh2XZTWLTWXZXZT>x\hadkwR[YV`\kT>VYT>[ÙvT>kWPdTWXZwy[YVY\kZT>]VYT>[UvT&~TWJwR[`kfwz\Z \ZYTk\ZaQXb\GkZ%~hJ.c ¨PG³YTT> `\gk\XbwR[` %JwRkGUvhJ[`kbwRxYTWXZT>xaQXZ\GhdMT>adUb cRad[`JVàQdTdP àk\XbwR[`wyk_adUWUvTW`\T>x¯fYT>[adcRc`XbT>x`wRUv\ZwzhJ[`ka>dTq~TWT>[UvhdXZXbT>Uv\WP\ZYTWXbfwRkTwz\_wRkXbçT âÃT>Uv\T>xEP ³_wRkÄ`XZT>xwRUv\ZwzhJ[\ZadkjÝwRk}T>V`wzQadczT>[]\\ha%UWcRadkbkZ'w UWaQ\ZwzhJ[ \Zadkj fwR\Z \gfihUWcRadkZkT>kÞadUWUvTW`\Zßad[`xåÞÃXZçT âT>Uv&\ % ß ~T>UWadV`kT\Z`T kkZ\T>ÈfwRcRc´ aQjdT XZT>x`wRUv\ZwzhJ[²TWXZXbhdXbkLhdX&adcRcGk\XbwR[`JkhJVY\ZkZwRxYT\Z`T cRad[YJVàQdTdP Ækk\T> àdkcRT>aQXb[YT>xaµJwzdT>[ cyad[YJVàQdT VY\hkZ\XbwR[YkZwzæWT Ø hJ[ÙvTÝwz\wRkaQ~czT\hUvhdXZXbT>Uv\Zczï`XZT>x`wRUv\adcRcÄk\XowR[YJk fwR\ZkZwzæWT ZØÄP %~hJcRkaQXZTT>[Ùvhx`T>x czhUWadcRcz~] Jágx`wRT>[`kbwzhJ[àdcY~wy[àQXZdT>Uv\hdXbkifwz\ZhJ[`cRhJ[`T[YhJ[áäæWTWXbh%UvhJhQá [YT>[]&\ %fYTWXZT T>VàdcRk\ZYT[V`%~TWXhdIcRad[YJVaQdTk&~hJcRkcRV`kÄhJ[`TLhdX}T>wz\Z`TWX\ZYTk\ZaQXZ\ik%~hJcwR[ \ZYT wR[`VY\hdX\ZYT%ßQT>[`x¯hd}k\XowR[YJß k%~@hJcwR[\Z`T&hJVY\V`\l wR[YMVY\_V`[`wz\Zk % 2hJVY\VY\V`[`wz\Zk noMP ®O&kZwRJ[`'w `T>k \ZàQ\_akZ%~hJcEwRkkTW\_ad[`é x ÞqO\ZaQ\wz\wRk[Yhd&\ §`\Z`TqxYT>UWwRkZwzhJ[2~hJV`[`xaQXZ LhdX_\ZYTq[YTW\úfhdXZj2hJVY\V`\_wRk1Ê × ÊP ½»_« [K²läV[ÝTW\adcã'P %eO>¤d¤dd §)wRczT>fk *ËcR ade[ %EO>¤d¤ddr §M³hJ[YjdT>k *à-wRcRT>&k %eO>¤d¤J} §MShxYXowzJVYTWæTW\adcã'P % O>¤d¤d¤JnoP ÄTWXZT\ZYT_k\XowR[YJkGwR[\ZYTwR[YV`\´kT>VYT>[ÙvT>kÄaQXbThd\ZYTLhdXbz §]wy[YVY\ád[`xhJVY\VY\GdT>Uv\hdXbkaQXZT Bágx`wyT>[`kZwzhJ[adcãP´|}XbwzhdX\h\ZYnT `XbkZ\hUWUWVYXZXZT>[UvThd T>wz\ZYTWX hdMX h%hd8X hdX aQ\_kZT>]VYT>[UvT~@TWJwy[`[`wR[YJk % aQXZThJkZkbwz~czTwR[µ\ZYT&[`Tv\_k\TWIP}³_V`&k %YTdP£Y'P %LhdfX ØÕr Âò[YMVY\W a a a a a ~~ë~~ì~ ³áQXZdTW\W a ¨³ a B~Ìa B~Ìa B~ a B~Ìa B~ ~~ë~~Ì³ [µTvad czTLhdXakTW\_hdGUvhJ[\Tv\ÃáäLXZTWT%XZhxVÙv\ZwzhJ[2XbV`czT>kiLhdX_\ZY]T [K dXoad aQXwykW} õ §:õ a õI~ %YfYTWXZÏ T%µÏ wyk\ZYT&k\ZaQXZ\Zwy[Y k&~hJAc %CõåwRk_a[YhJ[Yáä\TWXb wR[àdcEkZ%~hJcEad[`x wyk\ZYTT>`\úk\XbwR[YYP l ½»_« [, MålmShxYXowzJVYTW~æ *Æ-wyczT>&k %O>¤d¤d¡JnoP ³_YTkT>UvhJ[x àdczhdak\Xbwy[YÝLXZhJ,\Z`wykadcRwR[á xYXbhJT hdX wzXZXbhdX%cRad[YJVaQdT2wRkUvhJcRTW\T>cz`XZT>x`wyUv\ZaQ~czT LXZhJë\ZYT `Xbk\àdczòP³_YTÝ\ZadkjºwR[]dhJczdT>kad[ wR[]\TWXbT>xwRaQ\T%\ZwyTcyaQ2hdczT>[Yd\ZVs ÑPqÂÃ[YVY\ad[`x²hJVY\VY\dT>Uv\hdXbkqaQXZT rBágx`wyT>[`kZwzhJ[adcãP|}XbwzhdX\hµ\ZYT `Xok\hUWUWVYXbXZT>[ÙvTqhd Æ\gfihÝk&~hJcRkaQXbTqhJkZkZwz~cRTwR[µ\ZYT%[YTv\k\TWeP³_V`& k %YTdP£Y'P %M©hdfX ØÕµx²~]²kZwR wycRaQXUvhJ[]\Tv\ÃáäLXZTWTÝXbV`czT>kadk\ZYT [KdXbad aQX V`kbwR[Y\gfihÝ[YhJ[áä\TWXo wR[àdcEk%~@hJcykqAl õÑad[x ńoi õ §õ la õ ü§ (~ IÇ P ½ ¬« [KK S Âò[YMVY\ad[`xµhJVY\VY\dT>Uv\hdXbkaQXZT¢Qágx`wRT>[kZwzhJ[àdcãPÄ|}XbwzhdX\h\ZYnT `XbkZ\ihUWUWVYXZXbT>[ÙvThd \úfh kZ%~hJcRkaQXZTqhJkZkZwR~czTwR[µ\ZYT&[YTv\_k\TWeP³_]V&k %TdP£Y'P %LhdfX ØÕr ÂÃ[YVY\W a a a a a ~ ~~ë~~ U UÆU UU ³áQXZdTW\Wa ¨³ a B~Ìa B~Ìa B~Ìa B~Ìa B~~ ~~ë~ÍU8U UÆU UÆ³ ÎÆJø

RTS

_p

Óf

vf

²c

iy

y

=y

?y

f

aj

aj

aj

aj

aj

aj

£ÕÔ

ò

£ÓÔ

Ô

-

N-

-

aj

aj

aj

aj

aj

+j

+j

+j

£ÕÔ

§o

JÖ

aj

aj

aj

aj

ò

aj

aj

£ÓÔH£ Ö

×Ö§o

Ö

£ÓÔ

§o

¤ "Û

Ë>ï

1ï

}Û_íið ¯Û}ÙÚÜ

rdr

32mì

_³ `TV` wR[Y ±IT> aÑ©hdXUvhJ[]\Tv\ÃáäLXZTWT cRad[YJVaQdT>k2UWad[)~T²aQ`cRwzT>x\håkbYhf,\ZàQ\`[K[ÁwRk2[Yhd\ UvhJ[]\Tv\ÃáäLXZTWTdP}[wR[]\ZV`wz\ZwzdTTvcRad[àQ\ZwRhJ[wykG\ZàQ\wz\wRkG[YT>UvT>kZkbaQXZ%\h%UvhJ[`kZwRxYTWXÄ\Z`T[V`%~TWXÄhdÒ%k%~hJcRk \ZYT>[`XZhx`VÙWwR[YÄad[`xkk&~hJcRk%Y\Z`wRkXZT>V`wzXZT>kUvhJ[]\Tv\wR[YLhdXb aQ\ZwRhJ[EP

ÂM 6'I&>!3&>!#1j'!C(ÍÂM+BD$&>!#1 Ø

±IT>aQXb[`wR[Yåad[`x"\T>k\ZwR[Yadcz\TWXb[aQ\TdÑaQL\TWXT>adUbçTWhUb/=l ÕóOWÊdÊdÊ \Xbadwy[`wR[YåkT>]V`T>[ÙvT>kon fTïLXZTWTWæWTï\ZYT fiT>wzJ]\Zk ad[`xXbV`[Áaº\T>k\WPÕdT>["fYT>[Áadcyck\Xbwy[YJk aQXZT¯`XZhUvT>kZkT>xUvhdXbXZT>Uv\ZczÁx`VYXowR[Yï\XbadwR[`wy[YUvT>kbkZaQXZµ\h \T>k\aQJadwR[fwz\ZLXZhdæWT>[¯fT>wzJ]\Zk_hJ[UvTadcRcefT>wzJ]\Uoàd[YdT>kà>dT~TWT>[TvT>UWVY\T>xEP_aQXZ\ LXZhJçT>[`kbVYXbwR[Y\ZYTczT>aQXb[`wR[ìhd\ZYT}\XbadwR[`wR[`kZTW\I\ZYT\T>kZ\IadcRkhx`TW\TWXb wR[YT>kedT>[YTWXbadcRwzæ>aQ\ZwRhJ[TWXZ©hdXo ad[Ùv>T % fwRUbµfiTqx`wRx[Yhd\hd`\ZwR wzæWT~V`kZwR[Y %MaQadcRwRxàQ\ZwRhJ[kTW\WP ³XbadwR[wR[Yad[`x\T>k\kTW\Zkwy[ÙvhdXZhdXbaQ\T&adcRcIczTWJadcek\Xbwy[YJk_VY\h2aJwzdT>[¯czT>[Yd\ZEVs Øº©hdX 6, :%MV Ø LhdX [K[ad[`x sYcl ØH®¥Ñ¯n©hdÄX [K Ï Ï MMP²³IXbadwy[`wR[YkZ\XbwR[YJk%aQXZTÝXZT>kT>[]\T>xåwy[ Xbad[`x`hJÅhdXbx`TWX>Pï[cz TvT>cRaQXokG©XZhJ \ZYTUWcRadkZk&ÞadUWUvTW`\Zß%aQXbT_`XZT>kT>[]\T>xEPÄ³XbadwR[`wR[`wykk\hd`T>xhJ[ÙvTadcycY\XbadwR[`wy[YkT>]V`T>[ÙvT>k àdT_~TWT>[adUWUvTW`\T>ex %hdX}aQL\TWXaQ\}hJkZ\_OWÊ \XbadwR[`wy[YkZT>]VYT>[UvT>kWPÄ³_YiT },&B ,/03254û? (2>;@?A75Bwa&,;YwRkG\ZYTcyaQXZdT>k\ adUWUvTW`\T>x\T>k\kZTW\&lmadkZkZV` wy[Y\ZàQ\_\ZYTq[`TW\gfihdXZjdT>[YTWXbadcRwRæWT>k_aQ\adcRc©noP T>wzJ]\Uoàd[YdT>kiaQXZT adx`TaQ©\TWXT>adUb2kT>VYT>[ÙvTdPÄTaQcz\ZYThJT>[]\ZV` adczdhdXbwz\Z` lm|}cRadVY\ÄTW\iadcã'P % O>¤d¡d{Jn}fwz\Z2czT>aQXb[wR[YqXbaQ\nT õwRkOWÊ ¯ ad[`xÝ hJT>[\ZV°aQXoadTW\TWXÊPt¤d¤PcRc`XZT>kbV`cz\Zk}aQXbTadTWXbaQdT>khBdTWX OWÊµwR[`x`TW@T>[xYT>[\ZcRÝ\Xbadwy[YT>x¯[`TW\gfihdXZjkfwz\Z¯xßw @TWXbT>[\fT>wRJ\wR[`wR\ZwRadcRwzæ>aQ\ZwzhJ[k&lL\ZYT>kTOWÊµwR[`wz\ZwyadcRwzæ>aQ\ZwzhJ[`k aQXZTqwyxYT>[\ZwyUWadc©hdXT>adUbTvTWXbwRT>[]\onoP ½»_« +S TkZ\ZV`xY\XbadwR[wR[Y&kTW\ZkifwR\Z Ø]æ)ç5O Ö'×'×'Ö ê P}T\T>k\adcRc@kTW\Zkifwz\Zy Ø]æ)ç5O Ö'×'×'Ö ê ad[`x æ)ç Ö'×'×'Ö>OWÊdÊdÏ Ê ê Ï lmkZT>]VYT>[UvT>k_hdGczT>[Yd\Z sQÊdÊdÊnoP ½»_« 6. M S TµVkTÝ\úfhï\Xbadwy[`wR[YkZTW\ZkWÝan%³_YT2kZadTkTW\adkV`kT>xå~]ÑS_hxYXbwzJVYTWæ2ad[`x )wRczT>kµlÃO>¤d¤d¤Jn2~ ØWækç5O Ö'×'×'Ö>OdO ê %qÑæ4ç5O Ö'×'×'Ö>OdO ê fwz\Z[ Ø4®¥Ñ ,O>sålmkZT>]VYT>[UvT>khdczT>[Yd\Z ós¨¢]noP~@n ³_`TkTW\JwzdT>[ï~g ØAæç5O Ö'×'×'Ö>OdO ê %]Ñ æÄç5O Ö'×'×'Ö>OdO ê lmkZT>]VYT>[UvT>khdczT>[`d\Z ¢d¢]noP&T\T>k\qadcyc´kZTW\Zkfwz\Z Ø]æ)ç5 O Ö'×'×'Ö ê %WÑæcç5O Ö'×'×'Ö ê ad[`x æ)çOd5O Ö'×'×'ÖbrQÊ ê lmkT>VYT>[ÙvT>k_hdGczT>[Yd\Z sQÊdÊnoP Øæ7ç5O Ö'×'×'Ö ê ad[`x)~nÝfwz\Z ½ ¬« S TºkZ\ZV`xY\úfh"jwy[`x`k2hd%\XoadwR[`wR[YkTW\Zk>Áan2fwz\ZX Øxæxç Þ 5 O Ö ê P-adkT~n&adkZjk ©hdX a ha âÃhdX dT>[`TWXbadcRwzæ>aQ\ZwzhJ[kZ\TW\ZàQ\ kTWT> k adcRhJk\ wyhJkZkZwz~cRT aQn\ `Xok\Jcyad[ÙvTd&uwzdT>[dTWXZ²kZwR wycRaQX\XoadwR[`wR[YµkT>VYT>[ÙvT>kf`hJkT kZwzæWT>kx`'w @TWX~¯aQ\qhJk\&ds %czT>aQXb[²\h `XbhUvT>kZk%kT>VYT>[ÙvT>kqhd_aQXb~wz\XbaQXZïkZwzæWT TÝ\T>k\adcRckTW\Zk%fwz\Zé Økæ`ç fÖ'×'×'Ö ê % þæç5O Ö'×'×'Ö ÞçO ê ad[`x æ)ç Ö'×'×'ÖbrQÊdÊ ê lmkT>VYT>[ÙvT>khdÄczT>[Yd\Z ÕO>rQÊdÊnoP -

^¤

ÌÙ

X

K

K

Ç

bÇ

dc

-

Úc

®c

ªc

bÇ

bÇ

`Ç

Tc

K

K

K

K

Ç

OÛ

bÇ

«Û

K

·c

: Vþ.$DÌ@ 6ÑÂM" "1- '!3(Í G 6&>Ï+!"$E'8'I '7.$EG 6B Ø

³_`Twy[YVY\V`[`wz\ZkaQXZTmV`cRcRUvhJ[`[YT>Uv\T>x \hawRx`xYT>[ïcRadTWX&UvhJ[`kZwRkZ\ZwR[YµhdT> hdXZ¯~MczhUbjkfwz\ZOUvT>cRc T>adUoEPÄ³_YTUvT>cRchJVY\VY\ZkáQXZTLVcRczUvhJ[`[YT>Uv\T>x\h\ZYTUvT>cRcwR[YV`\Zk&%B\hadcRcJaQ\T>k&%]ad[`x\h\Z`ThJVY\MVY\´V[`wz\Zk&% fwRUbadcykh àdTx`wRXZT>Uv\ ÞkZYhdXZ\ZUWV`\ZßÝUvhJ[`[`T>Uv\ZwzhJ[`k_LXZhJ \ZYT&wR[YMVY\_V`[`wz\Zkqlm®GwzJV`XZT{PROnoP_®YhdXT>adUb¯\Zadkj fiT%kT>cRT>Uv\T>x¯\Z`T&\hdhJczhddfwR\Z wR[`wy adce[V`&~TWX_hdT>hdXZ~cRhUbjk\ZàQ\kZhJczdT>x\Z`T&\Zadkjfwz\ZYhJVY\ Tv\T>[`kZwzdT aQXbadTW\TWXhd`\ZwR wRæ>aQ\ZwzhJ[EP ±aQXZdTWX&\hdhJczhdJwzT>k%[YTWdTWX%cRT>x\h¯x`wykZadxYQad[\ZaQdT>k&TvUvTW`\%©hdXad[ wR[UvXZT>adkTqwR[UvhJMVY\ZaQ\ZwzhJ[àdceUvhJ czTvYwz\gdP cRcJaQ\T>&k %\ZYTUvT>cRcwz\ZkT>czad[`x \ZYTµhJVY\VY\V`[wz\aQXbT2~MwRadkT>xEPå³_YT2~wRadkfT>wRJ\Zk\h wy[YVY\%JaQ\>T % LhdXZdTW\&JaQ\Tad[`xºhJVY\VY\&JaQ\TµaQXbTÝwR[`wR\ZwRadcRwzæWT>x²fwz\Zü Þq5O × 1Ê %3®ds × Ê²ad[` x Þds × 1Ê %GXZT>kZ@T>Uv\ZwRdT>czdPcR\ZYhJVYJ [Yhd\ÝUvXbwz\ZwRUWadAc %\ZYT>kT¯ädcRV`T>k à>dT¯~TWT>[©hJV[`xT>wRXbwRUWadcRczº\h fhdXbjåfiT>cRAc §fiT¯VkT\ZYT> LhdXÝadcychJVYX TvTWXbwRT>[]\ZkWPI³_`T©hdXbdTW\IJaQ\T>k´k\ZaQXZ\I5h UWczhJkZT>ex %¨kZh_\ZàQ\I\Z`T}UvT>cRcRkewy[`wz\ZwRadcRcR_XZT>T>%~TWXeTWdTWXZ\Z`wR[YYP´T adcRkZh \XbwzT>x²x`ßw TWXZT>[]\~wyadkUvhJkZVcz\Zk_fiTWXZT]Vàdcywz\ZaQ\ZwzdT>cz\ZYTkZad>T %fwRUb¯kZVY`hdXZ\Zk hJVYXUWcRadwRÅ\ZàQ\%XZT>UWwRkT wy[`wz\ZwRadcRwRæ>aQ\ZwzhJ[wRk&[`hd\%UvXbwz\ZwyUWadcãPcychd\ZYTWXfT>wRJ\ZkaQXZTµwR[`wz\ZwRadcywzæWT>xïXbad[`xYhJ cz wR[å\ZYTXbad[Ydþ T «ßÞ_1Ê ×R5O ÖZ1Ê ×RO ¬äP)³_YTUvT>cRAc k wR[YVY\kZVàdkZ`wy[Y¯mV`[Ùv\ZwRhJX [ åwRk\ZYTwyxYT>[\ZwR\gåLV`[Uv\ZwzhJ[EP³_YT kZVàdkZwR[YLV`[Uv\ZwzhJ[µhdG\ZYThJVY\VY\_V[`wz\ZkwRk_akbwzJhJwRxµmV`[Ùv\ZwzhJ[µfwz\Zµ\ZYTqXbad[`dAT «ßÞds Öbys ¬äP

ÜdðdgìÿIÛ ×_ØÙÚGÛ ÚÅ$íÛ}ÛüëÙñ ×_ØÙÚGÛ ÚÜÛÙÜdð>Ú$ðVý´ÛÿGëÙ¥ò$ë2¥ÛÜ

rd{2×]êëìÚÄÛ_í

ÒË>ï

«2

32

Out


Output Gate Forget Gate Input Gate

In

®GwzJVYXbT{PROQ³_`XZTWTvágcRa>dTWXq±G³¶\hdhJczhddfwz\ZaµkZwR[`JczT&wR[YMVY\ad[`x¯hJVY\VY\WPS_T>UWV`XZXZT>[ÙvTwRkcywR wz\T>x \hµ\ZYT`wRxxYT>[¯cya>dTWX%IUvhJ[kZwRk\ZwR[`ÝYTWXZThdia2kbwR[YJczT±´³¶ÌT>hdXZ¯~czhUZjfwz\Zïa2kbwR[YJczTUvT>cRcãP&cRciOWÊ ÞV`[wz\Ãáä\hQágV`[`wz\ZßUvhJ[`[`T>Uv\ZwzhJ[`kaQXbT&kZYhBf[ lL~VY\~wyadk_ad[`xTWTWYhJcRTUvhJ[[YT>Uv\ZwzhJ[`k_aQXbT&[Yhd\onoP S_TWLTWXZT>[ÙvT ÄwRx`x`T>[ ³XbadwR[EP ³IXoadwR[EP hJcãP ÇiT>k\³T>k\ [`wz\Zk TW_\ « ØI¬ \X>P « OWÊ ¬ ³XbwãP « ØI¬ läV[µTW\adcã'P %eO>¤d¤dJn ä r O Ö'×'×'Ö>O>{QÊ 5 O>Ptr O O O Ö'×'×'Ö>O>{QÊ 5 lL)wRczT>k */}cR ade[ %EO>¤d¤drJn s O Ö'×'×'Ö>OdO 5 sQÊdÊdÊ ¢ QsQÊ O Ö'×'×'Ö>O>¡ 5 lm³IhJ[`jdT>k *Ë)wRczT>&k %eO>¤d¤Jdn s O Ö'×'×'Ö>OWÊ 5 OWÊ O> OWÊdÊ 5O Ö'×'×'Ö>O>s lmS_hxYXbwRJVYTWæTW\adcã'P %eO>¤d¤d¤Jn æ s O Ö'×'×'Ö>OdO 5 sd{J ¡ QrQÊ O Ö'×'×'Ö>O>{ 5 j

z

Oj

j ^j

^j

³áQ~czT{PROQ|XZTWwzhJV`kXZT>kZV`cR\Zk}LhdX\Z`T®G±p %kZYhBfwR[YµlL©XbhJ czTWL\i\h%XbwRJ\on\ZYT[]V`%~TWX}hdIwRx`xYT>[ V`[wz\Zk}hdXk\ZaQ\T&V`[`wR\Z&k %]\Z`TQadcRVYT>khd$ ØV`kT>xµx`VYXowR[Y&\XoadwR[`wR[Y]VYT>[UvT>k&%Y\ZYT [V`%~@TWXhd´LhJV`[`xkhJcRV`\ZwzhJ[`k B\XbwyadcRkad[`x\ZYT&cyaQXZdT>k\_adUWUvTW`\T>x¯\T>k\kTW\WP 9j

½»_« 6, S TqV`kThJ[YTqT>hdXb ~czhUZjïlLfwz\Z2hJ[YTUvT>cRc©noPÄ)wz\Z2TWTWYhJczTUvhJ[`[YT>Uv\ZwzhJ[k\ZYTWXZTaQXZT d¡adx5âV`kZ\ZaQ~czTÏ fT>Ï wRJ\ZklãTWTWYhJczT>%sd¡V[`wz\Ãáä\hQágV`[`wR\ad[x~wRadk_UvhJ[[YT>Uv\ZwzhJ[`konoP S TVkT\úfh~cRhUbjkifwz\Z hJ[YTUvT>cRcMT>adUbe %YXZT>kbV`cz\ZwR[Y%wR[¯OdOWÊadx5âV`kZ\ZaQ~czTfiT>wzJ]\Zk ½»_« 6, lã{TWTW`hJcz>T %`¤OqV`[`wz\Ãáä\hQágV[`wz\ad[`x O>~MwRadk_UvhJ[`[YT>Uv\ZwRhJ[`konoP Ï Ï McRad[YJVaQd>T %@~MVY\fwR\Z¢µwR[YVY\ ½ ¬« [K[ S TV`kT&\Z`TkZad T%\hdhJczhddadkLhdX\ZY_T [K ad[`xºhJVY\VY\V`[`wz\Zk&wR[`k\T>adxÑhddr %GXZT>kZV`cz\Zwy[YwR[Ñ¤QÊ²adVx âZV`k\ZaQ~cRT fT>wRJ\Zklã{@TWTWMYhJcz>T %GQs¯V`[`wR\Ãáä\hQágV`[`wz\ ad[`xO>s ~wRadkUvhJ[`[`T>Uv\ZwzhJ[`konoP -

-

:T .n#& "=3B` 6Bb=3´$B ½%»« [K S |}V`~cRwRkZ`T>xXZT>kZV`cz\Zk}hJ[ \Z`TM[K cRad[YJVàQdTaQXZTkZV` aQXowzæWT>x wR[Ý³aQ~cRT{PROQP}S^^k}\XoadwR[YT>x Ø

ÜÞÝßÀàá âÓã;äå9æçàèæéàèê3â;ëã?ì å9â+å9ßÀê9íëàã"ëîâ=ã"ëïìµæçâ;ëmðñBâ;ëòßÀëàÀóëâíæçâ;óôéå9â;â;æõ¿ëîîOßèä"æçàÀêvã"ëâ=ã"æçàÀêèöOåbà¿îæçàã"÷èëµõÀà¿åbô¿å9óóëïèã"ëî â;ëã@ø.ì å9â@æçàù&ú(ûçüçüçûÞý9þÿ3ëOóëïèãbävý9þÃäå9à¿î(í

â;ëòßÀëàèóëâ ßèï¸ã«ôçëàÀêbã"÷ø }ú (þ ã"÷Àëvëèå9óã ê(ëàèëäå9ôçæåbã"æ9à.ïëäbä"í¸å9àèóë

ì@åbâmßèàÀóôçëå0ä

ïÀïÀôñOæçàÀêðOä"ßèã"ë9ä"óëâ;ëåbä"ó÷}ãã"÷èë«ì+ëæçê(÷ã"â?ã"÷Àë«ðëâ=ãÃàèëãì9ä îOä"æçê(ßèëëÞã3å9ô =ú!!"ßèä;ã"÷èëäNæçíïOä#(ëâ ïëäbä"í¸å9àèóë>ãåbóóëïOãå9àÀóë«ßèïãø .ý$%

¤ "Û

Ë>ï

1ï

}Û_íið ¯Û}ÙÚÜ

32mì

&('

2« I¬

« ¬

)"*+-,/.10

)"*+-,/.10

Ø

> 36:

> :*%0

?6@

_« I¬

243%.536*+-78,9%+;: =(70

> 36:

Ø

?5Ö'×'×'Ö%?6@

A-ACB?%D(E

?6@-@

?5Ö'×'×'ÖA;@

?%GCB?%D(E

?6@-@

?5Ö'×'×'Ö&-G('HB?5Ö'×'×'Ö%?JIKG(E

?5Ö'×'×'ÖL;@

?%MCB?%D(E

?6@-@

?5Ö'×'×'Ö%?6@-@-@FB?5Ö'×'×'ÖI(@(G(E

?5Ö'×'×'ÖI(@

A-&CBA-G(E

?6@-@

?5Ö'×'×'Ö%?6@-@-@FB?5Ö'×'×'ÖM-A-G(E

?5Ö'×'×'Ö&;@

IKACBNI(@KE

?6@-@

?5Ö'×'×'Ö';M('HB?5Ö'×'×'ÖIKL;@KE

³

)O+;PQ73RMS0TASUOVW3%X

Long Short-Term Memory in Recurrent Neural Networks

Long Short-Term Memory in Recurrent Neural Networks

Suggest Documents

Long Short-Term Memory in Recurrent Neural Networks

Long Short-Term Memory in Recurrent Neural Networks

Optimizing Long Short-Term Memory Recurrent Neural Networks

Applying long short-term memory recurrent neural

A Long Short-Term Memory Recurrent Neural

Modelling Memory Functions with Recurrent Neural Networks

Memory and Information Processing in Recurrent Neural Networks

Recurrent Neural Networks

Long Short-Term Memory Based Recurrent Neural Network ...

Long Short Term Memory Recurrent Neural Network based ...

A Long-Short Term Memory Recurrent Neural Network Based ... - MDPI

Predictive State Recurrent Neural Networks

Network of Recurrent Neural Networks

Recurrent Neural Networks - Google Sites

Pixel Recurrent Neural Networks - arXiv

Recurrent Neural Networks Learn Deterministic

RNNSecureNet: Recurrent neural networks for

Random Recurrent Neural Networks Dynamics

Composite Recurrent Neural Networks for Long- Term Prediction of ...

Composite Recurrent Neural Networks for Long- Term Prediction of ...

CAPACITY AND TRAINABILITY IN RECURRENT NEURAL NETWORKS

Microscopic instability in recurrent neural networks

Recent Advances in Recurrent Neural Networks

exploring sparsity in recurrent neural networks - OpenReview