TUTORIAL ON PRACTICAL PREDICTION THEORY FOR ...


      !  " $#%#&'( ) *+-,/.021/.-3/45+-6&798;:=@6%ACBDAE1/6GFG, H9I;JLKNMNONP;K2Q!RTS!UDVXWZY\[;WZW/]_^`WZVXY9a;bcS\U;VXYLdcVXegfdch;S\egbjiT^`fDUV dlk W-VXm(a_^`YLd-egfTY\no^`WZWcV p_Yl^rq dcVXegfsWZ[DYlY\S\WZW9SLtg^`nX[_^rdcVXegf 8 VXm(a;nXVXYl^rdZVXegfDWvuwegbvnXSl^`bcf;VXfDx^`nXxgegbcV dch;myU;S\WcVXxgf 8 ^`f;U[;WcS\W VXfznXSl^`bcfDVXf;x!^`nXxgegbjV dZhDm{SL|SlY\[DdcVXegf~}C&hDVXW€dZ[DdcegbjVo^`n~VXW€m(Sl^`f'd?dZe"]NS^"Y\egm(a;bjS\h;S\f;WZV t'S Y\egm(a;VXno^rdcVXegfe`ubcS\Wc[;n dZWz‚&h;VXYƒh ^`bjS]„e`dchdZh;S\egbjSLdZVXYl^`nXn i bcVXxgegbjeg[;W…^`f;U a;bZ^`YLdcVXYl^`nXn i [;WcSLu†[;n‡} Gh;S\bjS9^`bjS"^"dˆ‚?e(VXm(aNegb=dƒ^`f'd&VXm(a;nXVXYl^rdZVXegfDW?e`u‰dch;SbjS\WZ[;n dcW%aDbcS\WZS\f'dZS\Uh;S\bjSgŠ ‹jŒŽ F egm(m(egfaDbZ^`YLdcVXYlS\W/uwegb/bjS\aNegbjdcVXf;xbcS\WZ[Dn dZW-VXfYlno^`WcWZV p_Yl^rdcVXegfWZh;eg[DnXU)Yƒh_^`f;xgS dce([;WZS-dch;S-dcS\Wcd%WZSLd&]„eg[Df;U~} ‹†' ‘bc^`VXfWZSLd-]Neg[;f;UDW-Yl^`fTWcegm(SLdZVXm(S\WG]NS9[;WcS\UdcezU;VXbjSlYLdcn im(e`dcV tg^rdcS9nXSl^`bjf;VXf;x ^`nXxgegbjV dZhDm(WŽ}

„’ “z”`•?–?—€˜&™%š&›E–€œ=˜%• 9ž‡Ÿ„ r l¡w¢‰£¤` "Ÿ„¤Ž£¥=¦5§‰¨'©Ž¡†ª„§‰ 9«(¬5¡‡¨`¬ ­‰Ÿ„¤l©r¡®©r¡wª¯§ ŸT Ž£'©°¡w§~©Žª©\«°ª£ž†£±s£§~©` ²=©Ž¬5£) Ž£'©9ª„¥&¤`ŸN¡†§~³ ´ Ÿ;³‘ 9ŸN§ ´ ©Ž¬5£) l£©°ª„¥% Ž¦5§5§2³ ´ Ÿ;³‘ rµ “ 9ž‡Ÿ„ r Ž¡®¢‰£¤r °ŸN¤r£(©r¬5£)±ª~ \©! l¡†±s­5ž†£…§5ª¯§~©Ž¤r¡w¶2¡‡ŸNž ´ £¨¡† Ž¡wª¯§ ±ŸN·2¡†§5¸)£ž†£±s£§~©" Žª \©r¦ ´ ³2¡†§5¸©Ž¬5£(©r¬5£ª¯¤Ž³TªN¥€žw£DŸN¤r§5¡w§‰¸)¨ž†Ÿ¯ Ž Ž¡w¢‰£¤` -¡† v¶„£¤r³¥=¦5§ ´ ŸN±s£§~©`ŸNž‘©rª  l©Ž¦ ´ ³~¡†§5¸©r¬5£…©Ž¬5£ª„¤r³ª„¥Gžw£DŸN¤r§5¡w§‰¸‰¹„¡†§ ¸¯£§5£¤rŸ„ž “ 9ž†Ÿ¯ Ž Ž¡w¢‰£¤` "Ÿ„¤Ž£ Ž¦‘º¨'¡†£§~©ržw³¨'ª¯±s­5žw£»©r¬‰Ÿ_© ±ŸN§2³¼­5¬5£§5ª„±s£§‰Ÿ½ª¯¾‰ l£¤Ž¶¯£ ´ ¡†§¿±Ÿ„¨`¬5¡†§5£žw£DŸN¤r§5¡w§‰¸{²ˆ©r¬5£ª¯¤Ž£©Ž¡‡¨Ÿ„žwž†³Àª¯¤£»‘­E£¤Ž¡†±s£§~©rŸ„žwž†³5µ ¨ŸN§Á¾C£ª„¾‰ Ž£¤r¶„£ ´ ¡w§À©Ž¬‰£¨ž†Ÿ¯ Ž Ž¡®¢C¨Ÿ_©r¡wª¯§Á Ž£'©l©r¡w§‰¸ “ £©¹&¨ž†Ÿ¯ Ž Ž¡®¢E£¤` …Ÿ„¤Ž£ l¡†±s­5žw£s£§5ª„¦5¸¯¬½©rª ±ŸN·¯£©Ž¬5£¡w¤ŸN§‰Ÿ„žw³‘ Ž¡† )£DŸ„ Ž³½©Žª¦‰§ ´ £¤` l©rŸN§ ´ “à ¬‰¡† ¨ª„±¾5¡w§EŸ_©Ž¡†ª„§¼ª„¥! Ž¦‘º¨'¡†£§~©T³¯£'©±s¡w§‰¡®Ä ±ŸNž¨ª„±s­5ž†£'»‘¡®©\³$¥=ª¯¤¨ŸN­5©Ž¦5¤r¡w§‰¸ ­5¬5£§5ª„±s£§‰Ÿ ±Ÿ„·„£ ©Ž¬‰£s l©Ž¦ ´ ³½ªN¥°¨'ž‡Ÿ„ r l¡w¢‰£¤` …£ Ž­E£D¨'¡‡ŸNž†žw³ ¥=¤r¦5¡w©l¥=¦5ž “ à ¬5£¸„ª~ŸNž&ª„¥v©Ž¬‰¡† ­‰Ÿ„­E£¤…¡‡ )ŸN§½¡†§~©Ž¤rª ´ ¦‰¨g©r¡wª¯§©Žª©Ž¬5£©Ž¬‰£ª„¤r³$ªN¥v­5¤r£ ´ ¡†¨'©Ž¡†ª„§½¥=ª¯¤¨ž†Ÿ¯ Ž Ž¡wÄ ¢E¨Ÿ_©r¡wª¯§ “zÅ ŸN§2³ªN¥©r¬5£ Ž£¤r£ Ž¦5ž®©` z¬‰Ÿ;¶¯£¾C££§Á­5¤Ž£D l£§~©Ž£ ´ £ž† Ž£«(¬‰£¤r£„¹EŸ„ž®©r¬5ª„¦‰¸„¬$©Ž¬‰£ l©\³~ž†£„¹ £žw£±£§~©rŸNž‘§EŸ_©Ž¦‰¤Ž£¯¹„ŸN§ ´ ¸¯£§5£¤rŸ„žw¡w©\³ªN¥C©Ž¬5£­5¤Ž£D l£§~©rŸ_©r¡wª¯§T±Ÿ;³¾C£z§5£« “/à ¬5¡‡ ¡‡ vŸ…©Ž¦‘©rª„¤r¡†Ÿ„žc¹  Žª)«9£!ž†¡w±s¡w©"ª„¦5¤­‰¤Ž£D l£§¯©`Ÿ_©r¡wª¯§T©rª…©r¬5ª¯ Ž£(¤r£ Ž¦5ž®©` -«(¬5¡‡¨`¬sŸ„¤Ž£!¾CªN©r¬©r¬5£ª¯¤Ž£©Ž¡‡¨Ÿ„žwž†³T Žª„¦5§ ´ Ÿ„§ ´ ­5¤`Ÿ„¨'©Ž¡‡¨Ÿ„žwž†³¦‰ Ž£'¥=¦5ž “ à ¬5£¤Ž£zŸ„¤Ž£z Ž£¶¯£¤`ŸNž‘¡†±­Cª„¤Ž©rŸ„§~©9Ÿ„ Ž­E£D¨g©` ªN¥?ž†£ŸN¤r§5¡†§5¸)«(¬5¡‡¨`¬s©Ž¬5£(©r¬5£ª¯¤Ž³¬5£¤r£¨Ÿ„ l©r ž†¡w¸¯¬~© ª¯§ “GÆ £¤r¬‰ŸN­E ?©Ž¬‰£v±sª¯ l©&¡w±s­Cª„¤Ž©rŸ„§¯©%ªN¥2©Ž¬5£D l£"¡† &©Ž¬5£"­5¤Žª¯¾5ž†£±@ªN¥‘­E£¤l¥=ª¯¤Ž±Ÿ„§‰¨'£¤r£­Cª„¤Ž©Ž¡†§5¸!¥=ª„¤ ¨ž†Ÿ¯ Ž Ž¡®¢E£¤`  “/Å ŸN§2³T­C£ª¯­5ž†£¦‰ Ž£ Žª„±s£(¥=ª„¤r±ÇªN¥?£±s­5¡†¤Ž¡‡¨Ÿ„ž‰¶_ŸN¤r¡‡ŸN§‰¨£!©ŽªT£D \©r¡w±ŸN©Ž£z¦5­‰­E£¤°Ÿ„§ ´ ž†ª_«9£¤(¾Eª¯¦5§ ´   “(à ¬5¡‡ z¡‡ ŸN§£¤Ž¤rª„¤ŽÄƒ­5¤Žª¯§5£­5¤rŸ¯¨g©r¡†¨£„¹EŸ„§ ´ ©Ž¬‰£©Ž£D \©… Ž£'©¾Eª¯¦5§ ´ ¡†§½ l£D¨g©r¡wª¯§½È ¡†±s­5žw¡†£ 9Ÿ)¾E£©l©r£¤°±£©Ž¬5ª ´ ¾2³§5£Ÿ„¤Žž†³Ÿ„§~³±s£'©Ž¤r¡‡¨ “-É ª„­C£'¥=¦‰žwž†³„¹~©Ž¬‰¡† "«(¡wž†ž‰¾C£¨ª„±s£¨'ª¯±s±ª¯§ ­5¤`Ÿ„¨'©Ž¡‡¨'£ “ Ê ¥ˆ©r£¤ ´ ¡‡ Ž¨¦‰ Ž Ž¡†§5¸Ë©Ž¬‰£©Ž£D \©$ l£©¾Cª„¦5§ ´ «°£À¨ª_¶„£¤s©Ž¬5£Ę̈¨Ÿ„±$Í  ΏŸNϪ¯¤¾Cª„¦5§ ´ ¹(©Ž¬5£  Ž¡w±s­5ž†£ l©(©Ž¤`ŸN¡†§ l£©¾Cª„¦5§ ´ ¹‰«(¬5¡‡¨`¬$£'»‘­5ž‡ŸN¡†§‰ ²jŸ„§ ´Ð ¦‰ŸN§~©r¡®¢‰£D rµ9©r¬5£T¨ª„±s±sª„§$­5¬5£§5ª„±s£§EŸ ª„¥Tª_¶„£¤l¢5©Ž©Ž¡†§5¸Ñ¡†§Ò Ž¡w±s­5ž†£ l©$¥=ª„¤r± “ÇÓ £ËŸNž‡ lª{­5¤rª_¶„£©r¬‰Ÿ_©©Ž¬5£¿Ì¨¨ŸN±$Í  ΏŸNϪ¯¤ ¾Cª„¦5§ ´ ¨ŸN§5§5ª„©°¾C£)¡w±s­5¤rª_¶„£ ´ «(¡w©Ž¬5ª¯¦‘©(¡w§E¨'ª„¤r­Cª„¤`Ÿ_©Ž¡†§5¸£'»2©Ž¤`ŸT¡†§‘¥=ª¯¤Ž±Ÿ_©r¡wª¯§Ÿ„§ ´ Ÿ„­5­5ž†³©Ž¬5£)¾Cª„¦5§ ´ ©rª ´ £¨'¡‡ Ž¡wª¯§ ©r¤Ž££  “ Ô


     ! " £»2©¹&«°£ ´ ¡‡ Ž¨¦‰ r ©\«9ª©Ž¤`ŸN¡†§  Ž£'©¾Cª„¦‰§ ´  ¹G©Ž¬‰£ Æ Ê "Ä$#!Ÿ;³„£D ¾Cª„¦5§ ´ Ÿ„§ ´ ©Ž¬5£&%5ŸN±s­5ž†£ 9 ª¯±­‰¤Ž£D Ž Ž¡wª¯§½¾Cª„¦5§ ´ ¹&«(¬5¡‡¨`¬À¬EŸ;¶„£T­‰¤Žª_¶¯£ ´ ©rª$¸„¡†¶„£­5¤rŸ¯¨g©r¡†¨ŸNž-¤Ž£D l¦‰ž®©` ¥=ª¯¤±sª„¤r£¸¯£§5£¤rŸ„ž ¨ž†Ÿ¯ Ž Ž¡®¢E£¤` ¹‘ Ž¦‰¨`¬Ÿ¯ '%2¦5­5­Cª„¤Ž©)(£¨'©Žª„¤ Å Ÿ„¨`¬5¡†§5£ (Ÿ„§ ´ " £¦5¤rŸ„ž " £©\«9ª¯¤Ž·‘  “ Ê ž†žªN¥©r¬5£s¤Ž£D l¦‰ž®©` ¬5£¤Ž£ l¬‰ª„¦5ž ´ ¾C££DŸ„ Ž¡wž†³ŸN­5­‰¤Žª~Ÿ„¨`¬‰Ÿ„¾5žw£ŸN§ ´ ¦5§ ´ £¤r l©rŸ„§ ´ ŸN¾‰žw£ “à ¬5£ ­5¤rª2ªN¥j °ŸN¤r£ Ž¡†±­‰žw£¯¹5ŸN§ ´ £'»5ŸN±s­5ž†£ !ŸN¤r£¸„¡†¶„£§ “/Æ ª¯¡w§~©Ž£¤r "©Žªs¤r£ž‡Ÿ_©r£ ´ «°ª„¤r·ŸN¤r£Ÿ„ž† ŽªT¸¯¡w¶¯£§ “ * ©¡‡ ¡w±s­Cª„¤Ž©rŸN§~©©rª½§5ª„©Ž£©r¬‰Ÿ_©sŸNž†ž"ª„¥(©Ž¬5£¤Ž£D l¦5žw©r T­5¤r£ Ž£§~©Ž£ ´ ¬5£¤r£¥jŸNž†ž"¡†§Ë©r¬5£ ¤r£Ÿ„žw± ª„¥-¨ž†Ÿ¯ Ž Ž¡†¨ŸNžG l©rŸN©Ž¡‡ \©r¡†¨  “ * §­‰ŸN¤Ž©Ž¡‡¨'¦5ž‡ŸN¤D¹5ŸNž†žG¤`ŸN§ ´ ª„±s¡wÏDŸ_©r¡wª¯§‰ (ŸN¤r£)ª_¶„£¤ ´ ¤rŸ;«z !ª„¥%©r¬5£ ´ ŸN©rŸ5¹ Ÿ„§ ´ ª¯¦5¤(¤r£ Ž¦5ž®©` !¬‰Ÿ;¶„£…©Ž¬‰£)¥=ª„¤r± ªN¥¨'ª¯§‘¢ ´ £§‰¨'£)¡†§¯©r£¤r¶_ŸNž‡  “ à ¬5£ž‡Ÿ;³„ª¯¦‘©°ª„¥%©r¬5¡‡  ´ ª‘¨'¦5±s£§¯©(¡‡ zŸ„ °¥=ª¯žwž†ª_«z ,+ § .­5¤r£ Ž£§~©r °©r¬5£)¥=ª„¤r±ŸNž€±sª ´ £ž - %2£¨'©Ž¡†ª„/ - %2£¨'©Ž¡†ª„§È­5¤r£ Ž£§~©r °©r¬5£)©Ž£D \©z Ž£'©z¾Cª„¦5§ ´ § 0s­5¤r£ Ž£§~©r °©r¬5£Ì¨¨ŸN±$Í  !ÎzŸ„Ϫ¯¤°¾Cª„¦‰§ ´ - %2£¨'©Ž¡†ª„1 § 2­5¤r£ Ž£§~©r °©r¬5£ Æ Ê "3Ä #!Ÿ;³„£D "¾Cª„¦‰§ ´ - %2£¨'©Ž¡†ª„/ § 4­5¤r£ Ž£§~©r °©r¬56 £ %‘ŸN±s­5ž†£T9ª„±s­5¤r£ r l¡†ª„§¾Eª¯¦5§ ´ - %2£¨'©Ž¡†ª„5 ¬Ã 5£¥=ª¯¤Ž±ŸNž-±sª ´ £ž9ŸN§ ´ ©Ž£D \© Ž£'©¾Eª¯¦5§ ´ ±T¦E \©¾C£¦5§ ´ £¤r l©Žª2ª ´ ¡†§¼ª„¤ ´ £¤©ŽªŸ„­5­5¤r£¨'¡‡Ÿ_©r£ Ÿ„žwž-ž‡Ÿ_©Ž£¤)¤Ž£D l¦5žw©r  “Tà ¬5£¤Ž£¡‡ )§5ª­EŸN¤Ž©Ž¡‡¨'¦5ž‡ŸN¤ ´ £­C£§ ´ £§‰¨³½¾C£'©\«°££§À©Ž¬‰£¶_Ÿ„¤Ž¡†ª„¦‰ ©Ž¤`ŸN¡†§À Ž£'© ¾Cª„¦5§ ´  !«9£)­5¤r£ Ž£§~© “

“ °˜&—8:9@?BA C8DE 7GF FHIKJ I

à ¬5£…©Ž¤r¦5££¤r¤Žª¯¤"¡‡ ° Žª„±s£©Ž¡†±£D 9¨ŸNž†žw£ ´ ©r¬5£ tl¸„£§5£¤`ŸNž†¡†ÏŸ_©r¡wª¯§£¤r¤Žª¯¤vu “NM §‘¥=ª„¤Ž©Ž¦‰§‰Ÿ_©r£ž†³„¹~©Ž¬5£ ©r¤Ž¦5££¤Ž¤rª„¤(¡‡ z§‰ªN©…Ÿ„§ª¯¾‰ Ž£¤r¶;Ÿ„¾5ž†£ Ð ¦‰Ÿ„§~©Ž¡w©\³¡†§ª„¦5¤±ª ´ £ž&¾C£¨ŸN¦‰ Ž£)©Ž¬5£ ´ ¡† l©Ž¤r¡†¾5¦‘©Ž¡†ª„§&¹ i ¹ ¡‡ !¦5§5·2§5ª_«(§ “É ª_«9£¶„£¤D¹~©Ž¬5£¤Ž£)¡‡ zŸ¤Ž£ž†ŸN©Ž£ ´Ð ¦‰Ÿ„§¯©r¡®©\³ «(¬5¡†¨`¬¡‡ (ª„¾‰ Ž£¤r¶_ŸN¾5ž†£ “ £ TWVSXZY[\Y](^B _TW[\[ [g¹aq8` b  W-1TUTN VN2U3$ZPOP?BA CWDdE b F FHZIeJ

I

` n

e

hi

g

jNkNl

i

`:L

F q FH IeJ

i

I

«(¬5£¤Ž£ ¡‡ -Ÿ(¥=¦5§E¨g©Ž¡†ª„§«(¬5¡‡¨`¬±ŸN­‰ t\©Ž¤r¦5£uv©rª e ŸN§ ´ tL¥jŸ„ž† Ž£u©rª ) “ Ê ž‡ Žª‰¹ ;=< >@?A CWDE b ¡†  Ÿ!­5¤rª„¾EŸNl ¾5FmI ¡†žw¡w©\³z©`ŸN·¯£§«(¡w©Ž¬¤r£ Ž­E£D¨g©G©rª(©Ž¬5£"¦5§5¡w¥=ª„¤r± ´ ¡‡ l©Ž¤r¡w¾5¦5©Ž¡†ª„§ª_¶„£¤?©Ž¬‰£" l£©&ªN¥‘£»‘Ÿ„±sFonP­5n@npžwI £D ¹ m “ à ¬5££±­‰¡w¤r¡†¨ŸNžG£¤Ž¤rª„¤!¡‡ z Žª„±s£'©r¡w±s£ ¨ŸNž†ž†£ ´ ©Ž¬5£:t\©Ž¤`ŸN¡†§5¡†§5¸s£¤r¤Žª¯¤vug¹tL©r£ l©z£¤r¤Žª¯¤vug¹2ª„¤ tlª„¾‘Ä  Ž£¤r¶„£ ´ £¤r¤Žª¯v¤ u ´ £­C£§ ´ ¡w§‰¸ ª„§Á«(¬5£'©r¬5£¤¡®©)¡‡ z©r¬5££¤Ž¤rª„¤…¤rŸN©Ž£ª¯§ÁŸ©r¤rŸ„¡w§‰¡w§5¸ l£©¹C©Ž£D \©) l£©¹ ª¯¤!Ÿ±sª„¤r£)¸„£§5£¤`ŸNž? Ž£'© “   L!#"PWZ ²c¨'ª¯§¯©r¡w§2¦5£ ´ µ à ¬5£z¨ž†Ÿ¯ Ž Ž¡®¢E£¤ q «(¬5¡†¨`¬Ÿ„žw«!Ÿ;³‘ %­‰¤Ž£ ´ ¡†¨'©r f> – >‘–=>‘– &˜%™ à ¬5£ Ž¡w±s­5ž†£ l©z¾Cª„¦5§ ´ Ÿ„¤Ž¡‡ Ž£ 9¥=ª„¤(©r¬5£¨'ž‡Ÿ„ r Ž¡†¨ŸNž€©Ž£D¨`¬5§5¡ Ð ¦5£ªN¥-¦‰ Ž¡w§5¸ n ¥=¤r£ Ž¬$£'»5ŸN±s­5ž†£  ©rª£¶_ŸNž†¦‰Ÿ_©r£)Ÿs¨ž†Ÿ¯ Ž Ž¡®¢E£¤ “-à ¬‰¡† ( Ž£¨'©Ž¡†ª„§¡† (ª¯¤Ž¸~ŸN§5¡†Ï£ ´ ¡†§~©Žª©\«9ª Ž¦5¾‰ Ž£¨'©Ž¡†ª„§‰ ,+ - %2¦5¾‰ Ž£¨'©Ž¡†ª„§sÈ “†’ ­‰¤Ž£D l£§¯©` G©r¬5£!¾‰Ÿ„ Ž¡‡¨"¦5­‰­E£¤¾Eª¯¦5§ ´ ª„§©r¬5£°©Ž¤r¦5£!£¤r¤Žª¯¤%¤`Ÿ_©r£„¹_¬‰Ÿ„§ ´ ³ ŸN­5­‰¤Žª;»‘¡†±sŸN©Ž¡†ª„§‰ ¹‘ŸN§ ´ Ÿžwª_«°£¤!¾Cª„¦5§ ´ - %2¦5¾‰ Ž£¨'©Ž¡†ª„§ È “ . ´ ¡‡ Ž¨¦‰ r l£D ©Ž¬5£½¡†±s­5ž†¡†¨Ÿ_©Ž¡†ª„§E ªN¥©Ž¬5£½©Ž£ l© Ž£'©¾Cª„¦5§ ´ ª¯§ £¤r¤Žª¯¤ ¤Ž£­Eª¯¤l©r¡w§‰¸…­5¤`Ÿ„¨'©Ž¡‡¨'£ “ Ê ¾E£©l©r£¤±s£'©r¬5ª ´ ¥=ª¯¤/£¤Ž¤rª„¤/¤r£­Cª„¤Ž©Ž¡†§5¸…¡‡ -ŸN­5­‰žw¡†£ ´ ©Žª) l£¶„£¤rŸ„ž ´ Ÿ_©`Ÿ„ Ž£'©` (ŸN§ ´ ©Ž¬5£¤r£ Ž¦5žw©r (Ÿ„¤Ž£ Ž¬5ª_«(§ “ È “w’¯“ WRJ62 STUw/ Z #°£'¥=ª„¤r£v l©rŸ_©r¡w§‰¸!©Ž¬5£v¾Cª„¦‰§ ´ ¹;«9£"§5ªN©r£"Ÿ°¥=£«¼¾‰Ÿ¯ l¡‡¨ª„¾E l£¤Ž¶_Ÿ_©r¡wª¯§‰ €«(¬5¡‡¨`¬ ±ŸN·¯£z©r¬5£…¤r£ Ž¦5žw©r °ž†£ r 9 Ž¦5¤r­5¤Ž¡‡ Ž¡w§5¸ “/à ¬‰£…­5¤r¡†§‰¨'¡†­‰ŸNž€ª¯¾‰ l£¤Ž¶_Ÿ„¾5žw£ Ð ¦EŸN§~©Ž¡w©\³¡† 9©r¬5£…£±s­5¡w¤r¡‡¨ŸNž £¤Ž¤rª„¤ q8` b ªN¥9Ÿ¨'ž‡Ÿ„ r l¡w¢‰£¤ “Ó ¬‰ŸN©¡‡ …©Ž¬5£ ´ ¡‡ l©Ž¤r¡w¾5¦5©Ž¡†ª„§Áª„¥v©Ž¬‰££±s­5¡w¤r¡‡¨ŸNž-£¤Ž¤rª„¤¥=ª„¤Ÿ¢5»‘£ ´ ¨ž†Ÿ¯ Ž Ž¡®¢E£\¤ ' ‰ª„¤(£DŸ„¨`¬£'»5ŸN±s­5ž†£„¹5ª¯¦5¤!¡w§ ´ £­C£§ ´ £§‰¨£TŸ„ r l¦‰±­5©Ž¡†ª„§¡w±s­5ž†¡w£D !©Ž¬‰£­5¤Žª¯¾‰ŸN¾‰¡wž†¡®©\³

 





 






    !

 !     !



 



! , [ !

 

©r¬‰Ÿ_©©r¬5£ ¨ž†Ÿ¯ Ž Ž¡®¢E£¤£¤r¤` ¡† ¸„¡†¶„£§¼¾2³À©Ž¬5£©r¤Ž¦‰££¤r¤Žª¯¤¹ qW7 “à ¬‰¡† ¨ŸN§¼¾C£ ±sª ´ £ž†£ ´ ¾2³¼Ÿ ¾5¡‡Ÿ„ Ž£ ´ ¨'ª¯¡w §
‰¡†­ +¬5£DŸ ´  !¡w¥/³¯ª„¦$ŸN¤r£)¤Ž¡†¸„¬~©(ŸN§ ´ ©rŸ„¡wž‡ !¡w¥/³¯ª„¦$ŸN¤r£«(¤rª„§‰¸ “ ¸ Á£¤r¤Žª¯¤r ²=¬5£DŸ ´  `µ…ª„¦‘©)ªN¥ n £'»5ŸN±s­5ž†£ s²j¨ª„¡†§ Ó ¬‰ŸN©)¡‡ ©Ž¬‰£­‰¤Žª¯¾‰ŸN¾5¡†ž†¡®©\³$ªN¥°ª„¾‰ Ž£¤r¶2¡w§‰
‰¡†­‰ `µ ' à ¬‰¡† s¡‡ Ÿ¼¶„£¤Ž³¼¥jŸ„±¡†ž†¡†Ÿ„¤ ´ ¡† l©Ž¤r¡†¾5¦‘©Ž¡†ª„§ ¡†§  l©rŸN©Ž¡‡ \©r¡†¨ ¨Ÿ„žwž†£ ´ ©Ž¬5/ £ #°¡†§5ª„±s¡‡ŸNžzŸ„§ ´  ŽªË¡®©$ l¬‰ª„¦5ž ´ ¾E£À¦5§‰ Ž¦5¤r­5¤Ž¡‡ Ž¡w§5¸¼©r¬‰Ÿ_©©Ž¬5£À¾Eª¯¦5§ ´  ­5¤r£ Ž£§~©Ž£ ´ ¬5£¤r£ÁŸN¤r£¥=¦5§ ´ ŸN±s£§~©`ŸNž†žw³ Ÿ #9¡†§5ª¯±¡‡ŸNž “ ´ £­E£§ ´ £§¯©z¦‰­Eª¯§©r¬5£¨'¦‰±T¦5ž‡Ÿ_©r¡w¶¯£ ´ ¡‡ l©Ž¤r¡w¾5¦5©Ž¡†ª„§$ªN¥s  W-1TUTN VN2U O D

'

/

!

 







3

 )

`

F*

 ;=E
g g
k D “ %2¡†§‰¨'£)©r¬5£   Ä ´ ¡†¶„£¤Ž¸¯£§‰¨£…Ÿ„­5­5ž†¡w£D !©Žª ´ ¡‡ \©r¤Ž¡†¾5¦‘©r¡wª¯§‰ (ª_¶¯£¤(¨'ª¯§¯©r¡w§2¦5ª¯¦‰ !¶_ŸNž†¦5£ ´ ­‰Ÿ„¤rŸ„±Ä £©Ž£¤` ¹5©Ž¬‰£ Æ Ê "Ä3#!Ÿ;³¯£ !¾Cª„¦5§ ´ ¨ŸN§¾C£T§‰ª„§~©Ž¤r¡w¶2¡‡ŸNž†žw³ ©Ž¡†¸„¬~©¡†§©Ž¬‰¡†  l£©l©r¡w§5¸Ÿ„ z«°£ž†ž “zà ¬5¡†  ¥jŸ¯¨g©(¡† z¦‰ l£ ´ ¡†§©Ž¬5£Ÿ„­5­5ž†¡†¨Ÿ_©r¡wª¯§ Ž£¨'©Ž¡†ª„§ “ Ó £(¢‰¤` \©" l©rŸN©Ž£zŸ¨'ª¯¦5­5ž†£z l¡†±s­5ž†£zžw£±s±sŸ¯ -©Ž¬EŸ_©9Ÿ„¤Ž£!¬‰Ÿ„§ ´ ³T¡†§s©Ž¬‰£z­5¤rª~ª„¥ “/à ¬‰£(¡w§~©Ž¦‰¡®©r¡wª¯§ ¾C£¬5¡†§ ´ ©Ž¬‰¡† (ž†£±s±Ÿ¡† (©Ž¬‰ŸN©(©Ž¬5££»2­C£¨'©Ž£ ´ ­5¤rª„¾EŸN¾5¡†žw¡w©\³ª„¥/ŸN§$£¶„£§~©(¡‡ !§5ªN©(©rª~ª Ž±ŸNž†ž “ W- xL Z$Z ;x B[ ^  F q I ? ;x B[ ^  ! B F )g,eED 






  E
b E; 7 < 

J [ + I@EK " Nª ©Ž£©r¬‰Ÿ_©+ b E 7  q ;=< b E 7 

e

+

F

q` b



;=   D g



3

 

` F b

oHo



7 I

1 W&‚€SŽ^gS\f;S\U‚&V dch…dch;SbjSlno^rdcV t'SvS\fdcbcega'i F h;S\bjf;e T]Neg[;f;U ‹ 0 S\m(mz^D}   egfdZh;S < VXfDegm(Vo^`n‡}

J [ + I@EK




    !

k

E

g

+




n

` +

E q8` b

e 4

qW7

F

¯£§‰ Ž£§&Í  °¡†§5£ Ð ¦‰Ÿ„žw¡w©\³Á²XwÀ¨'ª¯§‰¨Ÿ;¶¯£ 1

E  q8` b +

+

ŸN§ ´

e



4

qW7

E

F





n n



! , [ !

Ô

 

k

b 



 

    > k    D      

q`

n

 !     !

+q ` b

q`

  b

w +



e)d

q8` b

e)d

e



I

I ^

F

E +

n

I

w



g

^

+

F ` b

-



d

q87

e)d

qW7

e)d

+

e



I

n

4

7 +

`

n

q`

n

 b

n

e







µ+



 >   D

E 

`



e)d

E n  n

F

b q8` b

I

'

e



e)d



 7

 ` b 

Ó ¡w©Ž¬$©Ž¬‰£ Ž£…©\«°ªsžw£±s±sŸ¯ ¹2©r¬5£ Æ Ê "Ä3#!Ÿ;³¯£ 9©Ž¬5£ª„¤r£± ¡‡ (£DŸ„ Ž³s©Žª­5¤rª_¶„£ “ J [ + I@EKT²ƒÌ¥%©r¬5£ Æ Ê "$Ä #!Ÿ;³„£D "©r¬5£ª¯¤Ž£±µ  £'© 

) 

` 

q g q`

b I

`



;=< b E 7

b

+

+



+

+

q`



+

L

e

k >    D F q I    E
 S    j    F I   q e    o o  ` F I ; <  q8` b  E
E b 7  F I E F I ;=< b E 7 F q ` b ` q ` b  I Fq I e e    o o d  E    E
F I ;= u A k D l

 E > u A k D





  E S> A k D l   E > u A k D  E > u A k D  +

+

/

+

'0'



/



`



/

+





-



+

l



L  F L  F 

L -





H







H

 4

4



 

dGL

   ) H I     d

H

L 

'

)

I

 H   H



'




    !

 !     !



 



! , [ !

`Ô

 

 l¡†§5¸©Ž¬5£ Ž³2±s±£©Ž¤r¡†¨¡®©\³ ªN¥&©r¬5£ )R ŸN¦E Ž Ž¡†Ÿ„§G¹2©Ž¬‰¡† (¡‡ ,+

M

     L L L   H  b  E > u A k D  E S> u A k D H l  l¡†§5¸©Ž¬5£)¥jŸ¯¨g©(©r¬‰Ÿ_©!©Ž¬‰£ l¦5± ªN¥/©\«9ª R)ŸN¦E Ž Ž¡†Ÿ„§‰ !¡‡ (Ÿ R)Ÿ„¦‰ r l¡‡ŸN§+ 
` ?BA C8E b  L L I
E u A k
 % %    l F  
?A C8E b E  A

` L L I u   % F `

M

+

?
A C8E

+

+



+

+

+

-

+

`

-

  

l

?
A C8E b   F -

+

¢‰§‰¡† Ž¬5¡†§5¸©Ž¬5£­5¤rª2ªN¥ “

  

`

+F  

Ib

g-

oo



F  

g

I7

-

g

F H

 Ž¡†§5¸s©Ž¬5£¨ª„¤rª„ž†ž†Ÿ„¤Ž³¯¹„©r¬5£)©Ž¤r¦5£)£¤r¤rª„¤!¾Eª¯¦5§ ´   M

4

-

F  

L

II g-

% /

` '

L  rŸ_©r¡† l¢‰£ °©r¬5££ Ð ¦‰ŸN©Ž¡†ª„§+

I7

g



4

(




k

n

à ¬5¡‡ !¡‡ zŸN§¡w±s­5ž†¡‡¨'¡w©z£ Ð ¦‰Ÿ_©r¡wª¯§ ¥=ª¯¤   «(¬‰¡†¨`¬¨Ÿ„§¾C££Ÿ¯ l¡†ž†³  lª¯žw¶¯£ ´ §~¦‰±£¤Ž¡‡¨Ÿ„žwž†³ “ à ¬5£¾Cª„¦5§ ´ ¡‡ ( l©rŸ_©r£ ´ ¡†§©r£¤r± !ªN¥ ´ ªN©z­5¤rª ´ ¦‰¨g©` !¬5£¤r£„¹5 Žªs§‰Ÿ_©r¦5¤`ŸNž†žw³¡w©z¡‡ !­Eª~ Ž Ž¡w¾‰žw£)©rª ·¯£¤r§5£ž†¡wÏ£©Ž¬5£¤Ž£D l¦5žw©z¦‰ Ž¡w§‰¸±s£'©Ž¬‰ª ´  (¥=¤rª„± ? 4 B “ * §·„£¤Ž§5£žw¡†Ï£ ´ ¥=ª„¤r±¹5©Ž¬5£¾Eª¯¦5§ ´ ŸN­‰­5žw¡†£  ©rªs¨ž†Ÿ¯ Ž Ž¡w¢‰£¤` )²jŸ„ !ª¯¦‘©Ž­‰¦‘©z¾2 ³ % ( Å ž†£Ÿ„¤Ž§‰¡w§5¸sŸNž†¸„ª¯¤Ž¡w©Ž¬‰±s `µ"ªN¥/©Ž¬5£)¥=ª¯¤Ž±1+ ² 2 “†’ µ q

FHI

 l¡†¸„§  `

i g

hi

i

jNk



g

FH

HZI

 i

i

2¡†§‰¨£„¹-¾2³ÀŸ„ r Ž¦5±s­‘©Ž¡†ª„§G¹ ¡‡ TŸ$·¯£¤r§5£žZ¹%«°£·2§5ª_« ©Ž¬‰ŸN© g `      FHZI «(¬‰£¤r£  F H  H I  F H I i i i i   i ¡† s Žª„±s£ ­‰¤Žª \\£¨g©r¡wª¯§¿¡†§~©Žª¼ŸN§5ª„©Ž¬5£¤ Ž­‰Ÿ„¨£ “ i* §{·„£¤r§5£žw¡†Ï£ ´ =¥ ª„¤r±$¹v«9£¸„£©   ` Fg HZI  FH g HZI ¹ H   H  `  FH g HZI ¹      `  FH g H  I ¹ ´ £'¢‰§5¡†§5¸¿Ÿ„žwž ªN¥)H  ©Ž¬5£  A N j k §5£D¨'£D Ž r ŸN¤r³ Ð ¦EŸN§~©Ž¡w©Ž¡†£ !©rª¨ŸNž‡¨'¦‰ž†ŸN©Ž£©Ž¬5£§‰ª„¤r±sŸ„žw¡†Ï£ ´ ± ŸN ¤r¸„¡†§G¹ %





i g

i



i

g jNi k FH i Z H I i A FH I g A g A g   F H g H  I FH HI   jNk k   ̅§5£$£ž†£±s£§¯©s¤r£±ŸN¡†§‰ ¹"«(¬5¡‡¨`¬¿¡‡ ©Ž¬5£$¶_ŸNž†¦5£$ªN¥ - “ M §‘¥=ª„¤Ž©Ž¦5§EŸ_©Ž£žw³Ñ©Ž¬5£$¾Cª„¦5§ gtL

`

 



´ ¨Ÿ„§ ¾C£T§5ª¯§5±sª„§5ª„©Žª¯§5¡†¨¡w§©r¬5£T¶_Ÿ„žw¦5£ªN¥ ¹C¾5¦‘©)¡®©…©Ž¦5¤r§‰ …ª„¦‘©…©Ž¬‰ŸN©¥=ª¯¤…¨'ž‡Ÿ„ r Ž¡®¢‰£¤r žw£DŸN¤r§5£ ´ ¾2³  Ž¦5­5­Cª„¤Ž©9¶¯£¨g©rª„¤°±Ÿ„¨`¬5¡†§5£ 9ª„§¤Ž£DŸ„ Žª„§‰Ÿ„¾5ž†£ ´ Ÿ_©`Ÿ„ Ž£'©` ¹¯©Ž¬5£¤Ž£¡‡ 9ª¯§5ž†³ª¯§5£¶;Ÿ„žw¦‰£…ª„¥ «(¬5¡‡¨`¬ ¡‡ z²=ž†ª‘¨Ÿ„žwž†³„¹2ŸN§ ´ ©r¬~¦E v¸„ž†ª„¾EŸNž†žw³5µ/±s¡w§‰¡w±ŸNž “ Ê ¾5¡†§‰ŸN¤r³ Ž£Ÿ„¤r¨`¬ª_¶„£¤v lª¯±s£(¤Ž£DŸ„ Žª„§‰Ÿ„¾5žw£!¤`ŸN§5¸¯£ ª„¥ ²j rŸ;³¥=¤Žª¯± e ©Žª e )) µ9¨Ÿ„§ ¢E§ ´ ©Ž¬5£±s¡†§5¡w±Ÿ Ð ¦5¡‡¨`·2žw³¯¹‘¸„¡†¶„£§©Ž¬‰£­5¤Ž£D¨'ª¯±­‰¦‘©rŸN©Ž¡†ª„§ªN¥ ©r¬5£±ŸN¤r¸„¡†§‰  “ * ©…¡‡ z«°ª„¤Ž©Ž¬§5ª„©Ž¡†§5¸ ŸN¸~ŸN¡†§$¬5£¤r£©Ž¬EŸ_©…«°£TŸ„¤Ž£§5ª„© tŽ¨`¬5£ŸN©Ž¡†§5¸ u ©Ž¬5£¾Cª„¦5§ ´ ¬5ª¯ž ´  °¥=ª„¤zŸNž†ž?¶_ŸNž†¦5£ !ª„¥  l¡†±T¦‰ž®©`ŸN§5£ª„¦‰ Žžw³ “ à ¬5£!¨'ª¯±­‰¦‘©rŸN©Ž¡†ª„§‰Ÿ„ž„©r¡w±s£°ªN¥‘©Ž¬‰£9¾Cª„¦5§ ´ ¨Ÿ„ž†¨¦5ž‡Ÿ_©Ž¡†ª„§T¡†  ´ ª¯±¡†§‰ŸN©Ž£ ´ ¾~³©r¬5£°¨ŸNž‡¨'¦5ž‡Ÿ_©r¡wª¯§ ª„¥©Ž¬5£±sŸ„¤Ž¸¯¡w§E z«(¬5¡‡¨`¬½¡‡   F n «(¬‰£¤r£ n ¡† ©Ž¬5£§2¦5±T¾C£¤ªN¥" Ž¦5­5­Cª„¤Ž©…¶¯£¨'©Žª„¤` («(¡w©Ž¬ Ÿ §5ª¯§5Ï£¤ŽªŸ„ r lª‘¨¡†ŸN©Ž£ ´ “và ¬5¡‡ (¨'ª¯( ±sH ­5¦‘©rŸN©Ž¡†ª„§‰Ÿ„žC©Ž¡†±£)¡‡ °©\³2­5¡†¨ŸNž†žw³ ´ ª„±s¡w§EŸ_©Ž£ ´ ¾2³©r¬5£©Ž¡†±£ ª„¥&©Ž¬‰6 £ % ( Å ž†£ŸN¤r§5¡†§5¸  ŸNž†¸„ª¯¤Ž¡w©Ž¬5± “ -

-

-



-


     !

1 test set errors test set bound margin bound

error rate

0.8 0.6 0.4 0.2

0 liver bostonpima ion sonar dig1 dig2 adult problem

7!œ &šG —> à ‰¬ ¡† °¢‰¸¯¦5¤r£) Ž¬5ª_«z °©r¬5£)¤Ž£D l¦5žw©r !ª„¥-Ÿ„­5­5ž†³2¡w§5¸ % ( Å ž†¡w¸¯¬¯©(©rª A ´ Ÿ_©rŸ¯ l£©r !«(¡w©Ž¬Ÿ R)ŸN¦‰ r Ž¡†Ÿ„§ ·¯£¤r§5£ž?Ÿ„§ ´ Ÿ IE NÈE©r¤rŸ„¡w§ D©Ž£D \©z Ž­5ž†¡®© “và ¬5£ª„¾‰ Ž£¤r¶„£ ´ ©r£ l©!£¤Ž¤rª„¤!¤`Ÿ_©r£¡‡ !¸„¤`ŸN­‰¬5£ ´ Ÿ„ (Ÿ„§ “ ̅§©Ž¬5£)©r£ l© l£©¹‘«°£¨Ÿ„ž†¨¦5ž†ŸN©Ž£)Ÿ #°¡w§5ª¯±s¡†Ÿ„žG¨'ª„§5¢ ´ £§E¨'£)¡w§~©r£¤r¶;Ÿ„žv²=­5¤rª„¾EŸN¾5¡†žw¡w©\³ªN¥ ¾Cª„¦5§ ´ ¥jŸ„¡wž†¦5¤r£  ) )e µ"«(¬5¡‡¨`¬¦‰­5­E£¤(¾Eª¯¦5§ ´  °©r¬5£©Ž¤r¦5£)£¤r¤Žª¯¤9¤`Ÿ_©r£ “ ̅§ ©r¬5£©Ž¤`ŸN¡†§5¡†§5¸ Ž£'© «9£¨ŸNž‡¨'¦5ž‡Ÿ_©r£…n©r¬5£ Æ Ê "3Ä #!Ÿ;³¯£ °±ŸN¤r¸„¡†§¾Cª„¦5§ ´ ¥=ª¯¤ŸN§ª„­‘©r¡w±s¡†Ï£ ´ ¨`¬‰ª„¡‡¨'£ªN¥ “ 





-

2 “ È “w’¯“ T5458z 43K Ê ­5­5ž†¡‡¨Ÿ_©r¡wª¯§Àª„¥v©Ž¬‰¡† ¾Cª„¦5§ ´ ©Žª$ l¦‰­5­Eª¯¤l©)¶„£D¨g©rª„¤±sŸ¯¨`¬5¡†§5£ …¡† ª„¥9 l¡†¸„§‰¡®¢5Ä ¨ŸN§~©v¡w±s­Cª„¤Ž©rŸ„§‰¨'£z¾C£¨ŸN¦‰ Ž£ % ( Å  °ŸN¤r£(¤Ž£DŸ„ Žª„§‰Ÿ„¾5žw³T£G€£¨g©r¡w¶¯£ŸN§ ´ Ÿ ´ Ÿ„­‘©rŸ„¾5žw£…¨'ž‡Ÿ„ r l¡w¢‰£¤` -¡†§ ¨ª„±s±sª„§$ŸN§ ´ «(¡ ´ £ Ž­5¤Ž£DŸ ´ ¦‰ l£ “ Ê % ( Å ž†£Ÿ„¤Ž§‰ !Ÿ·„£¤Ž§‰£ž†¡wÏ£ ´ ¨'ž‡Ÿ„ r l¡w¢‰£¤(Ÿ¯ !­E£¤(£ Ð ¦‰Ÿ_©r¡wª¯§ 2 “w’ “ Ó £ŸN­5­5ž†³©r¬5£ l¦5­‰­Eª¯¤l©(¶„£D¨g©rª„¤!±Ÿ„¨`¬5¡†§5£©Žª A M  * ´ Ÿ_©`ŸN¾‰Ÿ¯ l£­5¤rª„¾5ž†£± (¨`¬5ª~ l£§©Žª¢5© ©r¬5£ ¨'¤r¡w©Ž£¤r¡‡Ÿ tL©\«°ª½¨'ž‡Ÿ„ r l£D$  uŸN§ ´ tl¤r£ŸNž"¶;Ÿ„žw¦‰£ ´ ¡†§5­5¦‘©¥=£Ÿ_©r¦5¤r£v  u “à ¬5£­‰¤Žª¯¾5žw£± ¶_ŸN¤r³Á¡†§  Ž¡wÏ£)ª_¶„£¤(Ÿ„§ª¯¤ ´ £¤(ªN¥-±ŸN¸¯§5¡®©r¦ ´ £¥=¤rª„± e ©Žª e $
£»‘Ÿ„±s­5žw£D  “ * §¢‰¸¯¦5¤rf£ 2 “ .«°£¦‰ Ž£Ÿ -v IE _[ È E©r¤rŸ„¡w§ ;©Ž£D \©z Ž­5žw¡w©(ªN¥/©Ž¬5£ ´ Ÿ_©rŸ “ £ % ( Å ž†¡w¸¯¬¯©«(¡w©Ž¬ ! Ÿ R)Ÿ„¦‰ Ž Ž¡‡ŸN§ ·„£¤r§5£žzŸN§ ´ ©Ž¬‰£ ´ £'¥jŸ„¦5ž®© * § ŸNž†žz£'»‘­C£¤r¡w±s£§¯©` ¹°«°£¦‰ l/ ¾‰Ÿ„§ ´ «(¡ ´ ©Ž¬ “ Î(£D l¦‰ž®©` (¥=ª„¤ªN©Ž¬‰£¤¨`¬5ª¯¡†¨£ (ª„¥-©Ž¬5£ t` ug¹5¾‰ŸN§ ´ «(¡ ´ ©Ž¬G¹€ŸN§ ´ ;ª„¤(·¯£¤r§5£žGŸ„­5­C£ŸN¤ ©rª¾C£ Ð ¦‰ŸNž†¡w©rŸ_©r¡w¶¯£ž†³  Ž¡w±s¡†ž†Ÿ„¤)²cŸNžw©Ž¬5ª¯¦5¸„¬ªN¥¨'ª¯¦5¤` l£ ´ H¡ GC£¤Ž¡†§5¸ Ð ¦‰ŸN§~©r¡®©`Ÿ_©Ž¡†¶„£žw³5µ “    ­5¤r£¨'¡‡ Ž£ž†³ÑŸ * ©¡‡ s¡w±s­Cª„¤Ž©rŸN§~©s©Žª¼§5ª„©Ž£©Ž¬EŸ_©©Ž¬‰£ Æ Ê "$Ä #!Ÿ;³„£D T±ŸN¤r¸„¡†§ ¾Eª¯¦5§ ´ ¡†U ¾Cª„¦5§ ´ ²jª„¤)¨'ª„§5¢ ´ £§E¨'£s¡w§~©r£¤r¶;Ÿ„ž‡µª„§Á©Ž¬5£©r¤Ž¦5£s£¤Ž¤rª„¤…¤rŸN©Ž£ªN¥"©r¬5£žw£DŸN¤r§5£ ´ ¨'ž‡Ÿ„ r l¡w¢‰£¤ “ * §‘Ä  l©Ž£DŸ ´ ¹E¡®©¡† …Ÿs©Ž¤r¦5££¤Ž¤rª„¤(¤`Ÿ_©r£¾Eª¯¦5§ ´ ª¯§½ŸN§½Ÿ¯ Ž Žª‘¨'¡‡Ÿ_©Ž£ ´  \©rª2¨`¬EŸ„ l©Ž¡‡¨¨'ž‡Ÿ„ r l¡w¢‰£¤…¨`¬5ª¯ Ž£§ Žª Ÿ¯ z©rª¬‰Ÿ;¶¯£Ÿ l¡†±s¡wž‡ŸN¤…©Ž£D \©£¤r¤rª„¤¤`Ÿ_©r£ “à ¬5£D l£¾Eª¯¦5§ ´  )¨ŸN§½¾C£s¤Ž£¸¯ŸN¤ ´ £ ´ Ÿ„ …¾Eª¯¦5§ ´  …¥=ª„¤ ©r¬5£ª¯¤Ž¡†¸„¡†§‰ŸNžC¨'ž‡Ÿ„ r l¡w¢‰£¤"ª„§‰žw³s¦5§ ´ £¤!ŸN§ Ÿ ´5´ ¡w©Ž¡†ª„§‰Ÿ„žCŸ¯ Ž Ž¦5±s­‘©Ž¡†ª„ nXSl^`bjf;VXf;x ^`nXxgegbjV dZhDm(W@^`YLdc[_^`nXn i nXSl^`bjf ^ÇY\no^`WZWcV p_S\b e`u¿dch;S uwegbcm)Š  WZVXxgf    "!  #g}‰RTSvU;egf~k d&h_^`fDU;nXS-dZhDVXW&uwegbcm¿h;S\bjS'}




    !

 !     !



 



! , [ !

t

 

1 test set errors test set bound margin bound 7/10 margin

error rate

0.8 0.6 0.4 0.2

0 liver bostonpima ion sonar dig1 dig2 adult problem

7!œ &šG —> § Ÿ ´5´ ¡w©Ž¡†ª„§©Žª¨'ª¯±s­‰ŸN¤r¡w§‰¸«(¡w©Ž¬$£¶¯£¤r³~©Ž¬5¡†§5¸s¡w§¢‰¸¯¦5¤Ž£R2 “ .5¹‘«9£)¸„¤`ŸN­‰¬ ©r¬5£±ŸN¤r¸„¡†§  ¾Cª„¦5§ ´ «(¬‰£§%^  /ªN¥/©Ž¬5£ ´ Ÿ_©rŸ¡‡ !¦‰ Ž£ ´ ¥=ª„¤!©r¬5£)©Ž¤`ŸN¡†§ Ž£'© “ " ª„©Ž£)©Ž¬EŸ_©(¡®©z¡†±s­5¤rª_¶„£   Žª„±s£«(¬EŸ_©(ª„§©Ž¬5£±Ÿ„¤Ž¸¯¡w§¾Eª¯¦5§ ´ ¨Ÿ„ž†¨¦5ž‡Ÿ_©Ž£ ´ ¦‰ Ž¡†§5¸s©Ž¬5£ IE
©r¤rŸ„¡w§ Ž£'©² I ’ E±ŸN¤r¸„¡†§ ¾Eª¯¦5§ ´ µ'¹5¾5¦‘©z§‰ªN©z£§‰ª„¦5¸¯¬©Žª¨ª„±s­C£'©Ž£«(¡w©Ž¬©r¬5£)©Ž£D \© Ž£'©(¾Cª„¦5§ ´ “ 


 

*

¦5§‰§5£¨£ r ŽŸ„¤Ž³©rª£¶¯£§s¬‰Ÿ;¶¯£zŸ)©Ž£D \©9 Ž£'© “ * §s¢‰¸„¦5¤r£:2 “ È)«°£¨'ª¯±­EŸN¤r£9©r¬5£z©r¤Ž¦‰£(£¤r¤Žª¯¤¾Eª¯¦5§ ´   ª¯§¿ŸNž†ž!ªN¥z©r¬5£ ´ Ÿ_©rŸ½©ŽªÀ©Ž¬5£©Ž¤r¦5££¤r¤Žª¯¤T¾Cª„¦5§ ´  T¸¯£§5£¤rŸN©Ž£ ´ ¥=¤Žª¯± ©Ž¬5£/I E _È[E©Ž¤`ŸN¡†§ D©r£ l©  Ž­5ž†¡®© “ à ¬5£9¤Ž£D l¦5žw©r / Ž¬5ª_« ©Ž¬‰ŸN©&©Ž¬‰£ Æ Ê "$Ä #°Ÿ;³¯£ ?±Ÿ„¤Ž¸¯¡w§¾Cª„¦5§ ´ ¡‡ G©r¡w¸¯¬¯©/£§5ª¯¦5¸„¬)©rª¸¯¡w¶¯£v¦‰ Ž£'¥=¦5ž ¡†§‘¥=ª„¤r±Ÿ_©r¡wª¯§G¹?¾5¦‘© l©Ž¡†žwž§5ª„©¨'ª„±s­C£'©r¡®©r¡w¶¯£«(¡w©Ž¬À©Ž¬‰£©r£ l© l£©)¾Cª„¦‰§ ´   “à ¬5¡‡ )¡† )¡†§¼ \©r¤Žª¯§5¸ ¨ª„§~©Ž¤`Ÿ„ l©…«(¡w©Ž¬¼Ÿ©r¤rŸ ´ ¡w©Ž¡†ª„§Áª„¥ Ð ¦‰Ÿ„§¯©r¡®©`Ÿ_©r¡w¶¯£ž†³¡w±s­5¤`Ÿ„¨'©Ž¡‡¨ŸNž-±ŸN¤r¸„¡†§½¾Cª„¦5§ ´   “à ¬5£¤r£sŸN¤r£  Ž£¶¯£¤`ŸNž2¦‰ Ž£ Ÿ;¶_ŸN¡†ž†Ÿ„¾5ž†£9¥=ª¯¤¾Eª¯¦5§ ´  v«(¬5¡†¨`¬s­5¤rª_¶2¡ ´ £! Žª„±s£!¡†§‘¥=ª„¤r±Ÿ_©Ž¡†ª„§¾5¦‘©9«(¬5¡†¨`¬Ÿ„¤Ž£!§5ª„© ¥=¦5ž†ž†³©Ž¡†¸„¬~© “ ² ’ µ à ¬5£³±s¡w¸¯¬¯©(¾C£¨'ª¯±T¾5¡†§5£ ´ «(¡w©Ž¬Ÿ©Ž¤`ŸN¡†§ D©r£ l©°¾Cª„¦‰§ ´ ? ’ 0 B “ ² .„µ à ¬5£©Ž¤`ŸN¡†§Á l£©…¾Cª„¦5§ ´ ±s¡w¸¯¬~©…£Ÿ¯ l¡†ž†³¾C£¨ª„±s£©r¡w¸¯¬~©Ž£¤…¥=ª„¤) Ž±sŸ„žwž†£¤ ŽŸ„±s­5žw£ l¡†Ï£D  “ à ¬5¡‡ («°Ÿ¯ °ª„¾‰ Ž£¤r¶„£ ´ ¡†§ ? ’ 0B “ ²cȯµ à ¬5£À©Ž¤`ŸN¡†§Ò Ž£'©¾Cª„¦‰§ ´ ±s¡†¸„¬~© l©Ž¡†ž†ž¬EŸ;¶„£½©r¬5£ ¤Ž¡†¸„¬~s © tŽ Ž¬‰ŸN­C£ u½¥=ª¯¤¨`¬5ª2ª~ l¡†§5¸{Ÿ„§ ª„­‘©r¡w±Ÿ„žG­‰ŸN¤`ŸN±s£©Ž£¤( Ž£'©Ž©Ž¡†§5¸‰¹5 Ž¦‰¨`¬$Ÿ„  t` u¡†§$Ÿ %2¦‰­5­Eª¯¤l'© (£D¨g©rª„¤ Å Ÿ¯¨`¬5¡w§‰£ “ 4 “ 9 < ˜ 85—> œ=˜%• ˜%šG•G™ à ¬5£ ŽŸ„±­‰žw£¨ª„±s­5¤r£ r l¡†ª„§$¾Cª„¦5§ ´ ? ’ 2BZT¹ ?F2B/¡‡ zž†¡†·„£©r¬5£ Æ Ê "$Ä #°Ÿ;³¯£ !¾Cª„¦5§ ´ ¡w§$©r¬‰Ÿ_©¡®© Ÿ„­5­5ž†¡w£D (©ŽªŸN¤r¾5¡w©Ž¤`ŸN¤r³ ­5¤r£¨¡† Ž¡wª¯§½¨'ª¯§¯©r¡w§2¦5ª¯¦‰ z¶_Ÿ„žw¦5£ ´ ¨'ž‡Ÿ„ r Ž¡®¢‰£¤r  “ M §‰žw¡†·„£T©Ž¬5£ Æ Ê "$Ä #!Ÿ;³„£D  ¾Cª„¦5§ ´ ¹v¡w©Ÿ„­5­5ž†¡w£D ±£DŸN§5¡†§5¸„¥=¦5žwž†³¼©Žª §5ª¯§‰ \©rª‘¨`¬‰Ÿ„ l©Ž¡‡¨ ¨ž†Ÿ¯ Ž Ž¡®¢E£¤`  “ËÅ Ÿ„¡w§‰ l©Ž¤r£Ÿ„± žw£DŸN¤r§5¡†§5¸ Ÿ„žw¸¯ª„¤r¡®©r¬5±  ´ ªz§5ª„©&ª„­‘©r¡w±s¡†Ï£©r¬5£" rŸN±s­5ž†£"¨'ª¯±s­5¤Ž£D Ž Ž¡†ª„§±£©Ž¤r¡†¨„¹_ lª!©r¬5£"¾Cª„¦5§ ´ ŸN­5­5ž†¡‡¨Ÿ_©r¡wª¯§ ¡‡ G Žª„±s£«(¬‰ŸN©G¤rŸ„¤Ž£¤ “ " ª„§5£©Ž¬5£žw£D Ž ¹©Ž¬5£¤Ž£ ´ ª(£»‘¡† l©% Žª„±s£¤Ž£DŸ„ Žª„§‰Ÿ„¾5ž†³z¨'ª„±s­C£'©r¡®©r¡w¶¯£vžw£DŸN¤r§5¡†§5¸ Ÿ„žw¸¯ª„¤r¡®©r¬5± "¥=ª¯¤(«(¬5¡‡¨`¬ ©r¬5£ ŽŸ„±s­5žw£¨ª„±s­5¤r£ r l¡†ª„§¾Eª¯¦5§ ´ ­5¤rª ´ ¦‰¨'£D ( l¡†¸„§5¡w¢E¨ŸN§~©(¤Ž£D l¦5žw©r  “ à ¬5£ Ž£¨'©Ž¡†ª„§¡† !ª¯¤Ž¸~ŸN§5¡†Ï£ ´ Ÿ„ °¥=ª¯žwž†ª_«z ,+ ²’l µ %2¦5¾‰ Ž£¨'©Ž¡†ª„5 § 4 “†’  \©`Ÿ_©Ž£D zŸN§ ´ ­5¤rª_¶„£D 9©Ž¬5£ rŸN±s­5ž†£¨'ª„±s­5¤r£ r Ž¡wª¯§ ¾Cª„¦5§ ´ “



 




     !

Sample Compression Bound

δ Verifier

Learner Learning Algorithm A

Draw Training Examples

m examples Choose Subset S’, c=A(S’)

Subset S’ Evaluate Bound For c=A(S’)

7!œ &šG — >
à ¬5£½¡†§~©Ž£¤`Ÿ„¨'©Ž¡†¶„£$­5¤rª2ªN¥)ªN¥žw£DŸN¤r§5¡†§5¸¼¥=ª„¤©r¬5£Á rŸN±s­5ž†£ '¨ ª¯±­‰¤Ž£D Ž Ž¡wª¯§¾Cª„¦‰§ ´ “ " ªN©r£)©Ž¬‰ŸN©z©Ž¬5£Tžw£DŸN¤r§5¡w§‰¸sŸ„žw¸¯ª„¤r¡®©r¬5± ¡‡ zŸN¤r¾5¡w©Ž¤`ŸN¤r³ ¬5£¤r£„¹E l¡†±s¡wž‡ŸN¤°©Žªs©r¬5£)©Ž£D \© Ž£'©(¾Cª„¦5§ ´ “ 


 

² .„µl%2¦5¾‰ Ž£¨'©Ž¡†ª„§l4 “ . Ž¬5ª_«z ©r¬‰Ÿ_©°©Ž¬5£ rŸN±s­5ž†£…¨ª„±s­5¤r£ r l¡†ª„§¾Cª„¦5§ ´ ¡‡ 9§‰£ŸN¤rž†³Ÿ¯ v©r¡w¸¯¬¯© Ÿ„ !­Cª¯ r l¡†¾5ž†£)¸„¡†¶„£§©Ž¬‰£ª„¾‰ Ž£¤r¶_Ÿ_©Ž¡†ª„§E  “ ²cȯl µ %2¦5¾‰ Ž£¨'©Ž¡†ª„§ 4 “ È ´ ¡‡ Ž¨¦‰ r l£D T¤r£ Ž¦5žw©r T¥=¤Žª¯± ©r¬5£$ŸN­5­5ž†¡‡¨Ÿ_©r¡wª¯§ÑªN¥©Ž¬5£$ ŽŸ„±s­5žw£¨'ª¯±Ä ­5¤r£ r l¡†ª„§¾Eª¯¦5§ ´ ©Žª Ž¦5­5­Cª„¤Ž©(¶„£¨'©Žª¯¤(±sŸ¯¨`¬5¡†§5£  “ 4 “w’¯“ W # L !w"XW  2%N, W M,MN2U1J 2STUN/ Zà ¬‰£ %‘Ÿ„±s­5žw£z9ª„±s­5¤r£ r Ž¡wª¯§¾Eª¯¦5§ ´  \©`Ÿ_©Ž£ ´ ¬5£¤Ž£¡‡  ´ ¡ G€£¤` T Žª„±s£«(¬‰Ÿ_©¥=¤rª„± ªN©r¬5£¤T¤Ž£D l¦5žw©r v² ? ’ 2B3?F2CB(ŸN§ ´ ª„©Ž¬5£¤r `µ¾2³ ¸¯£§5£¤rŸ„žw¡†ÏŸN©Ž¡†ª„§ Ÿ„§ ´  Ž¡†±­‰žw¡w¢E¨ŸN©Ž¡†ª„§¾5¦‘©(©r¬5£¾Eª¯¦5§ ´ ¾C£¬‰Ÿ;¶2¡†ª„¤!¡†  Ð ¦EŸNž†¡®©`Ÿ_©Ž¡†¶„£žw³ ¡ ´ £§~©Ž¡‡¨ŸNž “ %2¦5­‰­Eª~ l£s«°£¬‰Ÿ;¶¯£Ÿž†£ŸN¤r§5¡†§5¸$ŸNž†¸„ª„¤r¡w©Ž¬5±$¹ m «(¬5ª¯ Ž£©Ž¤`ŸN¡†§5¡w§‰¸¡‡)  tl Ž­‰ŸN¤` Ž£ u ¡w§À©Ž¬5£  Ž£§‰ Ž£9©Ž¬‰ŸN©©Ž¬5£(ª¯¦‘©Ž­5¦5©"¨'ž‡Ÿ„ r l¡w¢‰£¤/¡‡  ´ £­C£§ ´ £§~©v¦5F ­Cª„I §ª„§5ž†³TŸ) Ž¦5¾‰ Ž£'©ªN¥C©Ž¬5£ ´ ŸN©rŸ‰¹ m ` F ¤rŸNI ©Ž£„¹ m  ¥=ª¯¤ m  m “…à ¬5£ ŽŸ„±­‰žw£¨'ª„±s­5¤r£ r Ž¡wª¯§¾Cª„¦‰§ ´ ¡‡  ´ £­E£§ ´ £§¯©ª„§©r¬5£T£¤Ž¤rª„¤ F I © m d m  “à ¬5£±sª„©Ž¡†¶;ŸN©Ž¡†ª„§ ¬5£¤r£¡† …©Ž¬EŸ_©)©Ž¬5£s£»‘Ÿ„±s­5žw£D «(¬5¡†¨`¬Á©Ž¬5£ q ` b b  ª„§Á©Ž¬5£ Ž¦5¾‰ Ž£'R ž†£Ÿ„ ¤Ž§5¡†§5¸sŸNž†¸„ª¯¤Ž¡w©Ž¬5± ´ ª2£   ´ £­C£§ ´ ¦‰­Eª¯§½ŸN¤r)£ tŽŸNž†±sª¯ lv© u…¡†§ ´ £­E£§ ´ £§¯©ŸN§ ´  Žª«°£¨Ÿ„§ tŽŸNž†±sª¯ lv© u…¸¯£'©zŸ©Ž£ l© Ž£'©z¾Cª„¦5§ ´ “ (¡†£«°£ ´ Ÿ„ Ÿ„§T¡†§~©Ž£¤`Ÿ„¨'©Ž¡†¶„£°­5¤rª~ª„¥Eª„¥Ež†£ŸN¤r§5¡†§5¸s²=¡†§T¢‰¸¯¦5¤rK £ 4 “w’ µg¹_©r¬5£z ŽŸ„±s­5žw£!¨'ª¯±s­5¤Ž£D Ž Ž¡†ª„§ ¾Cª„¦5§ ´ ¡‡ ¦5§5¡ Ð ¦5£zŸN±sª¯§5¸¯ l©%©r¤rŸ„¡w§‰¡w§5¸ Ž£'©¾Cª„¦5§ ´  ¾E£D¨Ÿ„¦‰ l£!¡w© ´ ª2£ -§5ª„©v¤Ž£ Ð ¦5¡w¤r£ ^  ¡†§5¡w©Ž¡‡ŸNž ¨ª„±s±s¡®©r±£§~©°©rªŸ±£DŸ„ Ž¦5¤r£ª_¶¯£¤9©Ž¬5£¨ž†Ÿ¯ Ž Ž¡w¢‰£¤`  “ 











W02

, 0W 

b E; 7