TUTORIAL ON PRACTICAL PREDICTION THEORY FOR ...


      !  " $#%#&'( ) *+-,/.021/.-3/45+-6&798;:=@6%ACBDAE1/6GFG, H9I;JLKNMNONP;K2Q!RTS!UDVXWZY\[;WZW/]_^`WZVXY9a;bcS\U;VXYLdcVXegfdch;S\egbjiT^`fDUV dlk W-VXm(a_^`YLd-egfTY\no^`WZWcV p_Yl^rq dcVXegfsWZ[DYlY\S\WZW9SLtg^`nX[_^rdcVXegf 8 VXm(a;nXVXYl^rdZVXegfDWvuwegbvnXSl^`bcf;VXfDx^`nXxgegbcV dch;myU;S\WcVXxgf 8 ^`f;U[;WcS\W VXfznXSl^`bcfDVXf;x!^`nXxgegbjV dZhDm{SL|SlY\[DdcVXegf~}C&hDVXW€dZ[DdcegbjVo^`n~VXW€m(Sl^`f'd?dZe"]NS^"Y\egm(a;bjS\h;S\f;WZV t'S Y\egm(a;VXno^rdcVXegfe`ubcS\Wc[;n dZWz‚&h;VXYƒh ^`bjS]„e`dchdZh;S\egbjSLdZVXYl^`nXn i bcVXxgegbjeg[;W…^`f;U a;bZ^`YLdcVXYl^`nXn i [;WcSLu†[;n‡} Gh;S\bjS9^`bjS"^"dˆ‚?e(VXm(aNegb=dƒ^`f'd&VXm(a;nXVXYl^rdZVXegfDW?e`u‰dch;SbjS\WZ[;n dcW%aDbcS\WZS\f'dZS\Uh;S\bjSgŠ ‹jŒŽ F egm(m(egfaDbZ^`YLdcVXYlS\W/uwegb/bjS\aNegbjdcVXf;xbcS\WZ[Dn dZW-VXfYlno^`WcWZV p_Yl^rdcVXegfWZh;eg[DnXU)Yƒh_^`f;xgS dce([;WZS-dch;S-dcS\Wcd%WZSLd&]„eg[Df;U~} ‹†' ‘bc^`VXfWZSLd-]Neg[;f;UDW-Yl^`fTWcegm(SLdZVXm(S\WG]NS9[;WcS\UdcezU;VXbjSlYLdcn im(e`dcV tg^rdcS9nXSl^`bjf;VXf;x ^`nXxgegbjV dZhDm(WŽ}

„’ “z”`•?–?—€˜&™%š&›E–€œ=˜%• 9ž‡Ÿ„ r l¡w¢‰£¤` "Ÿ„¤Ž£¥=¦5§‰¨'©Ž¡†ª„§‰ 9«(¬5¡‡¨`¬ ­‰Ÿ„¤l©r¡®©r¡wª¯§ ŸT Ž£'©°¡w§~©Žª©\«°ª£ž†£±s£§~©` ²=©Ž¬5£) Ž£'©9ª„¥&¤`ŸN¡†§~³ ´ Ÿ;³‘ 9ŸN§ ´ ©Ž¬5£) l£©°ª„¥% Ž¦5§5§2³ ´ Ÿ;³‘ rµ “ 9ž‡Ÿ„ r Ž¡®¢‰£¤r °ŸN¤r£(©r¬5£)±ª~ \©! l¡†±s­5ž†£…§5ª¯§~©Ž¤r¡w¶2¡‡ŸNž ´ £¨¡† Ž¡wª¯§ ±ŸN·2¡†§5¸)£ž†£±s£§~©" Žª \©r¦ ´ ³2¡†§5¸©Ž¬5£(©r¬5£ª¯¤Ž³TªN¥€žw£DŸN¤r§5¡w§‰¸)¨ž†Ÿ¯ Ž Ž¡w¢‰£¤` -¡† v¶„£¤r³¥=¦5§ ´ ŸN±s£§~©`ŸNž‘©rª  l©Ž¦ ´ ³~¡†§5¸©r¬5£…©Ž¬5£ª„¤r³ª„¥Gžw£DŸN¤r§5¡w§‰¸‰¹„¡†§ ¸¯£§5£¤rŸ„ž “ 9ž†Ÿ¯ Ž Ž¡w¢‰£¤` "Ÿ„¤Ž£ Ž¦‘º¨'¡†£§~©ržw³¨'ª¯±s­5žw£»©r¬‰Ÿ_© ±ŸN§2³¼­5¬5£§5ª„±s£§‰Ÿ½ª¯¾‰ l£¤Ž¶¯£ ´ ¡†§¿±Ÿ„¨`¬5¡†§5£žw£DŸN¤r§5¡w§‰¸{²ˆ©r¬5£ª¯¤Ž£©Ž¡‡¨Ÿ„žwž†³Àª¯¤£»‘­E£¤Ž¡†±s£§~©rŸ„žwž†³5µ ¨ŸN§Á¾C£ª„¾‰ Ž£¤r¶„£ ´ ¡w§À©Ž¬‰£¨ž†Ÿ¯ Ž Ž¡®¢C¨Ÿ_©r¡wª¯§Á Ž£'©l©r¡w§‰¸ “ £©¹&¨ž†Ÿ¯ Ž Ž¡®¢E£¤` …Ÿ„¤Ž£ l¡†±s­5žw£s£§5ª„¦5¸¯¬½©rª ±ŸN·¯£©Ž¬5£¡w¤ŸN§‰Ÿ„žw³‘ Ž¡† )£DŸ„ Ž³½©Žª¦‰§ ´ £¤` l©rŸN§ ´ “à ¬‰¡† ¨ª„±¾5¡w§EŸ_©Ž¡†ª„§¼ª„¥! Ž¦‘º¨'¡†£§~©T³¯£'©±s¡w§‰¡®Ä ±ŸNž¨ª„±s­5ž†£'»‘¡®©\³$¥=ª¯¤¨ŸN­5©Ž¦5¤r¡w§‰¸ ­5¬5£§5ª„±s£§‰Ÿ ±Ÿ„·„£ ©Ž¬‰£s l©Ž¦ ´ ³½ªN¥°¨'ž‡Ÿ„ r l¡w¢‰£¤` …£ Ž­E£D¨'¡‡ŸNž†žw³ ¥=¤r¦5¡w©l¥=¦5ž “ à ¬5£¸„ª~ŸNž&ª„¥v©Ž¬‰¡† ­‰Ÿ„­E£¤…¡‡ )ŸN§½¡†§~©Ž¤rª ´ ¦‰¨g©r¡wª¯§©Žª©Ž¬5£©Ž¬‰£ª„¤r³$ªN¥v­5¤r£ ´ ¡†¨'©Ž¡†ª„§½¥=ª¯¤¨ž†Ÿ¯ Ž Ž¡wÄ ¢E¨Ÿ_©r¡wª¯§ “zÅ ŸN§2³ªN¥©r¬5£ Ž£¤r£ Ž¦5ž®©` z¬‰Ÿ;¶¯£¾C££§Á­5¤Ž£D l£§~©Ž£ ´ £ž† Ž£«(¬‰£¤r£„¹EŸ„ž®©r¬5ª„¦‰¸„¬$©Ž¬‰£ l©\³~ž†£„¹ £žw£±£§~©rŸNž‘§EŸ_©Ž¦‰¤Ž£¯¹„ŸN§ ´ ¸¯£§5£¤rŸ„žw¡w©\³ªN¥C©Ž¬5£­5¤Ž£D l£§~©rŸ_©r¡wª¯§T±Ÿ;³¾C£z§5£« “/à ¬5¡‡ ¡‡ vŸ…©Ž¦‘©rª„¤r¡†Ÿ„žc¹  Žª)«9£!ž†¡w±s¡w©"ª„¦5¤­‰¤Ž£D l£§¯©`Ÿ_©r¡wª¯§T©rª…©r¬5ª¯ Ž£(¤r£ Ž¦5ž®©` -«(¬5¡‡¨`¬sŸ„¤Ž£!¾CªN©r¬©r¬5£ª¯¤Ž£©Ž¡‡¨Ÿ„žwž†³T Žª„¦5§ ´ Ÿ„§ ´ ­5¤`Ÿ„¨'©Ž¡‡¨Ÿ„žwž†³¦‰ Ž£'¥=¦5ž “ à ¬5£¤Ž£zŸ„¤Ž£z Ž£¶¯£¤`ŸNž‘¡†±­Cª„¤Ž©rŸ„§~©9Ÿ„ Ž­E£D¨g©` ªN¥?ž†£ŸN¤r§5¡†§5¸)«(¬5¡‡¨`¬s©Ž¬5£(©r¬5£ª¯¤Ž³¬5£¤r£¨Ÿ„ l©r ž†¡w¸¯¬~© ª¯§ “GÆ £¤r¬‰ŸN­E ?©Ž¬‰£v±sª¯ l©&¡w±s­Cª„¤Ž©rŸ„§¯©%ªN¥2©Ž¬5£D l£"¡† &©Ž¬5£"­5¤Žª¯¾5ž†£±@ªN¥‘­E£¤l¥=ª¯¤Ž±Ÿ„§‰¨'£¤r£­Cª„¤Ž©Ž¡†§5¸!¥=ª„¤ ¨ž†Ÿ¯ Ž Ž¡®¢E£¤`  “/Å ŸN§2³T­C£ª¯­5ž†£¦‰ Ž£ Žª„±s£(¥=ª„¤r±ÇªN¥?£±s­5¡†¤Ž¡‡¨Ÿ„ž‰¶_ŸN¤r¡‡ŸN§‰¨£!©ŽªT£D \©r¡w±ŸN©Ž£z¦5­‰­E£¤°Ÿ„§ ´ ž†ª_«9£¤(¾Eª¯¦5§ ´   “(à ¬5¡‡ z¡‡ ŸN§£¤Ž¤rª„¤ŽÄƒ­5¤Žª¯§5£­5¤rŸ¯¨g©r¡†¨£„¹EŸ„§ ´ ©Ž¬‰£©Ž£D \©… Ž£'©¾Eª¯¦5§ ´ ¡†§½ l£D¨g©r¡wª¯§½È ¡†±s­5žw¡†£ 9Ÿ)¾E£©l©r£¤°±£©Ž¬5ª ´ ¾2³§5£Ÿ„¤Žž†³Ÿ„§~³±s£'©Ž¤r¡‡¨ “-É ª„­C£'¥=¦‰žwž†³„¹~©Ž¬‰¡† "«(¡wž†ž‰¾C£¨ª„±s£¨'ª¯±s±ª¯§ ­5¤`Ÿ„¨'©Ž¡‡¨'£ “ Ê ¥ˆ©r£¤ ´ ¡‡ Ž¨¦‰ Ž Ž¡†§5¸Ë©Ž¬‰£©Ž£D \©$ l£©¾Cª„¦5§ ´ «°£À¨ª_¶„£¤s©Ž¬5£Ę̈¨Ÿ„±$Í  ΏŸNϪ¯¤¾Cª„¦5§ ´ ¹(©Ž¬5£  Ž¡w±s­5ž†£ l©(©Ž¤`ŸN¡†§ l£©¾Cª„¦5§ ´ ¹‰«(¬5¡‡¨`¬$£'»‘­5ž‡ŸN¡†§‰ ²jŸ„§ ´Ð ¦‰ŸN§~©r¡®¢‰£D rµ9©r¬5£T¨ª„±s±sª„§$­5¬5£§5ª„±s£§EŸ ª„¥Tª_¶„£¤l¢5©Ž©Ž¡†§5¸Ñ¡†§Ò Ž¡w±s­5ž†£ l©$¥=ª„¤r± “ÇÓ £ËŸNž‡ lª{­5¤rª_¶„£©r¬‰Ÿ_©©Ž¬5£¿Ì¨¨ŸN±$Í  ΏŸNϪ¯¤ ¾Cª„¦5§ ´ ¨ŸN§5§5ª„©°¾C£)¡w±s­5¤rª_¶„£ ´ «(¡w©Ž¬5ª¯¦‘©(¡w§E¨'ª„¤r­Cª„¤`Ÿ_©Ž¡†§5¸£'»2©Ž¤`ŸT¡†§‘¥=ª¯¤Ž±Ÿ_©r¡wª¯§Ÿ„§ ´ Ÿ„­5­5ž†³©Ž¬5£)¾Cª„¦5§ ´ ©rª ´ £¨'¡‡ Ž¡wª¯§ ©r¤Ž££  “ Ô


     ! " £»2©¹&«°£ ´ ¡‡ Ž¨¦‰ r ©\«9ª©Ž¤`ŸN¡†§  Ž£'©¾Cª„¦‰§ ´  ¹G©Ž¬‰£ Æ Ê "Ä$#!Ÿ;³„£D ¾Cª„¦5§ ´ Ÿ„§ ´ ©Ž¬5£&%5ŸN±s­5ž†£ 9 ª¯±­‰¤Ž£D Ž Ž¡wª¯§½¾Cª„¦5§ ´ ¹&«(¬5¡‡¨`¬À¬EŸ;¶„£T­‰¤Žª_¶¯£ ´ ©rª$¸„¡†¶„£­5¤rŸ¯¨g©r¡†¨ŸNž-¤Ž£D l¦‰ž®©` ¥=ª¯¤±sª„¤r£¸¯£§5£¤rŸ„ž ¨ž†Ÿ¯ Ž Ž¡®¢E£¤` ¹‘ Ž¦‰¨`¬Ÿ¯ '%2¦5­5­Cª„¤Ž©)(£¨'©Žª„¤ Å Ÿ„¨`¬5¡†§5£ (Ÿ„§ ´ " £¦5¤rŸ„ž " £©\«9ª¯¤Ž·‘  “ Ê ž†žªN¥©r¬5£s¤Ž£D l¦‰ž®©` ¬5£¤Ž£ l¬‰ª„¦5ž ´ ¾C££DŸ„ Ž¡wž†³ŸN­5­‰¤Žª~Ÿ„¨`¬‰Ÿ„¾5žw£ŸN§ ´ ¦5§ ´ £¤r l©rŸ„§ ´ ŸN¾‰žw£ “à ¬5£ ­5¤rª2ªN¥j °ŸN¤r£ Ž¡†±­‰žw£¯¹5ŸN§ ´ £'»5ŸN±s­5ž†£ !ŸN¤r£¸„¡†¶„£§ “/Æ ª¯¡w§~©Ž£¤r "©Žªs¤r£ž‡Ÿ_©r£ ´ «°ª„¤r·ŸN¤r£Ÿ„ž† ŽªT¸¯¡w¶¯£§ “ * ©¡‡ ¡w±s­Cª„¤Ž©rŸN§~©©rª½§5ª„©Ž£©r¬‰Ÿ_©sŸNž†ž"ª„¥(©Ž¬5£¤Ž£D l¦5žw©r T­5¤r£ Ž£§~©Ž£ ´ ¬5£¤r£¥jŸNž†ž"¡†§Ë©r¬5£ ¤r£Ÿ„žw± ª„¥-¨ž†Ÿ¯ Ž Ž¡†¨ŸNžG l©rŸN©Ž¡‡ \©r¡†¨  “ * §­‰ŸN¤Ž©Ž¡‡¨'¦5ž‡ŸN¤D¹5ŸNž†žG¤`ŸN§ ´ ª„±s¡wÏDŸ_©r¡wª¯§‰ (ŸN¤r£)ª_¶„£¤ ´ ¤rŸ;«z !ª„¥%©r¬5£ ´ ŸN©rŸ5¹ Ÿ„§ ´ ª¯¦5¤(¤r£ Ž¦5ž®©` !¬‰Ÿ;¶„£…©Ž¬‰£)¥=ª„¤r± ªN¥¨'ª¯§‘¢ ´ £§‰¨'£)¡†§¯©r£¤r¶_ŸNž‡  “ à ¬5£ž‡Ÿ;³„ª¯¦‘©°ª„¥%©r¬5¡‡  ´ ª‘¨'¦5±s£§¯©(¡‡ zŸ„ °¥=ª¯žwž†ª_«z ,+ § .­5¤r£ Ž£§~©r °©r¬5£)¥=ª„¤r±ŸNž€±sª ´ £ž - %2£¨'©Ž¡†ª„/ - %2£¨'©Ž¡†ª„§È­5¤r£ Ž£§~©r °©r¬5£)©Ž£D \©z Ž£'©z¾Cª„¦5§ ´ § 0s­5¤r£ Ž£§~©r °©r¬5£Ì¨¨ŸN±$Í  !ÎzŸ„Ϫ¯¤°¾Cª„¦‰§ ´ - %2£¨'©Ž¡†ª„1 § 2­5¤r£ Ž£§~©r °©r¬5£ Æ Ê "3Ä #!Ÿ;³„£D "¾Cª„¦‰§ ´ - %2£¨'©Ž¡†ª„/ § 4­5¤r£ Ž£§~©r °©r¬56 £ %‘ŸN±s­5ž†£T9ª„±s­5¤r£ r l¡†ª„§¾Eª¯¦5§ ´ - %2£¨'©Ž¡†ª„5 ¬Ã 5£¥=ª¯¤Ž±ŸNž-±sª ´ £ž9ŸN§ ´ ©Ž£D \© Ž£'©¾Eª¯¦5§ ´ ±T¦E \©¾C£¦5§ ´ £¤r l©Žª2ª ´ ¡†§¼ª„¤ ´ £¤©ŽªŸ„­5­5¤r£¨'¡‡Ÿ_©r£ Ÿ„žwž-ž‡Ÿ_©Ž£¤)¤Ž£D l¦5žw©r  “Tà ¬5£¤Ž£¡‡ )§5ª­EŸN¤Ž©Ž¡‡¨'¦5ž‡ŸN¤ ´ £­C£§ ´ £§‰¨³½¾C£'©\«°££§À©Ž¬‰£¶_Ÿ„¤Ž¡†ª„¦‰ ©Ž¤`ŸN¡†§À Ž£'© ¾Cª„¦5§ ´  !«9£)­5¤r£ Ž£§~© “

“ °˜&—8:9@?BA C8DE 7GF FHIKJ I

à ¬5£…©Ž¤r¦5££¤r¤Žª¯¤"¡‡ ° Žª„±s£©Ž¡†±£D 9¨ŸNž†žw£ ´ ©r¬5£ tl¸„£§5£¤`ŸNž†¡†ÏŸ_©r¡wª¯§£¤r¤Žª¯¤vu “NM §‘¥=ª„¤Ž©Ž¦‰§‰Ÿ_©r£ž†³„¹~©Ž¬5£ ©r¤Ž¦5££¤Ž¤rª„¤(¡‡ z§‰ªN©…Ÿ„§ª¯¾‰ Ž£¤r¶;Ÿ„¾5ž†£ Ð ¦‰Ÿ„§~©Ž¡w©\³¡†§ª„¦5¤±ª ´ £ž&¾C£¨ŸN¦‰ Ž£)©Ž¬5£ ´ ¡† l©Ž¤r¡†¾5¦‘©Ž¡†ª„§&¹ i ¹ ¡‡ !¦5§5·2§5ª_«(§ “É ª_«9£¶„£¤D¹~©Ž¬5£¤Ž£)¡‡ zŸ¤Ž£ž†ŸN©Ž£ ´Ð ¦‰Ÿ„§¯©r¡®©\³ «(¬5¡†¨`¬¡‡ (ª„¾‰ Ž£¤r¶_ŸN¾5ž†£ “ £ TWVSXZY[\Y](^B _TW[\[ [g¹aq8` b  W-1TUTN VN2U3$ZPOP?BA CWDdE b F FHZIeJ

I

` n

e

hi

g

jNkNl

i

`:L

F q FH IeJ

i

I

«(¬5£¤Ž£ ¡‡ -Ÿ(¥=¦5§E¨g©Ž¡†ª„§«(¬5¡‡¨`¬±ŸN­‰ t\©Ž¤r¦5£uv©rª e ŸN§ ´ tL¥jŸ„ž† Ž£u©rª ) “ Ê ž‡ Žª‰¹ ;=< >@?A CWDE b ¡†  Ÿ!­5¤rª„¾EŸNl ¾5FmI ¡†žw¡w©\³z©`ŸN·¯£§«(¡w©Ž¬¤r£ Ž­E£D¨g©G©rª(©Ž¬5£"¦5§5¡w¥=ª„¤r± ´ ¡‡ l©Ž¤r¡w¾5¦5©Ž¡†ª„§ª_¶„£¤?©Ž¬‰£" l£©&ªN¥‘£»‘Ÿ„±sFonP­5n@npžwI £D ¹ m “ à ¬5££±­‰¡w¤r¡†¨ŸNžG£¤Ž¤rª„¤!¡‡ z Žª„±s£'©r¡w±s£ ¨ŸNž†ž†£ ´ ©Ž¬5£:t\©Ž¤`ŸN¡†§5¡†§5¸s£¤r¤Žª¯¤vug¹tL©r£ l©z£¤r¤Žª¯¤vug¹2ª„¤ tlª„¾‘Ä  Ž£¤r¶„£ ´ £¤r¤Žª¯v¤ u ´ £­C£§ ´ ¡w§‰¸ ª„§Á«(¬5£'©r¬5£¤¡®©)¡‡ z©r¬5££¤Ž¤rª„¤…¤rŸN©Ž£ª¯§ÁŸ©r¤rŸ„¡w§‰¡w§5¸ l£©¹C©Ž£D \©) l£©¹ ª¯¤!Ÿ±sª„¤r£)¸„£§5£¤`ŸNž? Ž£'© “   L!#"PWZ ²c¨'ª¯§¯©r¡w§2¦5£ ´ µ à ¬5£z¨ž†Ÿ¯ Ž Ž¡®¢E£¤ q «(¬5¡†¨`¬Ÿ„žw«!Ÿ;³‘ %­‰¤Ž£ ´ ¡†¨'©r f> – >‘–=>‘– &˜%™ à ¬5£ Ž¡w±s­5ž†£ l©z¾Cª„¦5§ ´ Ÿ„¤Ž¡‡ Ž£ 9¥=ª„¤(©r¬5£¨'ž‡Ÿ„ r Ž¡†¨ŸNž€©Ž£D¨`¬5§5¡ Ð ¦5£ªN¥-¦‰ Ž¡w§5¸ n ¥=¤r£ Ž¬$£'»5ŸN±s­5ž†£  ©rª£¶_ŸNž†¦‰Ÿ_©r£)Ÿs¨ž†Ÿ¯ Ž Ž¡®¢E£¤ “-à ¬‰¡† ( Ž£¨'©Ž¡†ª„§¡† (ª¯¤Ž¸~ŸN§5¡†Ï£ ´ ¡†§~©Žª©\«9ª Ž¦5¾‰ Ž£¨'©Ž¡†ª„§‰ ,+ - %2¦5¾‰ Ž£¨'©Ž¡†ª„§sÈ “†’ ­‰¤Ž£D l£§¯©` G©r¬5£!¾‰Ÿ„ Ž¡‡¨"¦5­‰­E£¤¾Eª¯¦5§ ´ ª„§©r¬5£°©Ž¤r¦5£!£¤r¤Žª¯¤%¤`Ÿ_©r£„¹_¬‰Ÿ„§ ´ ³ ŸN­5­‰¤Žª;»‘¡†±sŸN©Ž¡†ª„§‰ ¹‘ŸN§ ´ Ÿžwª_«°£¤!¾Cª„¦5§ ´ - %2¦5¾‰ Ž£¨'©Ž¡†ª„§ È “ . ´ ¡‡ Ž¨¦‰ r l£D ©Ž¬5£½¡†±s­5ž†¡†¨Ÿ_©Ž¡†ª„§E ªN¥©Ž¬5£½©Ž£ l© Ž£'©¾Cª„¦5§ ´ ª¯§ £¤r¤Žª¯¤ ¤Ž£­Eª¯¤l©r¡w§‰¸…­5¤`Ÿ„¨'©Ž¡‡¨'£ “ Ê ¾E£©l©r£¤±s£'©r¬5ª ´ ¥=ª¯¤/£¤Ž¤rª„¤/¤r£­Cª„¤Ž©Ž¡†§5¸…¡‡ -ŸN­5­‰žw¡†£ ´ ©Žª) l£¶„£¤rŸ„ž ´ Ÿ_©`Ÿ„ Ž£'©` (ŸN§ ´ ©Ž¬5£¤r£ Ž¦5žw©r (Ÿ„¤Ž£ Ž¬5ª_«(§ “ È “w’¯“ WRJ62 STUw/ Z #°£'¥=ª„¤r£v l©rŸ_©r¡w§‰¸!©Ž¬5£v¾Cª„¦‰§ ´ ¹;«9£"§5ªN©r£"Ÿ°¥=£«¼¾‰Ÿ¯ l¡‡¨ª„¾E l£¤Ž¶_Ÿ_©r¡wª¯§‰ €«(¬5¡‡¨`¬ ±ŸN·¯£z©r¬5£…¤r£ Ž¦5žw©r °ž†£ r 9 Ž¦5¤r­5¤Ž¡‡ Ž¡w§5¸ “/à ¬‰£…­5¤r¡†§‰¨'¡†­‰ŸNž€ª¯¾‰ l£¤Ž¶_Ÿ„¾5žw£ Ð ¦EŸN§~©Ž¡w©\³¡† 9©r¬5£…£±s­5¡w¤r¡‡¨ŸNž £¤Ž¤rª„¤ q8` b ªN¥9Ÿ¨'ž‡Ÿ„ r l¡w¢‰£¤ “Ó ¬‰ŸN©¡‡ …©Ž¬5£ ´ ¡‡ l©Ž¤r¡w¾5¦5©Ž¡†ª„§Áª„¥v©Ž¬‰££±s­5¡w¤r¡‡¨ŸNž-£¤Ž¤rª„¤¥=ª„¤Ÿ¢5»‘£ ´ ¨ž†Ÿ¯ Ž Ž¡®¢E£\¤ ' ‰ª„¤(£DŸ„¨`¬£'»5ŸN±s­5ž†£„¹5ª¯¦5¤!¡w§ ´ £­C£§ ´ £§‰¨£TŸ„ r l¦‰±­5©Ž¡†ª„§¡w±s­5ž†¡w£D !©Ž¬‰£­5¤Žª¯¾‰ŸN¾‰¡wž†¡®©\³

 





 






    !

 !     !



 



! , [ !

 

©r¬‰Ÿ_©©r¬5£ ¨ž†Ÿ¯ Ž Ž¡®¢E£¤£¤r¤` ¡† ¸„¡†¶„£§¼¾2³À©Ž¬5£©r¤Ž¦‰££¤r¤Žª¯¤¹ qW7 “à ¬‰¡† ¨ŸN§¼¾C£ ±sª ´ £ž†£ ´ ¾2³¼Ÿ ¾5¡‡Ÿ„ Ž£ ´ ¨'ª¯¡w §
‰¡†­ +¬5£DŸ ´  !¡w¥/³¯ª„¦$ŸN¤r£)¤Ž¡†¸„¬~©(ŸN§ ´ ©rŸ„¡wž‡ !¡w¥/³¯ª„¦$ŸN¤r£«(¤rª„§‰¸ “ ¸ Á£¤r¤Žª¯¤r ²=¬5£DŸ ´  `µ…ª„¦‘©)ªN¥ n £'»5ŸN±s­5ž†£ s²j¨ª„¡†§ Ó ¬‰ŸN©)¡‡ ©Ž¬‰£­‰¤Žª¯¾‰ŸN¾5¡†ž†¡®©\³$ªN¥°ª„¾‰ Ž£¤r¶2¡w§‰
‰¡†­‰ `µ ' à ¬‰¡† s¡‡ Ÿ¼¶„£¤Ž³¼¥jŸ„±¡†ž†¡†Ÿ„¤ ´ ¡† l©Ž¤r¡†¾5¦‘©Ž¡†ª„§ ¡†§  l©rŸN©Ž¡‡ \©r¡†¨ ¨Ÿ„žwž†£ ´ ©Ž¬5/ £ #°¡†§5ª„±s¡‡ŸNžzŸ„§ ´  ŽªË¡®©$ l¬‰ª„¦5ž ´ ¾E£À¦5§‰ Ž¦5¤r­5¤Ž¡‡ Ž¡w§5¸¼©r¬‰Ÿ_©©Ž¬5£À¾Eª¯¦5§ ´  ­5¤r£ Ž£§~©Ž£ ´ ¬5£¤r£ÁŸN¤r£¥=¦5§ ´ ŸN±s£§~©`ŸNž†žw³ Ÿ #9¡†§5ª¯±¡‡ŸNž “ ´ £­E£§ ´ £§¯©z¦‰­Eª¯§©r¬5£¨'¦‰±T¦5ž‡Ÿ_©r¡w¶¯£ ´ ¡‡ l©Ž¤r¡w¾5¦5©Ž¡†ª„§$ªN¥s  W-1TUTN VN2U O D

'

/

!

 







3

 )

`

F*

 ;=E
0T=9 ©r¤Ž¦5£¼£¤Ž¤rª„¤¤rŸN©Ž£„¹©Ž¬5£ § ^  zà ¬‰¡† "¡† vŸ)¸¯¤rŸ„­5¬ªN¥C©Ž¬5£¨ª„§‘¢ ´ £§‰¨£(¡w§~©r£¤r¶;Ÿ„ž† -¡†±s­5ž†¡w£ ´ ¾~³ ©Ž¬5£9©Ž£D \©- l£©%¾Cª„¦5§ ´ ²jÈ “ È~µ€ª„§©Ž¬5£9žw£¥ˆ©¹„Ÿ„§ ´ ©Ž¬5£!ŸN­‰­5¤Žª;»‘¡†±Ÿ_©Ž£v¨ª„§‘¢ ´ £§‰¨£ ¡w§~©r£¤r¶;Ÿ„ž† ¡†±s­5žw¡†£ ´ ¦‰ Ž¡w§‰¸ ©r¬5£¨'ª„±s±sª„§Á©\«9ª$ Ž¡w¸¯±sŸ ¤Ž¦‰žw£±sªN©Ž¡†¶_Ÿ_©r£ ´ ¾~³ Ÿ„ Ž³2±­5©ŽªN©r¡†¨T§5ª„¤r±ŸNž†¡®©\³ª„§©r¬5£T¤r¡†¸„¬~© “zà ¬5£¦5­5­C£¤¾Eª¯¦5§ ´  …ªN¥-©r¬5£©r£ l©  l£©°¾Cª„¦‰§ ´ ¬‰Ÿ;¶¯£ ! `*) ) $ ¥jŸN¡†žw¦‰¤Ž£…¤rŸN©Ž£„¹2 ŽªTŸ„ "©Žª¾E£)¨'ª¯±­EŸN¤`ŸN¾5ž†£(«(¡®©r¬ ©Ž¬5£ $ 㠎¡w¸¯±ŸÁŸN­‰­5¤Žª~Ÿ„¨`¬ n “Ñv à ¬‰£©r£ l© l£©¾Cª„¦5§ ´ ¡‡ s¾E£©l©r£¤s¾E£¬‰Ÿ;¶„£ ´ Ÿ¯  ©Ž¬5£¨'ª¯§‘¢ ´ £§‰¨'£(¡†§~©Ž£¤r¶_ŸNž5¡‡ "¨ª„§‘¢E§5£ ´ ©rª©Ž¬5£¡w§~©Ž£¤Ž¶_Ÿ„ž )g e3D ŸN§ ´ ¡‡ v§‰£¶„£¤ ª_¶„£¤lăª„­‘©r¡w±s¡‡ \©r¡†¨ “ 








©r£ l©z£¤r¤Žª¯¤9¤`Ÿ_©r£¡‡  ) “/Ó ¬‰£§«°£ª¯¾‰ Ž£¤r¶„£­E£¤l¥=£D¨g©¨'ž‡Ÿ„ r Ž¡®¢E¨Ÿ_©r¡wª¯§G¹2ª„¦5¤z¨ª„§‘¢ ´ £§‰¨£¡†§~©Ž£¤Ž¶_ŸNž  Ž¬5ª„¦‰ž ´ Z 9¬‰Ÿ;¶¯£ Ž¡†Ï£ ) ¥=ª„¤zŸ„§~³¢‰§5¡w©Ž£ n “ à ¬5£¾‰Ÿ¯ l¡‡¨­5¤rª„¾5ž†£± «(¡®©r¬À©r¬5¡‡ ŸN­5­‰¤Žª~Ÿ„¨`¬Á¡‡ )©r¬‰Ÿ_©©r¬5£ #°¡w§5ª¯±s¡†Ÿ„ž ´ ¡† l©Ž¤r¡†¾5¦‘©Ž¡†ª„§¼¡‡ §5ª„©  Ž¡w±s¡†ž†Ÿ„¤©rª Ÿ R)ŸN¦‰ r l¡‡ŸN§Ë«(¬‰£§Ñ©r¬5££¤r¤Žª¯¤T¤`Ÿ_©Ž£¡† §5£DŸN¤ ) “ %2¡w§E¨'£ª„¦5¤¸„ª~ŸNž"¡‡ T¢‰§ ´ ¡†§5¸¼Ÿ ¨ž†Ÿ¯ Ž Ž¡®¢E£¤z«(¡w©Ž¬ Ÿ  Ž±sŸ„žwž%©Ž¤r¦5££¤r¤Žª¯¤¹E¡®©¡† …£ r l£§~©Ž¡‡ŸNžG©r¬‰Ÿ_©©Ž¬‰£T±s£Ÿ„§‰ «°£T¦E l£T©Žª £¶_ŸNž†¦‰Ÿ_©r£ ¨ž†Ÿ¯ Ž Ž¡®¢E£¤` T«°ª„¤r·Ë¡w§ ©r¬5¡† s¤r£¸¯¡w±s£ “ à ¬5£$©r£ l© l£©¾Eª¯¦5§ ´ ¨ŸN§  rŸ_©Ž¡‡ l¥=³Ë©Ž¬5¡‡ s¤r£ Ð ¦5¡†¤Ž£±£§~© ²cŸN§ ´ ¹‘¡†§¥jŸ„¨'©¹5ª¯­E£¤rŸN©Ž£ °«°£ž†ž?¥=ª„¤zŸ„žwž€©r¤Ž¦‰££¤Ž¤rª„¤!¤r£¸¯¡w±s£ `µ “ ² ’ µ à ¬5£)©Ž£D \© Ž£'©(¾Cª„¦5§ ´ Ÿ„­5­5¤rª¯Ÿ¯¨`¬ ¡‡K  ZT T8[Tª¯­‘©Ž¡†±s¡† l©Ž¡‡¨ “ ² .„µ à ¬5£©r£ l©… l£©z¾Eª¯¦5§ ´ ¾‰Ÿ¯ l£ ´ ¨'ª„§5¢ ´ £§E¨'£¡†§¯©r£¤r¶_ŸNž&Ÿ„žw«!Ÿ;³‘ !¤Ž£©Ž¦5¤r§‰ zŸ„§¦‰­5­E£¤…Ÿ„§ ´ žwª_«°£¤!¾Cª„¦5§ ´ ¡†§ )g e3D “ à ¬5£ $ ` ±s£©Ž¬5ª ´ ¡‡ )Ÿ ¤r£ž†¡‡¨ªN¥"©Ž¡†±£D «(¬5£§Á¨ª„±s­5¦‘©`Ÿ_©r¡wª¯§‰ŸNž/£ G€ª„¤Ž©«!Ÿ„ …£'»‘­C£§‰ Ž¡†¶„£ “ * ©¡†  §5ª_«@ l¡†±s­5ž†£TŸN§ ´ £DŸ„ Ž³©Žª¨ŸNž‡¨'¦5ž‡Ÿ_©r£Ÿs¾Eª¯¦5§ ´ ¾‰Ÿ¯ l£ ´ ¦‰­Eª¯§$©Ž¬5£T¨¦5±T¦‰ž†ŸN©Ž¡†¶„£ ´ ¡‡ \©r¤Ž¡†¾5¦‘©r¡wª¯§ ª„¥&©Ž¬‰R £ #9¡†§5ª¯±¡‡ŸNžv²c l£s £ ? BG¥=ª„¤zŸ­5¤rª„¸¯¤rŸ„±Ç«(¬‰¡†¨`¬ ´ ª2£ °©r¬5¡‡ rµ “ Ÿ t  £Ÿ„¤Ž§‰£¤ uŸN©l©Ž£±s­‘©r  à ¬5£©Ž£D \© l£©¾Cª„¦5§ ´ ¨Ÿ„§Ñ¾C£ ©r¬5ª„¦5¸¯¬~©ª„¥Ÿ„ Ÿ½¸¯Ÿ„±s£ «(¬5£¤Ž£ ©rª¨ª„§2¶2¡w§‰¨£Ÿ¤r£Ÿ„ Žª„§EŸN¾5ž†£ t (£¤r¡®¢E£¤ uTªN¥"©r¬5£ŸN±sª„¦‰§¯©)ªN¥9žw£DŸN¤r§5¡w§‰¸«(¬5¡‡¨`¬ ¬‰Ÿ„ ª‘¨¨¦5¤Ž¤r£ ´ “ Æ ¡‡¨g©rª„¤r¡†Ÿ„žwž†³«°£¨ŸN§¤r£­5¤r£ Ž£§~©°©Ž¬5¡‡ zŸ„ !¡†§¢‰¸„¦‰¤Ž£È “ 2 “












    !

 !     !



 



! , [ !

 

ԎÔ

Test Set Bound δ

Verifier

Learner Choose c

classifier c Draw Examples Evaluate Bound

5ª¯¤/©Ž¬5¡‡  ´ ¡‡ŸN¸„¤`ŸNz ± tl¡w§E¨'¤r£Ÿ„ Ž¡†§5¸(©Ž¡†±s£ u!¡† /­Cª„¡†§¯©r¡w§‰¸ ´ ª_«(§5Ä 7!œ &šG—> «°Ÿ„¤ ´   “à ¬5£ª„§‰žw³¤r£ Ð ¦5¡†¤Ž£±s£§~©¥=ª¯¤)Ÿ„­5­5ž†³2¡w§5¸©Ž¬‰¡† ¾Cª„¦5§ ´ ¡‡ ©r¬‰Ÿ_©©Ž¬‰£ žw£DŸN¤r§5£¤…±T¦‰ l©¨ª„±s±s¡®©…©ŽªŸ¨'ž‡Ÿ„ r l¡w¢‰£¤«(¡w©Ž¬5ª¯¦‘©…·2§5ª_«(ž†£ ´ ¸¯£ªN¥©r¬5£©r£ l© £'»5ŸN±s­5ž†£  “ Ê  Ž¡†±¡†ž‡ŸN¤ ´ ¡‡ŸN¸„¤`ŸN±@¥=ª¯¤%©r¤rŸ„¡w§ l£©¾Eª¯¦5§ ´  -¡‡ /­5¤Ž£D l£§~©Ž£ ´ ž†ŸN©Ž£¤ ²jŸ„§ ´ ¡† … lª¯±s£«(¬‰ŸN©z±ª¯¤Ž£¨ª„±s­5ž†¡†¨Ÿ_©Ž£ ´ µ “°Ó £T¨Ÿ„§$©Ž¬5¡†§5·ªN¥/©Ž¬5£¾Cª„¦‰§ ´ Ÿ„ Ÿ©r£¨`¬5§5¡ Ð ¦5£T¾2³«(¬5¡‡¨`¬©Ž¬5f£ t  £Ÿ„¤Ž§‰£¤ u¨ŸN§½¨'ª¯§2¶~¡†§‰¨£©Ž¬5f£ t (£¤r¡®¢‰£v¤ u ©Ž¬‰ŸN©ž†£Ÿ„¤Ž§5¡†§5¸Ñ¬EŸ„ ª‘¨¨'¦5¤r¤Ž£ ´ ²jŸN§ ´ ©Ž¬5£ ´ £¸¯¤Ž££$©Žª{«(¬5¡†¨`¬ ¡w©¬EŸ„ ª‘¨'Ä ¨'¦5¤r¤r£ ´ µ “ 59Ÿ„¨`¬ ª„¥?©Ž¬5£­5¤rª2ªN¥j °¨ŸN§¾E£…©Ž¬‰ª„¦5¸¯¬¯©°ªN¥%Ÿ¯ 9Ÿ¨'ª¯±±¦5§5¡‡¨ŸN©Ž¡†ª„§ ­5¤rªN©Žª‘¨ª„ž€¥=ª„¤zŸ„§¡†§~©Ž£¤rŸ¯¨g©Ž¡†¶„£­5¤rª2ªN¥%ª„¥/ž†£Ÿ„¤Ž§‰¡w§5¸¾2³©r¬5£  £DŸN¤r§5£¤ “ 







›€› 9

 





Ô


     !

Occam’s Razor Bound

δ Verifier

Learner "Prior", P(c)

Draw Training Examples

m examples Choose c

Classifier, c Evaluate Bound

* §Ëª¯¤ ´ £¤©rªÀŸN­‰­5žw³À©Ž¬‰£$̨¨ŸN±$Í  ÎzŸNϪ„¤¾Cª„¦‰§ ´ ¡w©¡‡  7!œ &šG —>  5§ £¨£ r ŽŸ„¤Ž³ ©r¬‰Ÿ_©©Ž¬5£Ë¨`¬5ª¯¡†¨£Áª„¥ t\­‰¤Ž¡†ª„¤ uÁ¾E£ ±Ÿ ´ £ ¾C£'¥=ª„¤r£¼ l££¡†§5¸{Ÿ„§~³ ©Ž¤`ŸN¡†§5¡†§5¸£'»5Ÿ„±­‰žw£D  “!à ¬5£§&¹‰©r¬5£T¾Cª„¦5§ ´ ¡† …¨ŸNž‡¨'¦‰ž†ŸN©Ž£ ´ ¾‰Ÿ¯ l£ ´ ¦5­Cª„§$©Ž¬‰£ ¨`¬5ª¯ Ž£§s¨'ž‡Ÿ„ r Ž¡®¢‰£¤ “ " ª„©Ž£°©Ž¬‰ŸN©v¡®©GY 4 tlžw£¸¯ŸNž u"©rª)¨`¬‰ª¯ Ž£"©r¬5£z¨'ž‡Ÿ„ r l¡w¢‰£¤/¾EŸ„ Ž£ ´ ¦5­Cª„§©Ž¬5£­‰¤Ž¡†ª„
¤ q Ÿ„ !«°£ž†žGŸ„ °©r¬5££±s­5¡†¤Ž¡‡¨Ÿ„ž?£¤r¤Žª¯¤ q ` b “ F I 


 

“w’¯“ W  OO[L M  L 2 , y 2SUw/ Z)à ¬5¡‡ ¾Cª„¦5§ ´ ¡†§Ë±sª„¤r£ŸN­5­5¤rª;»‘¡w±ŸN©Ž£¥=ª„¤r± ¬‰Ÿ„  Ÿ„­5­E£DŸN¤r£ ´ £ž† Ž£«(¬5£¤Ž£? ’ B ? ’ 4CB “ÇÓ £À¦‰ l£1t\­‰¤Ž¡†ª„¤ u¿²=«(¡w©Ž¬ Ð ¦5ª„©Ž£D rµ¬5£¤r£½¾C£¨Ÿ„¦‰ Ž£Á¡w©¡‡ Ÿ„§ Ÿ„¤Ž¾5¡w©Ž¤`ŸN¤r³Ë­5¤rª„¾‰Ÿ„¾5¡†žw¡w©\³ ´ ¡‡ l©Ž¤r¡w¾5¦5©Ž¡†ª„§ ª_¶„£¤ ¨'ž‡Ÿ„ r Ž¡®¢‰£¤r Ÿ„§ ´ §5ª„© §5£¨£ r ŽŸ„¤Ž¡†ž†³¿Ÿx#!Ÿ;³„£D l¡‡ŸN§ ­5¤r¡†ª„¤ “%à ¬5£ ´ ¡‡ \©r¡w§‰¨'©Ž¡†ª„§¡‡ 9¡w±s­Cª„¤Ž©rŸN§~©D¹¯¾C£¨ŸN¦‰ Ž£z©r¬5£©r¬5£ª¯¤Ž³¬5ª¯ž ´  9¤r£¸¯Ÿ„¤ ´ žw£D Ž ª„¥G«(¬5£'©r¬5£¤ ª¯¤°§‰ªN©s Ÿ #!Ÿ;³„£ Ž¡‡ŸN§­5¤r¡†ª„¤(¡‡ !¦‰ l£ ´ “ 0





W02

q ?A@W B[

, W0 ^ 

d&

 à ¬5£s Ž£ Ð ¦5£§E¨'£sªN¥"­‰¡†¨'©Ž¦5¤r£ )¡‡ …©Ž¬5£s­5¡‡¨g©rª„¤r¡†Ÿ„ž/¤r£­‰¤Ž£D l£§‘Ä ©rŸN©Ž¡†ª„§Áª„¥©Ž¬5£­5¤rª2ªN¥vªN¥©r¬5£Ì¨¨Ÿ„±$Í  …ÎzŸNϪ„¤ #°ª¯¦5§ ´ “à ¬5£T¢‰¤r l©¢E¸„¦5¤r£  l¬‰ª_«z %Ÿz Ž£'©-ªN¥‰¨ž†Ÿ¯ Ž Ž¡w¢‰£¤` ¹D£Ÿ¯¨`¬«(¡w©Ž¬TŸz©rŸN¡†ž2¨'¦‘©-ªN¥‰ Žª„±s£"¶_Ÿ„¤Ž³2¡†§5¸ ´ £­‘©Ž¬ “ à ¬5£z Ž£¨'ª¯§ ´ ­5¡†¨'©Ž¦5¤r£( l¬‰ª_«z /Ÿ„§ª¯¾‰ Ž£¤r¶„£ ´ ©Ž¤`ŸN¡†§5¡†§5¸…£¤Ž¤rª„¤Ÿ„§ ´ ©r¬5£!­Eª~ Ž Ž¡wÄ ¾5ž†'£ #9¡†§5ª¯±¡‡ŸNž ´ ¡† l©Ž¤r¡w¾‰¦‘©Ž¡†ª„§‰ ¥=ª„¤vŸ)¨`¬5ª¯ Ž£§¨'ž‡Ÿ„ r l¡w¢‰£¤ “/à ¬5£!©Ž¬5¡†¤ ´ ­‰¡†¨'©Ž¦5¤r£  l¬‰ª_«z G©r¬5£"©r¤Ž¦‰£"£¤r¤rª„¤` ?«(¬5¡‡¨`¬ŸN¤r£"¨ª„§‰ Ž¡† l©Ž£§~©%«(¡w©Ž¬©r¬5£°ª„¾‰ Ž£¤r¶_Ÿ_©Ž¡†ª„§Ÿ„§ ´ ©Ž¬5£)©`ŸN¡†ž&¨'¦5©r  “và ¬5£)¥=ª„¦5¤Ž©Ž¬­5¡‡¨g©r¦5¤Ž£ Ž¬5ª_«z 9©Ž¬‰£)©Ž¤r¦5£)£¤r¤Žª¯¤°¾Cª„¦‰§ ´ “ 





«(¬5¡‡¨`¬¡w±s­5ž†¡w£D  "

; 7< b E=

 F

q87

q y

°¡w§

` Fqb

#



£¸¯ŸN©Ž¡†§5¸©Ž¬5¡‡ zŸN¸~ŸN¡†§¨ª„±s­5ž†£'©r£ °©Ž¬‰£­5¤Žª2ª„¥ “

! g

H

F q ItI

!

L

0 “w’¯“w’¯“  ](](^ V  4 ^ B[ = [t B  @^ [\YTE43K „¦E \©Ÿ¯ T«(¡w©Ž¬Ñ©Ž¬5£ ©Ž£ l© l£©T¾Eª¯¦5§ ´ ¹"«9£ ¨ŸN§Ñ¤r£ž‡Ÿ_» ©r¬5£s̨¨ŸN±$Í  ÎzŸ„Ϫ¯¤z¾Eª¯¦5§ ´ ²=©Ž¬5£ª„¤r£±r0 “†’ µ(«(¡®©r¬½©Ž¬‰£s9¬5£¤Ž§5ªG Ÿ„­5­5¤rª;»2¡†±Ÿ_©r¡wª¯§‰ !©Žª¸„£© Ÿs Žª„±s£«(¬‰ŸN©(±ª¯¤Ž£…©Ž¤`Ÿ„¨'©rŸ„¾5žw£)£»2­‰¤Ž£D Ž Ž¡wª¯§ “  2, 2Z""XLz, Z4$Z10 T8[\Z  ]](^BV  4 ^ [ 7 8z+95: ;x B[ ^B  X[\Y [=4 F q I T8[ ]W P^_T8[=4=?A@W [





^B 

! B







1

 

F

)g,eED

b E ;=7
 zà ¬5¡‡ ¡† sŸ½­‰žwª„©s¨ª„±s­‰ŸN¤r¡†§5¸Á¨'ª¯§‘¢ ´ £§‰¨'£¡w§~©r£¤r¶;Ÿ„ž† ¾5¦5¡†žw© ¾‰Ÿ„ Ž£ ´ ¦5­Cª„§½©r¬5£©Ž£ l© l£©¾Cª„¦5§ ´ ²jÈ “ È~µz«(¡w©Ž¬ ŸN§ AE
.E
©Ž¤`ŸN¡†§ D©r£ l©  l­‰žw¡w©ª„§©Ž¬‰£žw£¥ˆ©ŸN§ ´ ©r¬5£Ì¨¨ŸN±$Í  °ÎŸNϪ¯¤(¾Eª¯¦5§ ´ ² 0 “w’ µ°«(¡®©r¬ŸNž†ž ´ Ÿ_©`Ÿ ¡w§s©r¬5£(©r¤rŸ„¡w§5¡†§5¸ Ž£'©vª¯§©r¬5£z¤r¡w¸¯¬~© “-à ¬5£Ì¨¨ŸN±$Í  -¤rŸ„Ϫ„¤-¾Cª„¦5§ ´ ¡† 9 lª¯±£Ä ©Ž¡†±s£ ! l¦‰­E£¤Ž¡†ª„¤°ª„§©r¬5£) l±ŸNž†ž†£¤ ´ Ÿ_©`ŸT Ž£'©r °ŸN§ ´ ŸNž†«°Ÿ;³‘ "§5ª„§2¶_Ÿ„¨¦5ª„¦‰ …²=¡†§ ¨'ª¯§¯©r¤rŸ¯ \©!©rª±Ÿ„§~³ ªN©Ž¬‰£¤!©Ž¤`ŸN¡†§$ l£©z¾Eª¯¦5§ ´  `µ “ 


  



“ È “w’¯“ [tTE458z 4EK à ¬5£Ì¨¨Ÿ„±Í  sÎzŸNϪ„¤¾Eª¯¦5§ ´ ¡† s l©Ž¤rª„§5¸¯žw³¼¤r£ž‡Ÿ_©r£ ´ ©rª ¨ª„±s­5¤r£ r l¡†ª„§ “ * § ‰­ Ÿ„¤l©r¡†¨¦5ž‡ŸN¤D¹E¥=ª¯¤ŸN§2³½ Ž£žw¥ˆÄc©r£¤r±¡†§‰ŸN©Ž¡†§5¸ ´ £ r¨'¤r¡w­‘©r¡wª¯§Àž‡ŸN§‰¸„¦‰Ÿ„¸„£„¹ q ¹&«°£¨ŸN§ Ÿ„ r lª‘¨¡†ŸN©Ž£sŸ >  D «(¡w©Ž¬ ©Ž¬5£ ­5¤Žª¯­E£¤l©\³$©r¬‰Ÿ_© tl­5¤r¡wª¯¤vu e F “ I 9ª¯§‰ Ž£ Ð ¦5£§~©ržw³¯¹& l¬‰ª„¤Ž© q ` $   q  F I F I ´ £D Ž¨¤Ž¡†­‘©r¡wª¯§Ñžw£§5¸N©r¬{¨ž†Ÿ¯ Ž Ž¡®¢E£¤` ©Ž£§ ´ ©rªÀ¬EŸ;¶„£ Ÿ½©Ž¡†¸„¬~©Ž£¤s¨ª„§2¶„£¤Ž¸¯£§‰¨£ŸN§ ´ ©Ž¬5£$­C£§‰Ÿ„ž®©\³ ©r£¤r±¹
k>  D ¡† …©Ž¬‰£§2¦5±¾E£¤…ª„T¥ tl§‰Ÿ_©`$  u²j¾5¡®©` )¾‰Ÿ„ Ž£T£;µ “ ‰ª„¤)ŸN§2³$ž†Ÿ„§5¸„¦EŸN¸„£T¢5»‘£ ´ ¾E£¥=ª„¤r£  Ž££¡w§5¸©Ž¬5£…©Ž¤`ŸN¡†§ l£©¹‘¨'ž‡Ÿ„ r Ž¡®¢‰£¤r "«(¡®©r¬  Ž¬5ª„¤Ž©Ž£¤ ´ £D Ž¨¤Ž¡†­‘©Ž¡†ª„§ žw£§5¸N©r¬‰ "¬EŸ;¶„£(©r¡w¸¯¬¯©r£¤9¾Eª¯¦5§ ´   ª¯§ ©r¬5£)©Ž¤r¦5£)£¤r¤Žª¯¤!¤rŸN©Ž£ “ ̅§5£z­EŸN¤Ž©Ž¡‡¨'¦5ž‡ŸN¤ržw³T¦‰ l£¥=¦5ž ´ £ r¨'¤r¡w­‘©r¡wª¯§sž†Ÿ„§5¸„¦‰Ÿ„¸„£°©Žª¨'ª„§E l¡ ´ £¤¡‡ ©r¬5£z£»2£D¨'¦‘©r¡wª¯§s©Ž¤`Ÿ„¨£(ªN¥ Ÿ žw£DŸN¤r§5¡w§‰¸ ŸNž†¸„ª¯¤Ž¡w©Ž¬5± “ * ¥9«°£¨ŸN¤r£'¥=¦5ž†žw³$§5ª„©Ž£T©Ž¬5£s l£ Ð ¦5£§‰¨£ª„¥ ´ Ÿ_©`Ÿ_Ä ´ £­C£§ ´ £§~©¨`¬5ª„¡‡¨'£D  «(¬5¡‡¨`¬ÑŸ½ž†£Ÿ„¤Ž§‰¡w§5¸½Ÿ„žw¸¯ª„¤r¡®©r¬5± ±sŸ„·„£D ¹%©Ž¬5£§Ñ©r¬5£ ª¯¦‘©Ž­5¦5©¨'ž‡Ÿ„ r Ž¡®¢‰£¤T¨Ÿ„§Ñ¾C£ Ž­E£D¨'¡w¢‰£ ´ ¾2³ Ÿ Ž£ Ð ¦5£§‰¨'£ Ž¦‰¨`¬ËŸ„:  t ._§ ´ ¨`¬5ª¯¡†¨£„¹%©Ž¬5¡†¤ ´ ¨`¬‰ª„¡‡¨'£„¹&¢‰¤` \©¨`¬‰ª„¡‡¨'£„¹%£'©`¨ “†“w“ u à ¬5¡† T¡† ©Ž¬5£¡ ´ £DŸ ¾C£¬5¡†§ ´ ±¡‡¨'¤rª‘¨`¬5ª„¡‡¨'£T¾Eª¯¦5§ ´ @   ? ’ ECB “ Î(£ Ž¦5žw©r …¥=ª„¤…©Ž¬‰¡† )ŸN­‰­5¤Žª~Ÿ„¨`¬ŸN¤r£T¤r£­Cª„¤Ž©Ž£ ´ ¡†§ /¡w¸¯¦5¤r£ 0 “ ÈzŸ„§ ´ Ÿ„¤Ž£9 \©r¤Žª¯§5¸£§5ª„¦5¸¯¬©ŽªŸ„¨'©/Ÿ„ /ŸN§T£±s­5¡†¤Ž¡‡¨Ÿ„ž¯£»2¡‡ l©Ž£§E¨'£°­5¤Žª2ª„¥‘©Ž¬‰ŸN©Ì¨¨ŸN±$Í  %ÎzŸ„Ϫ¯¤ ¾Cª„¦5§ ´  (¨Ÿ„§¾C£±Ÿ ´ £)©r¡w¸¯¬¯©(£§5ª„¦5¸¯¬ ¥=ª¯¤(¦‰ Ž£'¥=¦5ž&Ÿ„­5­5ž†¡†¨Ÿ_©r¡wª¯§ “ 0















“   9
> )˜%šG•G™  à ¬5£ Æ Ê "Ä$#!Ÿ;³„£ !¾Cª„¦‰§ ´ ? ’ 4CB/¡† ­‰ŸN¤Ž©Ž¡‡¨'¦5ž‡ŸN¤rž†³£'»5¨'¡w©Ž¡†§5¸¾C£¨ŸN¦‰ Ž£¡w©…¨Ÿ„§­5¤Žª_¶2¡ ´ £ Ð ¦‰ŸN§‘Ä ©r¡®©`Ÿ_©Ž¡†¶„£žw³ ¦‰ Ž£'¥=¦5ž!¤r£ Ž¦5žw©r ¥=ª¯¤¨ž†Ÿ¯ Ž Ž¡®¢E£¤` «(¡w©Žf ¬ [tT(^B ^ 8 T=9¼­EŸN¤`ŸN±s£'©r£¤`  “Áà ¬‰¡† T¡w§‰¨žw¦ ´ £  2










    !

δ Verifier

 !     !

 





! , [ !

 

Ô

PAC−Bayes Bound Learner "Prior", P(c)

Draw Training Examples

m examples "Posterior", Q(c)

Choose Q(c)

Evaluate Bound

7!œ &šG—> à ¬5£ tl¡w§~©r£¤`Ÿ„¨g©r¡w¶¯£­5¤rª2ªN¥ªN¥vž†£Ÿ„¤Ž§5¡†§5¸uŸ¯ Ž Žª‘¨'¡‡Ÿ_©r£ ´ «(¡®©r¬ ©Ž¬5£ Æ Ê "Ä$#!Ÿ;³„£D !¾Eª¯¦5§ ´ “(à ¬5£)¢‰¸¯¦5¤r£¡† z©Ž¬5£T rŸN±s£Ÿ¯ !¥=ª„¤(©r¬5£Ì¨¨ŸN±$Í   ¤rŸ„Ϫ¯¤z¾Eª¯¦5§ ´ ¹C£'»5¨£­‘©z©r¬‰Ÿ_©…¡w§‰ l©Ž£DŸ ´ ª„¥¨ª„±s±s¡®©Ž©Ž¡†§5¸©rªŸ Ž¡w§5¸¯žw£¨'ž‡Ÿ„ r l¡wÄ ¢‰£¤D¹5©r¬5£ Æ Ê "$Ä #°Ÿ;³¯£ !¾Cª„¦5§ ´ Ÿ„­5­5ž†¡w£D (©rªŸN§2³ ´ ¡‡ l©Ž¤r¡w¾5¦5©Ž¡†ª„§$ª_¶„£¤z¨'ž‡Ÿ„ r l¡wÄ ¢‰£¤`  “ 


 

 Ž¦‰¨`¬¨'ª¯±s±ª¯§5ž†³¦‰ Ž£ ´ ¨'ž‡Ÿ„ r l¡w¢‰£¤` vŸ¯  %2¦5­‰­Eª¯¤l©K(£¨g©rª„¤ Å Ÿ„¨`¬5¡†§5£D "ŸN§ ´ " £¦5¤rŸ„ž " £'©\«°ª„¤r·2  “ Ô Ã ¬5¡‡ ( Ž£¨g©r¡wª¯§¡‡  ´ ¡†¶2¡ ´ £ ´ ¡w§~©rª©r¬5¤r££)­‰ŸN¤Ž©r ,+ ²’l µ %2¦5¾‰ Ž£¨'©Ž¡†ª„/ § 2 “†’ %~©rŸN©Ž£ ŸN§ ´ ­5¤rª_¶„£ 9©r¬5£ Æ Ê "Ä3#!Ÿ;³„£D  #9ª¯¦5§ ´ “ ² .„l µ %2¦5¾‰ Ž£¨'©Ž¡†ª„/ § 2 “ . l¬5ª_«z °©r¬‰Ÿ_©(©r¬5£ Æ Ê "3Ä #!Ÿ;³¯£ K#9ª¯¦5§ ´ ¡‡ !§5£Ÿ„¤Žž†³ Ÿ„ °©r¡w¸¯¬~©(Ÿ¯ !­Eª~ \Ä  l¡†¾5ž†£¸„¡†¶„£§ ©r¬5£)ª„¾‰ Ž£¤r¶_Ÿ_©Ž¡†ª„§E  “ ²cȯl µ %2¦5¾‰ Ž£¨'©Ž¡†ª„c § 2 “ È ´ ¡† r¨'¦‰ r Ž£ ¤Ž£D l¦‰ž®©`  ¥=¤rª„± ©r¬5£ËŸN­5­‰žw¡‡¨ŸN©Ž¡†ª„§ ª„¥T©r¬5£ Æ Ê "Ä$#!Ÿ;³„£D  ¾Eª¯¦5§ ´ ©Žª Ž¦5­5­Cª„¤Ž©(¶„£D¨g©Žª¯¤!±sŸ¯¨`¬5¡†§5£  “ 2 “w’¯“ W " 
J@L W M J62 STUw/ Z)à ¬5£ Æ Ê "3Ä #!Ÿ;³¯£ ¾Eª¯¦5§ ´ ¬‰Ÿ¯ $¾E££§y¡w±s­5¤rª_¶„£ ´ ¾2³ ©r¡w¸¯¬¯©r£§5¡†§5z ¸ ? ’ .CB)Ÿ„§ ´ ©Ž¬5£§ «(¡w©Ž¬ ŸÑ±T¦‰¨`¬  Ž¡†±­‰žw£¤ ­5¤rª2ªN¥ ? ’ ACB l¡†§‰¨'£Á¡®©$«°Ÿ¯ ª„¤r¡†¸„¡†§‰ŸNž†žw³  l©rŸN©Ž£ ´ “à ¬5£ l©rŸN©Ž£±s£§¯©ŸN§ ´ ­5¤rª2ªN¥9­‰¤Ž£D l£§¯©r£ ´ ¬5£¤Ž£¡†§‰¨'ª¯¤Ž­Cª„¤`Ÿ_©r£©Ž¬5£D l£¡†±s­5¤Žª_¶¯£±s£§~©`  Ÿ„§ ´ ¡†±s­5¤rª_¶„£ª„§©Ž¬5£±  Žžw¡†¸„¬~©Žž†³ “ à ¬5£ Æ Ê "3Ä #!Ÿ;³¯£ ¾Eª¯¦5§ ´ ¡†  ´ £­E£§ ´ £§¯©¦‰­Eª¯§¿©\«°ª ´ £¤r¡w¶¯£ ´¿Ð ¦‰Ÿ„§~©Ž¡w©Ž¡†£ ¹9ŸN§ Ÿ;¶„£¤rŸ„¸„£ ©r¤Ž¦5£)£¤Ž¤rª„¤ +  7 9 E qW7 Ÿ„§ ´ Ÿ„§Ÿ;¶¯£¤`ŸN¸¯£z©Ž¤`ŸN¡†§£¤r¤Žª¯¤,+  ` b 9 E q ` b à ¬5£D l£ Ð ¦EŸN§~©Ž¡w©Ž¡†£ ¨Ÿ„§½¾E£¡†§¯©r£¤r­5¤r£'©Ž£ ´ Ÿ„ ©Ž¬5£T©Ž¤`ŸN¡†§½£¤r¤rª„¤Ÿ„§ ´ ©Ž¤r¦5££¤r¤Žª¯¤zªN¥©r¬5£±s£©rŸ_Ä ¨ž†Ÿ¯ Ž Ž¡®¢E£¤?«(¬5¡‡¨`¬¨`¬5ª2ª¯ Ž£ GŸ(¨ž†Ÿ¯ Ž Ž¡w¢‰£¤&Ÿ¯¨¨ª„¤ ´ ¡w§‰¸9©rª{£¶¯£¤r³z©Ž¡†±s£9Ÿ(¨ž†Ÿ¯ Ž Ž¡®¢C¨Ÿ_©r¡wª¯§)¡† &±sŸ ´ £ “ * ¥€«9£!¤r£'¥=£¤-©Žª)©Ž¬‰¡† ±s£©rŸ_ÄL¨'ž‡Ÿ„ r Ž¡®¢‰£¤Ÿ„ s¹„©Ž¬5£(§5ª„©rŸN©Ž¡†ª„§¥=ª„¤£¤r¤rª„¤-¤rŸN©Ž£D -¡† v¨ª„§‰ Ž¡‡ \©r£§~©«(¡®©r¬ ª¯¦5¤!£ŸN¤rž†¡w£¤°§‰ªN©rŸN©Ž¡†ª„§ “ Ô Gh;S\bjSvVXWG^°Yl^lt'Sl^rd/hDSlbjS dZh;Sv]Neg[;f;U…egf;n i…^`a;a;nXVXS\W?dZe°WcdceYƒh;^`WcdcVXY-t'SlbjWZVXegfDW%e`u‘dch;SvY\no^`WZWZV p;SlbjWŽ} , eŽ‚Gq SLt'Slb 8 dch;Sa;bjeg]_^`];VXnXV dˆidZh;^rd(dZh;SWjdZeYƒh_^`WjdZVXYY\no^`WcWZV p_S\b(U;V „S\bcW(uwbcegm dch;S)Y\no^`WZWcV p_S\bYl^`f ]NS)mz^`U;S…t'S\b=i WZmz^`nXn‡} 



+

+

Ô


     !

à ¬5:£ tl¡w§~©r£¤`Ÿ„¨g©r¡w¶¯£…­‰¤Žª2ªN¥-ªN¥/žw£DŸN¤r§5¡†§5¸ u)¶~¡†£«(­Cª„¡†§~©(ªN¥/©Ž¬5£ Æ Ê "Ä$#°Ÿ;³¯£ °¾Cª„¦5§ ´ ¡‡ ( Ž¬5ª_«(§ †¡ §¢‰¸„¦5¤r£62 “†’„“ * ©z¡† £ r l£§¯©r¡†Ÿ„žwž†³©r¬5£ ŽŸ„±s£Ÿ„ !¥=ª¯¤!©Ž¬5£Ì¨¨Ÿ„±Í  (ÎzŸ„Ϫ„¤(¾Cª„¦‰§ ´ £'»5¨'£­‘©(¥=ª„¤ ©r¬5£¨'ª¯±±s¡w©Ž±s£§~©!©Žªs©r¬5£±s£'©`Ÿ„¨'ž‡Ÿ„ r Ž¡®¢‰£¤  ¤`Ÿ_©r¬5£¤!©r¬‰ŸN§©Ž¬5£¨ž†Ÿ¯ Ž Ž¡w¢‰£¤ q “ 

@W B[

W02

^ 

, 0W 

&Z,0 J )g,eED


Z

! B F

b E; 7
g g
k D “ %2¡†§‰¨'£)©r¬5£   Ä ´ ¡†¶„£¤Ž¸¯£§‰¨£…Ÿ„­5­5ž†¡w£D !©Žª ´ ¡‡ \©r¤Ž¡†¾5¦‘©r¡wª¯§‰ (ª_¶¯£¤(¨'ª¯§¯©r¡w§2¦5ª¯¦‰ !¶_ŸNž†¦5£ ´ ­‰Ÿ„¤rŸ„±Ä £©Ž£¤` ¹5©Ž¬‰£ Æ Ê "Ä3#!Ÿ;³¯£ !¾Cª„¦5§ ´ ¨ŸN§¾C£T§‰ª„§~©Ž¤r¡w¶2¡‡ŸNž†žw³ ©Ž¡†¸„¬~©¡†§©Ž¬‰¡†  l£©l©r¡w§5¸Ÿ„ z«°£ž†ž “zà ¬5¡†  ¥jŸ¯¨g©(¡† z¦‰ l£ ´ ¡†§©Ž¬5£Ÿ„­5­5ž†¡†¨Ÿ_©r¡wª¯§ Ž£¨'©Ž¡†ª„§ “ Ó £(¢‰¤` \©" l©rŸN©Ž£zŸ¨'ª¯¦5­5ž†£z l¡†±s­5ž†£zžw£±s±sŸ¯ -©Ž¬EŸ_©9Ÿ„¤Ž£!¬‰Ÿ„§ ´ ³T¡†§s©Ž¬‰£z­5¤rª~ª„¥ “/à ¬‰£(¡w§~©Ž¦‰¡®©r¡wª¯§ ¾C£¬5¡†§ ´ ©Ž¬‰¡† (ž†£±s±Ÿ¡† (©Ž¬‰ŸN©(©Ž¬5££»2­C£¨'©Ž£ ´ ­5¤rª„¾EŸN¾5¡†žw¡w©\³ª„¥/ŸN§$£¶„£§~©(¡‡ !§5ªN©(©rª~ª Ž±ŸNž†ž “ W- xL Z$Z ;x B[ ^  F q I ? ;x B[ ^  ! B F )g,eED 






  E
b E; 7 < 

J [ + I@EK " Nª ©Ž£©r¬‰Ÿ_©+ b E 7  q ;=< b E 7 

e

+

F

q` b



;=   D g



3

 

` F b

oHo



7 I

1 W&‚€SŽ^gS\f;S\U‚&V dch…dch;SbjSlno^rdcV t'SvS\fdcbcega'i F h;S\bjf;e T]Neg[;f;U ‹ 0 S\m(mz^D}   egfdZh;S < VXfDegm(Vo^`n‡}

J [ + I@EK




    !

k

E

g

+




n

` +

E q8` b

e 4

qW7

F

¯£§‰ Ž£§&Í  °¡†§5£ Ð ¦‰Ÿ„žw¡w©\³Á²XwÀ¨'ª¯§‰¨Ÿ;¶¯£ 1

E  q8` b +

+

ŸN§ ´

e



4

qW7

E

F





n n



! , [ !

Ô

 

k

b 



 

    > k    D      

q`

n

 !     !

+q ` b

q`

  b

w +



e)d

q8` b

e)d

e



I

I ^

F

E +

n

I

w



g

^

+

F ` b

-



d

q87

e)d

qW7

e)d

+

e



I

n

4

7 +

`

n

q`

n

 b

n

e







µ+



 >   D

E 

`



e)d

E n  n

F

b q8` b

I

'

e



e)d



 7

 ` b 

Ó ¡w©Ž¬$©Ž¬‰£ Ž£…©\«°ªsžw£±s±sŸ¯ ¹2©r¬5£ Æ Ê "Ä3#!Ÿ;³¯£ 9©Ž¬5£ª„¤r£± ¡‡ (£DŸ„ Ž³s©Žª­5¤rª_¶„£ “ J [ + I@EKT²ƒÌ¥%©r¬5£ Æ Ê "$Ä #!Ÿ;³„£D "©r¬5£ª¯¤Ž£±µ  £'© 

) 

` 

q g q`

b I

`



;=< b E 7

b

+

+



+

+

q`



+

L

e

k >    D F q I    E
 S    j    F I   q e    o o  ` F I ; <  q8` b  E
E b 7  F I E F I ;=< b E 7 F q ` b ` q ` b  I Fq I e e    o o d  E    E
F I ;= / A k D A

?
A C8E b +

+

?
A C8E b

 E > u A k D l

 E > u A k D





  E S> A k D l   E > u A k D  E > u A k D  +

+

/

+

'0'



/



`



/

+





-



+

l



L  F L  F 

L -





H







H

 4

4



 

dGL

   ) H I     d

H

L 

'

)

I

 H   H



'




    !

 !     !



 



! , [ !

`Ô

 

 l¡†§5¸©Ž¬5£ Ž³2±s±£©Ž¤r¡†¨¡®©\³ ªN¥&©r¬5£ )R ŸN¦E Ž Ž¡†Ÿ„§G¹2©Ž¬‰¡† (¡‡ ,+

M

     L L L   H  b  E > u A k D  E S> u A k D H l  l¡†§5¸©Ž¬5£)¥jŸ¯¨g©(©r¬‰Ÿ_©!©Ž¬‰£ l¦5± ªN¥/©\«9ª R)ŸN¦E Ž Ž¡†Ÿ„§‰ !¡‡ (Ÿ R)Ÿ„¦‰ r l¡‡ŸN§+ 
` ?BA C8E b  L L I
E u A k
 % %    l F  
?A C8E b E  A

` L L I u   % F `

M

+

?
A C8E

+

+



+

+

+

-

+

`

-

  

l

?
A C8E b   F -

+

¢‰§‰¡† Ž¬5¡†§5¸©Ž¬5£­5¤rª2ªN¥ “

  

`

+F  

Ib

g-

oo



F  

g

I7

-

g

F H

 Ž¡†§5¸s©Ž¬5£¨ª„¤rª„ž†ž†Ÿ„¤Ž³¯¹„©r¬5£)©Ž¤r¦5£)£¤r¤rª„¤!¾Eª¯¦5§ ´   M

4

-

F  

L

II g-

% /

` '

L  rŸ_©r¡† l¢‰£ °©r¬5££ Ð ¦‰ŸN©Ž¡†ª„§+

I7

g



4

(




k

n

à ¬5¡‡ !¡‡ zŸN§¡w±s­5ž†¡‡¨'¡w©z£ Ð ¦‰Ÿ_©r¡wª¯§ ¥=ª¯¤   «(¬‰¡†¨`¬¨Ÿ„§¾C££Ÿ¯ l¡†ž†³  lª¯žw¶¯£ ´ §~¦‰±£¤Ž¡‡¨Ÿ„žwž†³ “ à ¬5£¾Cª„¦5§ ´ ¡‡ ( l©rŸ_©r£ ´ ¡†§©r£¤r± !ªN¥ ´ ªN©z­5¤rª ´ ¦‰¨g©` !¬5£¤r£„¹5 Žªs§‰Ÿ_©r¦5¤`ŸNž†žw³¡w©z¡‡ !­Eª~ Ž Ž¡w¾‰žw£)©rª ·¯£¤r§5£ž†¡wÏ£©Ž¬5£¤Ž£D l¦5žw©z¦‰ Ž¡w§‰¸±s£'©Ž¬‰ª ´  (¥=¤rª„± ? 4 B “ * §·„£¤Ž§5£žw¡†Ï£ ´ ¥=ª„¤r±¹5©Ž¬5£¾Eª¯¦5§ ´ ŸN­‰­5žw¡†£  ©rªs¨ž†Ÿ¯ Ž Ž¡w¢‰£¤` )²jŸ„ !ª¯¦‘©Ž­‰¦‘©z¾2 ³ % ( Å ž†£Ÿ„¤Ž§‰¡w§5¸sŸNž†¸„ª¯¤Ž¡w©Ž¬‰±s `µ"ªN¥/©Ž¬5£)¥=ª¯¤Ž±1+ ² 2 “†’ µ q

FHI

 l¡†¸„§  `

i g

hi

i

jNk



g

FH

HZI

 i

i

2¡†§‰¨£„¹-¾2³ÀŸ„ r Ž¦5±s­‘©Ž¡†ª„§G¹ ¡‡ TŸ$·¯£¤r§5£žZ¹%«°£·2§5ª_« ©Ž¬‰ŸN© g `      FHZI «(¬‰£¤r£  F H  H I  F H I i i i i   i ¡† s Žª„±s£ ­‰¤Žª \\£¨g©r¡wª¯§¿¡†§~©Žª¼ŸN§5ª„©Ž¬5£¤ Ž­‰Ÿ„¨£ “ i* §{·„£¤r§5£žw¡†Ï£ ´ =¥ ª„¤r±$¹v«9£¸„£©   ` Fg HZI  FH g HZI ¹ H   H  `  FH g HZI ¹      `  FH g H  I ¹ ´ £'¢‰§5¡†§5¸¿Ÿ„žwž ªN¥)H  ©Ž¬5£  A N j k §5£D¨'£D Ž r ŸN¤r³ Ð ¦EŸN§~©Ž¡w©Ž¡†£ !©rª¨ŸNž‡¨'¦‰ž†ŸN©Ž£©Ž¬5£§‰ª„¤r±sŸ„žw¡†Ï£ ´ ± ŸN ¤r¸„¡†§G¹ %





i g

i



i

g jNi k FH i Z H I i A FH I g A g A g   F H g H  I FH HI   jNk k   ̅§5£$£ž†£±s£§¯©s¤r£±ŸN¡†§‰ ¹"«(¬5¡‡¨`¬¿¡‡ ©Ž¬5£$¶_ŸNž†¦5£$ªN¥ - “ M §‘¥=ª„¤Ž©Ž¦5§EŸ_©Ž£žw³Ñ©Ž¬5£$¾Cª„¦5§ gtL

`

 



´ ¨Ÿ„§ ¾C£T§5ª¯§5±sª„§5ª„©Žª¯§5¡†¨¡w§©r¬5£T¶_Ÿ„žw¦5£ªN¥ ¹C¾5¦‘©)¡®©…©Ž¦5¤r§‰ …ª„¦‘©…©Ž¬‰ŸN©¥=ª¯¤…¨'ž‡Ÿ„ r Ž¡®¢‰£¤r žw£DŸN¤r§5£ ´ ¾2³  Ž¦5­5­Cª„¤Ž©9¶¯£¨g©rª„¤°±Ÿ„¨`¬5¡†§5£ 9ª„§¤Ž£DŸ„ Žª„§‰Ÿ„¾5ž†£ ´ Ÿ_©`Ÿ„ Ž£'©` ¹¯©Ž¬5£¤Ž£¡‡ 9ª¯§5ž†³ª¯§5£¶;Ÿ„žw¦‰£…ª„¥ «(¬5¡‡¨`¬ ¡‡ z²=ž†ª‘¨Ÿ„žwž†³„¹2ŸN§ ´ ©r¬~¦E v¸„ž†ª„¾EŸNž†žw³5µ/±s¡w§‰¡w±ŸNž “ Ê ¾5¡†§‰ŸN¤r³ Ž£Ÿ„¤r¨`¬ª_¶„£¤v lª¯±s£(¤Ž£DŸ„ Žª„§‰Ÿ„¾5žw£!¤`ŸN§5¸¯£ ª„¥ ²j rŸ;³¥=¤Žª¯± e ©Žª e )) µ9¨Ÿ„§ ¢E§ ´ ©Ž¬5£±s¡†§5¡w±Ÿ Ð ¦5¡‡¨`·2žw³¯¹‘¸„¡†¶„£§©Ž¬‰£­5¤Ž£D¨'ª¯±­‰¦‘©rŸN©Ž¡†ª„§ªN¥ ©r¬5£±ŸN¤r¸„¡†§‰  “ * ©…¡‡ z«°ª„¤Ž©Ž¬§5ª„©Ž¡†§5¸ ŸN¸~ŸN¡†§$¬5£¤r£©Ž¬EŸ_©…«°£TŸ„¤Ž£§5ª„© tŽ¨`¬5£ŸN©Ž¡†§5¸ u ©Ž¬5£¾Cª„¦5§ ´ ¬5ª¯ž ´  °¥=ª„¤zŸNž†ž?¶_ŸNž†¦5£ !ª„¥  l¡†±T¦‰ž®©`ŸN§5£ª„¦‰ Žžw³ “ à ¬5£!¨'ª¯±­‰¦‘©rŸN©Ž¡†ª„§‰Ÿ„ž„©r¡w±s£°ªN¥‘©Ž¬‰£9¾Cª„¦5§ ´ ¨Ÿ„ž†¨¦5ž‡Ÿ_©Ž¡†ª„§T¡†  ´ ª¯±¡†§‰ŸN©Ž£ ´ ¾~³©r¬5£°¨ŸNž‡¨'¦5ž‡Ÿ_©r¡wª¯§ ª„¥©Ž¬5£±sŸ„¤Ž¸¯¡w§E z«(¬5¡‡¨`¬½¡‡   F n «(¬‰£¤r£ n ¡† ©Ž¬5£§2¦5±T¾C£¤ªN¥" Ž¦5­5­Cª„¤Ž©…¶¯£¨'©Žª„¤` («(¡w©Ž¬ Ÿ §5ª¯§5Ï£¤ŽªŸ„ r lª‘¨¡†ŸN©Ž£ ´ “và ¬5¡‡ (¨'ª¯( ±sH ­5¦‘©rŸN©Ž¡†ª„§‰Ÿ„žC©Ž¡†±£)¡‡ °©\³2­5¡†¨ŸNž†žw³ ´ ª„±s¡w§EŸ_©Ž£ ´ ¾2³©r¬5£©Ž¡†±£ ª„¥&©Ž¬‰6 £ % ( Å ž†£ŸN¤r§5¡†§5¸  ŸNž†¸„ª¯¤Ž¡w©Ž¬5± “ -

-

-



-


     !

1 test set errors test set bound margin bound

error rate

0.8 0.6 0.4 0.2

0 liver bostonpima ion sonar dig1 dig2 adult problem

7!œ &šG —> à ‰¬ ¡† °¢‰¸¯¦5¤r£) Ž¬5ª_«z °©r¬5£)¤Ž£D l¦5žw©r !ª„¥-Ÿ„­5­5ž†³2¡w§5¸ % ( Å ž†¡w¸¯¬¯©(©rª A ´ Ÿ_©rŸ¯ l£©r !«(¡w©Ž¬Ÿ R)ŸN¦‰ r Ž¡†Ÿ„§ ·¯£¤r§5£ž?Ÿ„§ ´ Ÿ IE NÈE©r¤rŸ„¡w§ D©Ž£D \©z Ž­5ž†¡®© “và ¬5£ª„¾‰ Ž£¤r¶„£ ´ ©r£ l©!£¤Ž¤rª„¤!¤`Ÿ_©r£¡‡ !¸„¤`ŸN­‰¬5£ ´ Ÿ„ (Ÿ„§ “ ̅§©Ž¬5£)©r£ l© l£©¹‘«°£¨Ÿ„ž†¨¦5ž†ŸN©Ž£)Ÿ #°¡w§5ª¯±s¡†Ÿ„žG¨'ª„§5¢ ´ £§E¨'£)¡w§~©r£¤r¶;Ÿ„žv²=­5¤rª„¾EŸN¾5¡†žw¡w©\³ªN¥ ¾Cª„¦5§ ´ ¥jŸ„¡wž†¦5¤r£  ) )e µ"«(¬5¡‡¨`¬¦‰­5­E£¤(¾Eª¯¦5§ ´  °©r¬5£©Ž¤r¦5£)£¤r¤Žª¯¤9¤`Ÿ_©r£ “ ̅§ ©r¬5£©Ž¤`ŸN¡†§5¡†§5¸ Ž£'© «9£¨ŸNž‡¨'¦5ž‡Ÿ_©r£…n©r¬5£ Æ Ê "3Ä #!Ÿ;³¯£ °±ŸN¤r¸„¡†§¾Cª„¦5§ ´ ¥=ª¯¤ŸN§ª„­‘©r¡w±s¡†Ï£ ´ ¨`¬‰ª„¡‡¨'£ªN¥ “ 





-

2 “ È “w’¯“ T5458z 43K Ê ­5­5ž†¡‡¨Ÿ_©r¡wª¯§Àª„¥v©Ž¬‰¡† ¾Cª„¦5§ ´ ©Žª$ l¦‰­5­Eª¯¤l©)¶„£D¨g©rª„¤±sŸ¯¨`¬5¡†§5£ …¡† ª„¥9 l¡†¸„§‰¡®¢5Ä ¨ŸN§~©v¡w±s­Cª„¤Ž©rŸ„§‰¨'£z¾C£¨ŸN¦‰ Ž£ % ( Å  °ŸN¤r£(¤Ž£DŸ„ Žª„§‰Ÿ„¾5žw³T£G€£¨g©r¡w¶¯£ŸN§ ´ Ÿ ´ Ÿ„­‘©rŸ„¾5žw£…¨'ž‡Ÿ„ r l¡w¢‰£¤` -¡†§ ¨ª„±s±sª„§$ŸN§ ´ «(¡ ´ £ Ž­5¤Ž£DŸ ´ ¦‰ l£ “ Ê % ( Å ž†£Ÿ„¤Ž§‰ !Ÿ·„£¤Ž§‰£ž†¡wÏ£ ´ ¨'ž‡Ÿ„ r l¡w¢‰£¤(Ÿ¯ !­E£¤(£ Ð ¦‰Ÿ_©r¡wª¯§ 2 “w’ “ Ó £ŸN­5­5ž†³©r¬5£ l¦5­‰­Eª¯¤l©(¶„£D¨g©rª„¤!±Ÿ„¨`¬5¡†§5£©Žª A M  * ´ Ÿ_©`ŸN¾‰Ÿ¯ l£­5¤rª„¾5ž†£± (¨`¬5ª~ l£§©Žª¢5© ©r¬5£ ¨'¤r¡w©Ž£¤r¡‡Ÿ tL©\«°ª½¨'ž‡Ÿ„ r l£D$  uŸN§ ´ tl¤r£ŸNž"¶;Ÿ„žw¦‰£ ´ ¡†§5­5¦‘©¥=£Ÿ_©r¦5¤r£v  u “à ¬5£­‰¤Žª¯¾5žw£± ¶_ŸN¤r³Á¡†§  Ž¡wÏ£)ª_¶„£¤(Ÿ„§ª¯¤ ´ £¤(ªN¥-±ŸN¸¯§5¡®©r¦ ´ £¥=¤rª„± e ©Žª e $
£»‘Ÿ„±s­5žw£D  “ * §¢‰¸¯¦5¤rf£ 2 “ .«°£¦‰ Ž£Ÿ -v IE _[ È E©r¤rŸ„¡w§ ;©Ž£D \©z Ž­5žw¡w©(ªN¥/©Ž¬5£ ´ Ÿ_©rŸ “ £ % ( Å ž†¡w¸¯¬¯©«(¡w©Ž¬ ! Ÿ R)Ÿ„¦‰ Ž Ž¡‡ŸN§ ·„£¤r§5£žzŸN§ ´ ©Ž¬‰£ ´ £'¥jŸ„¦5ž®© * § ŸNž†žz£'»‘­C£¤r¡w±s£§¯©` ¹°«°£¦‰ l/ ¾‰Ÿ„§ ´ «(¡ ´ ©Ž¬ “ Î(£D l¦‰ž®©` (¥=ª„¤ªN©Ž¬‰£¤¨`¬5ª¯¡†¨£ (ª„¥-©Ž¬5£ t` ug¹5¾‰ŸN§ ´ «(¡ ´ ©Ž¬G¹€ŸN§ ´ ;ª„¤(·¯£¤r§5£žGŸ„­5­C£ŸN¤ ©rª¾C£ Ð ¦‰ŸNž†¡w©rŸ_©r¡w¶¯£ž†³  Ž¡w±s¡†ž†Ÿ„¤)²cŸNžw©Ž¬5ª¯¦5¸„¬ªN¥¨'ª¯¦5¤` l£ ´ H¡ GC£¤Ž¡†§5¸ Ð ¦‰ŸN§~©r¡®©`Ÿ_©Ž¡†¶„£žw³5µ “    ­5¤r£¨'¡‡ Ž£ž†³ÑŸ * ©¡‡ s¡w±s­Cª„¤Ž©rŸN§~©s©Žª¼§5ª„©Ž£©Ž¬EŸ_©©Ž¬‰£ Æ Ê "$Ä #!Ÿ;³„£D T±ŸN¤r¸„¡†§ ¾Eª¯¦5§ ´ ¡†U ¾Cª„¦5§ ´ ²jª„¤)¨'ª„§5¢ ´ £§E¨'£s¡w§~©r£¤r¶;Ÿ„ž‡µª„§Á©Ž¬5£©r¤Ž¦5£s£¤Ž¤rª„¤…¤rŸN©Ž£ªN¥"©r¬5£žw£DŸN¤r§5£ ´ ¨'ž‡Ÿ„ r l¡w¢‰£¤ “ * §‘Ä  l©Ž£DŸ ´ ¹E¡®©¡† …Ÿs©Ž¤r¦5££¤Ž¤rª„¤(¤`Ÿ_©r£¾Eª¯¦5§ ´ ª¯§½ŸN§½Ÿ¯ Ž Žª‘¨'¡‡Ÿ_©Ž£ ´  \©rª2¨`¬EŸ„ l©Ž¡‡¨¨'ž‡Ÿ„ r l¡w¢‰£¤…¨`¬5ª¯ Ž£§ Žª Ÿ¯ z©rª¬‰Ÿ;¶¯£Ÿ l¡†±s¡wž‡ŸN¤…©Ž£D \©£¤r¤rª„¤¤`Ÿ_©r£ “à ¬5£D l£¾Eª¯¦5§ ´  )¨ŸN§½¾C£s¤Ž£¸¯ŸN¤ ´ £ ´ Ÿ„ …¾Eª¯¦5§ ´  …¥=ª„¤ ©r¬5£ª¯¤Ž¡†¸„¡†§‰ŸNžC¨'ž‡Ÿ„ r l¡w¢‰£¤"ª„§‰žw³s¦5§ ´ £¤!ŸN§ Ÿ ´5´ ¡w©Ž¡†ª„§‰Ÿ„žCŸ¯ Ž Ž¦5±s­‘©Ž¡†ª„ nXSl^`bjf;VXf;x ^`nXxgegbjV dZhDm(W@^`YLdc[_^`nXn i nXSl^`bjf ^ÇY\no^`WZWcV p_S\b e`u¿dch;S uwegbcm)Š  WZVXxgf    "!  #g}‰RTSvU;egf~k d&h_^`fDU;nXS-dZhDVXW&uwegbcm¿h;S\bjS'}




    !

 !     !



 



! , [ !

t

 

1 test set errors test set bound margin bound 7/10 margin

error rate

0.8 0.6 0.4 0.2

0 liver bostonpima ion sonar dig1 dig2 adult problem

7!œ &šG —> § Ÿ ´5´ ¡w©Ž¡†ª„§©Žª¨'ª¯±s­‰ŸN¤r¡w§‰¸«(¡w©Ž¬$£¶¯£¤r³~©Ž¬5¡†§5¸s¡w§¢‰¸¯¦5¤Ž£R2 “ .5¹‘«9£)¸„¤`ŸN­‰¬ ©r¬5£±ŸN¤r¸„¡†§  ¾Cª„¦5§ ´ «(¬‰£§%^  /ªN¥/©Ž¬5£ ´ Ÿ_©rŸ¡‡ !¦‰ Ž£ ´ ¥=ª„¤!©r¬5£)©Ž¤`ŸN¡†§ Ž£'© “ " ª„©Ž£)©Ž¬EŸ_©(¡®©z¡†±s­5¤rª_¶„£   Žª„±s£«(¬EŸ_©(ª„§©Ž¬5£±Ÿ„¤Ž¸¯¡w§¾Eª¯¦5§ ´ ¨Ÿ„ž†¨¦5ž‡Ÿ_©Ž£ ´ ¦‰ Ž¡†§5¸s©Ž¬5£ IE
©r¤rŸ„¡w§ Ž£'©² I ’ E±ŸN¤r¸„¡†§ ¾Eª¯¦5§ ´ µ'¹5¾5¦‘©z§‰ªN©z£§‰ª„¦5¸¯¬©Žª¨ª„±s­C£'©Ž£«(¡w©Ž¬©r¬5£)©Ž£D \© Ž£'©(¾Cª„¦5§ ´ “ 


 

*

¦5§‰§5£¨£ r ŽŸ„¤Ž³©rª£¶¯£§s¬‰Ÿ;¶¯£zŸ)©Ž£D \©9 Ž£'© “ * §s¢‰¸„¦5¤r£:2 “ È)«°£¨'ª¯±­EŸN¤r£9©r¬5£z©r¤Ž¦‰£(£¤r¤Žª¯¤¾Eª¯¦5§ ´   ª¯§¿ŸNž†ž!ªN¥z©r¬5£ ´ Ÿ_©rŸ½©ŽªÀ©Ž¬5£©Ž¤r¦5££¤r¤Žª¯¤T¾Cª„¦5§ ´  T¸¯£§5£¤rŸN©Ž£ ´ ¥=¤Žª¯± ©Ž¬5£/I E _È[E©Ž¤`ŸN¡†§ D©r£ l©  Ž­5ž†¡®© “ à ¬5£9¤Ž£D l¦5žw©r / Ž¬5ª_« ©Ž¬‰ŸN©&©Ž¬‰£ Æ Ê "$Ä #°Ÿ;³¯£ ?±Ÿ„¤Ž¸¯¡w§¾Cª„¦5§ ´ ¡‡ G©r¡w¸¯¬¯©/£§5ª¯¦5¸„¬)©rª¸¯¡w¶¯£v¦‰ Ž£'¥=¦5ž ¡†§‘¥=ª„¤r±Ÿ_©r¡wª¯§G¹?¾5¦‘© l©Ž¡†žwž§5ª„©¨'ª„±s­C£'©r¡®©r¡w¶¯£«(¡w©Ž¬À©Ž¬‰£©r£ l© l£©)¾Cª„¦‰§ ´   “à ¬5¡‡ )¡† )¡†§¼ \©r¤Žª¯§5¸ ¨ª„§~©Ž¤`Ÿ„ l©…«(¡w©Ž¬¼Ÿ©r¤rŸ ´ ¡w©Ž¡†ª„§Áª„¥ Ð ¦‰Ÿ„§¯©r¡®©`Ÿ_©r¡w¶¯£ž†³¡w±s­5¤`Ÿ„¨'©Ž¡‡¨ŸNž-±ŸN¤r¸„¡†§½¾Cª„¦5§ ´   “à ¬5£¤r£sŸN¤r£  Ž£¶¯£¤`ŸNž2¦‰ Ž£ Ÿ;¶_ŸN¡†ž†Ÿ„¾5ž†£9¥=ª¯¤¾Eª¯¦5§ ´  v«(¬5¡†¨`¬s­5¤rª_¶2¡ ´ £! Žª„±s£!¡†§‘¥=ª„¤r±Ÿ_©Ž¡†ª„§¾5¦‘©9«(¬5¡†¨`¬Ÿ„¤Ž£!§5ª„© ¥=¦5ž†ž†³©Ž¡†¸„¬~© “ ² ’ µ à ¬5£³±s¡w¸¯¬¯©(¾C£¨'ª¯±T¾5¡†§5£ ´ «(¡w©Ž¬Ÿ©Ž¤`ŸN¡†§ D©r£ l©°¾Cª„¦‰§ ´ ? ’ 0 B “ ² .„µ à ¬5£©Ž¤`ŸN¡†§Á l£©…¾Cª„¦5§ ´ ±s¡w¸¯¬~©…£Ÿ¯ l¡†ž†³¾C£¨ª„±s£©r¡w¸¯¬~©Ž£¤…¥=ª„¤) Ž±sŸ„žwž†£¤ ŽŸ„±s­5žw£ l¡†Ï£D  “ à ¬5¡‡ («°Ÿ¯ °ª„¾‰ Ž£¤r¶„£ ´ ¡†§ ? ’ 0B “ ²cȯµ à ¬5£À©Ž¤`ŸN¡†§Ò Ž£'©¾Cª„¦‰§ ´ ±s¡†¸„¬~© l©Ž¡†ž†ž¬EŸ;¶„£½©r¬5£ ¤Ž¡†¸„¬~s © tŽ Ž¬‰ŸN­C£ u½¥=ª¯¤¨`¬5ª2ª~ l¡†§5¸{Ÿ„§ ª„­‘©r¡w±Ÿ„žG­‰ŸN¤`ŸN±s£©Ž£¤( Ž£'©Ž©Ž¡†§5¸‰¹5 Ž¦‰¨`¬$Ÿ„  t` u¡†§$Ÿ %2¦‰­5­Eª¯¤l'© (£D¨g©rª„¤ Å Ÿ¯¨`¬5¡w§‰£ “ 4 “ 9 < ˜ 85—> œ=˜%• ˜%šG•G™ à ¬5£ ŽŸ„±­‰žw£¨ª„±s­5¤r£ r l¡†ª„§$¾Cª„¦5§ ´ ? ’ 2BZT¹ ?F2B/¡‡ zž†¡†·„£©r¬5£ Æ Ê "$Ä #°Ÿ;³¯£ !¾Cª„¦5§ ´ ¡w§$©r¬‰Ÿ_©¡®© Ÿ„­5­5ž†¡w£D (©ŽªŸN¤r¾5¡w©Ž¤`ŸN¤r³ ­5¤r£¨¡† Ž¡wª¯§½¨'ª¯§¯©r¡w§2¦5ª¯¦‰ z¶_Ÿ„žw¦5£ ´ ¨'ž‡Ÿ„ r Ž¡®¢‰£¤r  “ M §‰žw¡†·„£T©Ž¬5£ Æ Ê "$Ä #!Ÿ;³„£D  ¾Cª„¦5§ ´ ¹v¡w©Ÿ„­5­5ž†¡w£D ±£DŸN§5¡†§5¸„¥=¦5žwž†³¼©Žª §5ª¯§‰ \©rª‘¨`¬‰Ÿ„ l©Ž¡‡¨ ¨ž†Ÿ¯ Ž Ž¡®¢E£¤`  “ËÅ Ÿ„¡w§‰ l©Ž¤r£Ÿ„± žw£DŸN¤r§5¡†§5¸ Ÿ„žw¸¯ª„¤r¡®©r¬5±  ´ ªz§5ª„©&ª„­‘©r¡w±s¡†Ï£©r¬5£" rŸN±s­5ž†£"¨'ª¯±s­5¤Ž£D Ž Ž¡†ª„§±£©Ž¤r¡†¨„¹_ lª!©r¬5£"¾Cª„¦5§ ´ ŸN­5­5ž†¡‡¨Ÿ_©r¡wª¯§ ¡‡ G Žª„±s£«(¬‰ŸN©G¤rŸ„¤Ž£¤ “ " ª„§5£©Ž¬5£žw£D Ž ¹©Ž¬5£¤Ž£ ´ ª(£»‘¡† l©% Žª„±s£¤Ž£DŸ„ Žª„§‰Ÿ„¾5ž†³z¨'ª„±s­C£'©r¡®©r¡w¶¯£vžw£DŸN¤r§5¡†§5¸ Ÿ„žw¸¯ª„¤r¡®©r¬5± "¥=ª¯¤(«(¬5¡‡¨`¬ ©r¬5£ ŽŸ„±s­5žw£¨ª„±s­5¤r£ r l¡†ª„§¾Eª¯¦5§ ´ ­5¤rª ´ ¦‰¨'£D ( l¡†¸„§5¡w¢E¨ŸN§~©(¤Ž£D l¦5žw©r  “ à ¬5£ Ž£¨'©Ž¡†ª„§¡† !ª¯¤Ž¸~ŸN§5¡†Ï£ ´ Ÿ„ °¥=ª¯žwž†ª_«z ,+ ²’l µ %2¦5¾‰ Ž£¨'©Ž¡†ª„5 § 4 “†’  \©`Ÿ_©Ž£D zŸN§ ´ ­5¤rª_¶„£D 9©Ž¬5£ rŸN±s­5ž†£¨'ª„±s­5¤r£ r Ž¡wª¯§ ¾Cª„¦5§ ´ “



 




     !

Sample Compression Bound

δ Verifier

Learner Learning Algorithm A

Draw Training Examples

m examples Choose Subset S’, c=A(S’)

Subset S’ Evaluate Bound For c=A(S’)

7!œ &šG — >
à ¬5£½¡†§~©Ž£¤`Ÿ„¨'©Ž¡†¶„£$­5¤rª2ªN¥)ªN¥žw£DŸN¤r§5¡†§5¸¼¥=ª„¤©r¬5£Á rŸN±s­5ž†£ '¨ ª¯±­‰¤Ž£D Ž Ž¡wª¯§¾Cª„¦‰§ ´ “ " ªN©r£)©Ž¬‰ŸN©z©Ž¬5£Tžw£DŸN¤r§5¡w§‰¸sŸ„žw¸¯ª„¤r¡®©r¬5± ¡‡ zŸN¤r¾5¡w©Ž¤`ŸN¤r³ ¬5£¤r£„¹E l¡†±s¡wž‡ŸN¤°©Žªs©r¬5£)©Ž£D \© Ž£'©(¾Cª„¦5§ ´ “ 


 

² .„µl%2¦5¾‰ Ž£¨'©Ž¡†ª„§l4 “ . Ž¬5ª_«z ©r¬‰Ÿ_©°©Ž¬5£ rŸN±s­5ž†£…¨ª„±s­5¤r£ r l¡†ª„§¾Cª„¦5§ ´ ¡‡ 9§‰£ŸN¤rž†³Ÿ¯ v©r¡w¸¯¬¯© Ÿ„ !­Cª¯ r l¡†¾5ž†£)¸„¡†¶„£§©Ž¬‰£ª„¾‰ Ž£¤r¶_Ÿ_©Ž¡†ª„§E  “ ²cȯl µ %2¦5¾‰ Ž£¨'©Ž¡†ª„§ 4 “ È ´ ¡‡ Ž¨¦‰ r l£D T¤r£ Ž¦5žw©r T¥=¤Žª¯± ©r¬5£$ŸN­5­5ž†¡‡¨Ÿ_©r¡wª¯§ÑªN¥©Ž¬5£$ ŽŸ„±s­5žw£¨'ª¯±Ä ­5¤r£ r l¡†ª„§¾Eª¯¦5§ ´ ©Žª Ž¦5­5­Cª„¤Ž©(¶„£¨'©Žª¯¤(±sŸ¯¨`¬5¡†§5£  “ 4 “w’¯“ W # L !w"XW  2%N, W M,MN2U1J 2STUN/ Zà ¬‰£ %‘Ÿ„±s­5žw£z9ª„±s­5¤r£ r Ž¡wª¯§¾Eª¯¦5§ ´  \©`Ÿ_©Ž£ ´ ¬5£¤Ž£¡‡  ´ ¡ G€£¤` T Žª„±s£«(¬‰Ÿ_©¥=¤rª„± ªN©r¬5£¤T¤Ž£D l¦5žw©r v² ? ’ 2B3?F2CB(ŸN§ ´ ª„©Ž¬5£¤r `µ¾2³ ¸¯£§5£¤rŸ„žw¡†ÏŸN©Ž¡†ª„§ Ÿ„§ ´  Ž¡†±­‰žw¡w¢E¨ŸN©Ž¡†ª„§¾5¦‘©(©r¬5£¾Eª¯¦5§ ´ ¾C£¬‰Ÿ;¶2¡†ª„¤!¡†  Ð ¦EŸNž†¡®©`Ÿ_©Ž¡†¶„£žw³ ¡ ´ £§~©Ž¡‡¨ŸNž “ %2¦5­‰­Eª~ l£s«°£¬‰Ÿ;¶¯£Ÿž†£ŸN¤r§5¡†§5¸$ŸNž†¸„ª„¤r¡w©Ž¬5±$¹ m «(¬5ª¯ Ž£©Ž¤`ŸN¡†§5¡w§‰¸¡‡)  tl Ž­‰ŸN¤` Ž£ u ¡w§À©Ž¬5£  Ž£§‰ Ž£9©Ž¬‰ŸN©©Ž¬5£(ª¯¦‘©Ž­5¦5©"¨'ž‡Ÿ„ r l¡w¢‰£¤/¡‡  ´ £­C£§ ´ £§~©v¦5F ­Cª„I §ª„§5ž†³TŸ) Ž¦5¾‰ Ž£'©ªN¥C©Ž¬5£ ´ ŸN©rŸ‰¹ m ` F ¤rŸNI ©Ž£„¹ m  ¥=ª¯¤ m  m “…à ¬5£ ŽŸ„±­‰žw£¨'ª„±s­5¤r£ r Ž¡wª¯§¾Cª„¦‰§ ´ ¡‡  ´ £­E£§ ´ £§¯©ª„§©r¬5£T£¤Ž¤rª„¤ F I © m d m  “à ¬5£±sª„©Ž¡†¶;ŸN©Ž¡†ª„§ ¬5£¤r£¡† …©Ž¬EŸ_©)©Ž¬5£s£»‘Ÿ„±s­5žw£D «(¬5¡†¨`¬Á©Ž¬5£ q ` b b  ª„§Á©Ž¬5£ Ž¦5¾‰ Ž£'R ž†£Ÿ„ ¤Ž§5¡†§5¸sŸNž†¸„ª¯¤Ž¡w©Ž¬5± ´ ª2£   ´ £­C£§ ´ ¦‰­Eª¯§½ŸN¤r)£ tŽŸNž†±sª¯ lv© u…¡†§ ´ £­E£§ ´ £§¯©ŸN§ ´  Žª«°£¨Ÿ„§ tŽŸNž†±sª¯ lv© u…¸¯£'©zŸ©Ž£ l© Ž£'©z¾Cª„¦5§ ´ “ (¡†£«°£ ´ Ÿ„ Ÿ„§T¡†§~©Ž£¤`Ÿ„¨'©Ž¡†¶„£°­5¤rª~ª„¥Eª„¥Ež†£ŸN¤r§5¡†§5¸s²=¡†§T¢‰¸¯¦5¤rK £ 4 “w’ µg¹_©r¬5£z ŽŸ„±s­5žw£!¨'ª¯±s­5¤Ž£D Ž Ž¡†ª„§ ¾Cª„¦5§ ´ ¡‡ ¦5§5¡ Ð ¦5£zŸN±sª¯§5¸¯ l©%©r¤rŸ„¡w§‰¡w§5¸ Ž£'©¾Cª„¦5§ ´  ¾E£D¨Ÿ„¦‰ l£!¡w© ´ ª2£ -§5ª„©v¤Ž£ Ð ¦5¡w¤r£ ^  ¡†§5¡w©Ž¡‡ŸNž ¨ª„±s±s¡®©r±£§~©°©rªŸ±£DŸ„ Ž¦5¤r£ª_¶¯£¤9©Ž¬5£¨ž†Ÿ¯ Ž Ž¡w¢‰£¤`  “ 











W02

, 0W 

b E; 7