Bayesian Example Selection Using BaBiES - ScholarlyCommons

2 downloads 0 Views 247KB Size Report
Active learning is widely used to select which examples from a pool should be ... We present the BaBiES criterion, an approximation of Bayesian A-optimal ...
University of Pennsylvania

ScholarlyCommons Technical Reports (CIS)

Department of Computer & Information Science

January 2004

Bayesian Example Selection Using BaBiES Andrew I. Schein University of Pennsylvania

S. Ted Sandler University of Pennsylvania

Lyle H. Ungar University of Pennsylvania, [email protected]

Follow this and additional works at: http://repository.upenn.edu/cis_reports Recommended Citation Andrew I. Schein, S. Ted Sandler, and Lyle H. Ungar, "Bayesian Example Selection Using BaBiES", . January 2004.

University of Pennsylvania Department of Computer and Information Science Technical Report No. MS-CIS-04-08. This paper is posted at ScholarlyCommons. http://repository.upenn.edu/cis_reports/10 For more information, please contact [email protected].

Bayesian Example Selection Using BaBiES Abstract

Active learning is widely used to select which examples from a pool should be labeled to give best results when learning predictive models. It is, however, sometimes desirable to choose examples before any labeling or machine learning has occurred. The optimal experimental design literature has many theoretically attractive optimality criteria for example selection, but most are intractable when working with large numbers of predictive features. We present the BaBiES criterion, an approximation of Bayesian A-optimal design for linear regression using binary predictors, which is both simple and extremely fast. Empirical evaluations demonstrate that, in spite of selecting all examples prior to learning, BaBiES is competitive with standard active learning methods for a variety of document classification tasks. Comments

University of Pennsylvania Department of Computer and Information Science Technical Report No. MSCIS-04-08.

This technical report is available at ScholarlyCommons: http://repository.upenn.edu/cis_reports/10

      ! #"$ $%&'( )* +,.-./1032541687:9=?,A@B7C6BDA0>-7FEG,.- H?03/EI,.-KJ8L:H?0NMO6P, QRES/ TVU WYX[Z]\]^_U `3\a[bCcAad^_WGeS\fU ZXd`Igihj`SbkadZf^lX1\]mnad`oSp mqUr`Ip)U sutIU v`GmnwdUrZfx]my\{zladb|Ur`G`Ix]z>}qw1X[`ImnX ~FU w>mn`GU'X[}n} €€d€‚„ƒ X[}n`>eS\Vo3\]ZfU U \ |81‘YtG’mn}n“.X”–gS•RU “d}nWG‘[tG—Gm…˜GXG™3†Gš|ˆ›C‡Š”{œ>‰r—>‹IS‰ ‘d‚d›Œ[ž.jŽdŸd€d>›G‹ ‘3˜R’1šd—S• ¢¡>’3“ £œ>¤Gš[—>—u jš3˜dœ ¥§¦V¨©SªG«B¬R© x–\]Z¢X1\fU ¯mqU®xprX[`imq^_WGZfa1wdU°^lXdp¢tImq`GU°}nUrXdZ]`Gmn`G¯_Xdprp)eGZ¢Xdp zd¸ ‡p­\fmqwU}qU®X[Zf`Gmn`G¯°mnxB±m…gSUr}qz!eIx–U®g \fa x–Ur}qU®p­\C±tGm…p¢t ‡Vp­\fmqwU}nUrX[Zf`Gmn`G¯ X[WIWGZ]a3Xdp¢tGU®xÈ\]a*\]tGU¹WYa>ad}´IXdx]UrgÃx]U)\  U)²GX[^_WG}nUrx_bkZfad^³XOWYa>a}x–tIadeG}…g§´YUµ}nXd´YUr}qU®g¶\fa f \ mq`I"¯ !£cAadtG`ÇU)\ÀXd} ¸n†‰®‹d‹ $Ž # ~FU ±m…x %'&°X[}nUd†V‰r‹‹ $Œ # o>UreG`G¯ ¯dmnwdU´RUrx–\§Z]U®x–eG}q\fx·±tGUr`}qU®X[Zf`Gmn`G¯*WGZfUrgGmnp)\]mnwdU U \8X[}õ¸q†I‰r‹d‹ ( ):\fXdÂdUuX\fZfXdmq`IUrg!^_Xp¢tGmn`GUu}qU®X[Zf`Gmq`I¯ÍXd}q¯adZfmy\ftG^ ^aSgSUr}nxr¸$h£\„m…x † tGa1±U wdUrZr† x–a^_U)\]mn^_Urx gGUrx]mqZ¢X[´G}nU d X `Ig§WGm…p¢Â¶\ftGUµ`IU)²>\U ²SXd^_WG}qUNbkZ]a^ \ftGUµWRa>ad}bkaZ}…X[´RU }  \]a¹p¢tIa3a3x–UºU ²GX[^_WG}nUrx»´YU bkadZfU¶X[`>z¼}nXd´YUr}qmn`G¯aZ n m G ` ¯ºXdprp)aZfgSmn`G¯µ\]aOX»^_U®Xdx]eGZ]Uia[bÍU)²SWRUrp­\fUrg·´YUr`GU)ÉI\r¸Çhj`  ^_Xp¢tGmn`GUº}qU®X[Zf`Gmn`G¯KtIXdxNa>prp)eGZfZfUrgˆ¸½sutGU¾adWG\]m  f \ U f Z }qU®Xmq`G¯i^lXdp¢tGmn`GU_}nUrX[Zf`Gmn`G¯i±my\ftÈ}…X[´RU }nmn`G¯$mnx°mn`3\]U `YgSUrg ^_Xd}¿U)²SWYUrZ]mn^_U `3\fXd}:gSUrx]mn¯d`}nmq\]U Z¢X1\feGZfUVtIXx^_Xd`>z f \ » a WIZ]a1w>m…gSU´RU)\]\]U Z_^_UrXx–eIZ]U®x„adbU)²SWRUrp­\fUrg¶´YUr`GU)ÉG\Àx]mq`Yp)U \]tGUradZfU)\]m…p Xd}q}nz'X1\]\]Z¢Xdp­\fmqwU8aWS\]mn^lX[}nmy\{z p Z]mq\]UrZ]m…XbkaZ _ ^  a Z]UÍmq`GbkadZf^_X[\]mnad`imnxuX`Ga1±`·Xdx ±tGU `Á±adZfÂ3mn`G¯¼±my\ftÃ}nXdZ]¯U·`>eG^!´YUrZfx¾a[b$WGZ]U  Õ  Ô ? â n ã ßÀÙ1ÖRÛ¢ÞdÔ +Û *]ãnßÛ¢ä¿âõÙ[ֈÜ>Û­Ú]-ã ,.ä !0/IUrgSaZ]a1w¿†S‰r‹ 1( # cAtIX[}nad`IU Z gSmnp)\]mnwdU§bkU®X1\]eIZ]U®x ¸ ƒ UÄWIZ]U®x–Ur`\º\ftGUÅuXdÅm…ÆAo %32BU Z¢gSmq`IU }n}qmõ†B‰r‹ ‹ 4 )ubka>p eIx]UrxÍad`\ftGUZ]Ur}nX[\]UrgµWGZfad´G}nU ^ a[b p)Zfmy\fU Zfmqa`:†X[`ÄX[WIWGZ]aeG^„´RU ZladbÍ}…X[´RU }nUrg ^lX[\–\]mn`G¯Èx]p¢tGUr^_Urx„gGU ^_ad`Ix–\]Z¢X1\fUl\]tYX1\9X !k`Ga`Ix–U®å3eGU `3\]m…X[:} ) U ²GX[^_WG}nUrx{æ\]t>eIxÍZfUrgSeYp)mn`G¯i\]tGUÀp ax–\°adb´IeGmq}…gSmn`G¯^lXdp¢tGmn`GU ÅuXm…gSUrx:X´IXdx]U }nmq`IUBbkadZF\]Urx–\]mn`G¯±tIU)\]tIU Z ´GmnXx–U®g'x]Xd^_WG}qmn`G¯ WI}qmn`G¯l^_U)\ftGaSgGxua[bBx–Ur}qU®p­\fmq`G¯_U ²SXd^_WG}qU®x ¸ éûríê3ë  ìí÷níörîyð{ï®þ ëfð{û ñjò{îóòõîúörôíör ÷A¢ø¿îúë¢ë¢í 큢í3ëñ£ôdéYùúë ï®]û êí3í3îyî û­û üuùë¢ Aý3ëfû ýSðjörò{þ'ðjòë¢í1òö ö ÷Aÿ ö®õÿþ'ý dò{ëfð öòC; ð{õí ë¢ýò{ùyê3û¢îú=ñ ë¢þ'< örëfí[?ð > ò+A[ûrð{ùúë ùIB®ñ{û ûrðCþ'dùyýëfùyñjëfñAñ ö ÷n÷:ð{örñ{þ ûrþ'ò{ý3ê3ùúëîúíýSBVö1ñ£öròjùIð–û û ð{ò{ëë B ò–ôû  >aw} x5mq`3\faÈgSm…x ] admn`\lx]U)\fx fzyl†C\ftGU\fZfXdmq`Imq`G¯Ox]U)\ Xd`I|g {N†S\]tGU°ZfUrx]mngGeIX[}ˆx]U)\\]a_^lX1²Smn^_-m erUV^lXdp¢tImq`GU°}nUrXdZ]`Gmn`G¯ ^_aSgSUr}WYUrZ–bkaZ]^lX[`Yp)Ud}¸ /Ga}q}na1±mq`I¯µ\ftGU`Ga[\¢X1\fmqa`·X[`Yg·gSU  D E

p mnx]mnad`O\]tIU adZfzÈWGZ]U®x–Ur`3\fX1\fmqa`Oa[b@!õcAtIXd}qa`GU Z~%2BU Z¢gSmq`IU }n}qmõ† ‰®‹d‹4 )µ±AUKgSU ÉI`GUKX¼eS\]mn}qmq\{z bkeI`Ip­\fmqa`3€!0ƒ‚C„…‚†y‚ L )»bkadZ ¯3X[eG¯mq`I¯^_aSgSU }8WYUrZ–bkaZ]^lX[`Yp)U_X1bà\]UrZ x–Ur}qU®p­\fmq`G¯NX\fZfXdmq`Gmn`G¯ x]U)d\ yÄadbSÉG²SUrg!x–Um e U¸.sutGUAwdUrp)\]aZ „ip)adZfZfUrx]WYa`IgGx¿\fa\ftGUAWYX  Z¢X[^_U \]U Z¢xCadb.X'\fZfXdmq`IUrgl^_aSgSU }õ† ‡_mnxuX wUrp)\]adZAa[b:ZfUrx]WYa`Ix]Urx bkaZl\]tIU\]Z¢X[mn`Gmn`G¯ºx–U \r†X[`Y}g ¶gSU `Gad\]U®xXOgSUrp mnx]mnad`Ç\]a¾´RU ^lXgSUX1bà\fU ZA\]tGUV^_a>gGU }Im…x8\]Z¢X[mn`GUrgˆ¸sutGUÍx–Um e Uadbˆ\]tGU\]Z¢X[mn`  mn`G¯x–U ‰\ ˆ ysˆ[m…xC`Ia[\A¯mqwU `ÀX x]z>^„´Rad}¿Xdx8my\mnxXdxfx–eI^U®gÉG²SUrg \ftGZfadeG¯tGadeS\\]tGm…xU ²SWYa3x–mq\]mnad`:¸ÍsutGU!ÅuXWRUrp)\fX[\]mnad`Ix!mnx„\]tYX1\ \ftGU_WGZfmqaZÍa1wU ZÍWIX[Z¢X[^_U \]U Z¢xÍX[}n}qa1±xbkadZ'X[`ÈUrx–\]mn^lX1\fmqa`µa[b ZfUrx]WRad`Ix]U°wdUrp)\]aZ L ±tImnp¢t$mn`\feGZf`$Xd}q}na1±xeIxu\faÀp)ad^_WGeG\]U U ²SWYU®p­\]U®gieGWRgIX1\]U®xubkadZ „8¸ cVeIZ'¯daXd}mnx°\fagSU wU }nadWÈU)²GX[^_WG}nUlx–Ur}qU®p­\]mnad`ºxfp¢tGU ^_U®xÍbkadZ aWS\]mn^lX[}ˆžÔ *)Û®Üdã árâ ãàÕ1ä»Ù>árá Z*­Ù>Cá T$adb8XÅuXeG^„´RU Za[bWGZ]U®gSm…p­\]aZfxr¸Chj`$WGmn}qad\x{\feIgSmnUrx±U bkaeG`IgÀ\]tIX[\´>|z ( ‚‚d‚ WIZ]U®gSmnp)\]aZfxr†d\ftGU°mq`>wU Z¢x–U±uXdx8\]a>aU)²  WRU `Yx–mnwdU„\]ap ad^_WGeS\fU„a`µX|Ur`\fmqeI^ h]h–hp ad^_WGeS\fU ZͱtGU ` Ur^_WG}qa1zUrgmn`Ix–m…gSUX°¯Z]UrUrgSz!U)²GX[^_WG}nUx]U }nUrp­\fmqa`lX[}n¯dadZfmq\]tG^$¸ sutIU'`3eI^„´RU Z¢xAadb WIZ]U®gSmnp)\]aZfxeIx]Urgmq`$aeGZU ^_WGmnZ]m…p Xd}ˆU w1X[}  eYX1\]mnad`iZ¢X[`G¯UbkZfad^ €dŒŽdŽ \]|a 14 Œ€ †3Xd`IgÀm…gSUrXd}q}nzÀ±U°±AaeG}ng }nmnÂdUV\]a±AaZ]Â_ad`igSad^lXdmq`IxA±mq\]tiU wdUr`¯Z]U®X1\]UrZ8`>eG^!´YUrZfxa[b WIZ]U®gSmnp)\]aZfxr¸ ƒ UµWIZ]U®x–Ur`\iX¾tGUreGZ]m…x–\]m…pµX[WIWGZ]az>^bkadZ f ù E zUrx]mnXdÝ ` ù = `IXdZ][z ú²GX[^_WG}nU 7 U }nUrp)\]mnad`:¸ sutIUOÅuXÅAm…ÆAo XdWGWGZfamn`G¯lad`i\ftGU'\]tIU `>eG^„´RU Zadb\]mn^U®xubkUrX[\]eGZfUrø xË Xd`IÄg ª ûip)a  aSp p eGZ®†IX[`Ig\]tGUgSmnXd¯da`IX[}:w1Xd}qeGU®x !0¿ ª ¿­) ÊUÊ p)a`  x]m…x{\fmq`G¯adbA\ftGU_\]ad\fXd}C`>eG^„´RU Z'adbA\fmq^_Urx bkUrX[\]eGZfŠU ËaSp p eGZfxr¸ ƒ U_WGZfadWRax]U!X[WIWGZ]amn`G¯mq\bkZ]a^ \]tGUuWRa3a} /10 þ!k±ý./1mq\], tIady eS\eGW¿gGX[\]mn`G¯_\]tGU p adeG`3\¢x  d)­¸

 Béê ë Að{ë¢+ë [ô  û  î ð A ù B®ö ð{îúò{êþ gGmnXd¯dad`YX[}õ†A}…Xdx–\ÀZ]a1±'†Xd`IgÇ}nXx{\p ad}neG^_`:¸ sutGUµXdWGWGZfaad}ˆp)aeG`3\fx   Xd`Igl\fZfXdmq`Imq`G¯x–U \8bkU®X1\]eIZ]U°p)aeG`3\fx   gSU  xfp)Zfmn´Gmq`I¯\ftGU_`3eI^„´RU Z'a[bA\]mn^_Urx'XWGZ]U®gSm…p­\]a!Z ŠaSprp)eGZ¢xÍmq` \ftGUÀWRa>ad}AXd`Ig»\]Z¢X[mn`Gmn`G¯Nx–U \„ZfUrx]WYU®p­\]mnwdUr}qz¸sutGUZ]U®x–eG}q\]mn`G¯ a´ ] Urp)\]mnwdUÍbkeG`Ip)\]mnad`i\]a_^_mn`Gmq^_Um e U m…x f ÅuXdÅm…ÆAžo !0y~)Ȍ Î  #s"À  ‚ !{¡‰ ( ) $%'( &

±tIU ZfU \]tIUx]eG^) m…xVa1wdU ZVWGZfUrgSm…p­\fadZ¢x!+*0°-,°„±AaZfgi\fadÂdUr`Ix mn`OXigGa>p eG^_U `3\ p)}…Xdxfx–mqÉYprX1\]mnad`»gSad^lX[mn`ƒ)VZ¢X1\]tIU ZÍ\]tIXd`»a´  x]U Zfw1X1\fmqa`Ix ¸Ov`I}qmnÂdU\]tGU‡  aWS\]mn^lX[}nmy\{zOad´ ] Urp­\fmqwU ½ †C\ftGU ÅuXÅAm…ÆAoÇeG\]mn}qmq\{zKp X[`´YUÈp)ad^_WGeG\]Urg¹å3eGmnp¢Â>}nzd†^lX[Â>mn`G¯¶mq\ bkU®Xdx]mq´I}qUl\faµX[WIWG}qzOX¯Z]UrUrgSzÈX[}n¯dadZfmq\]tG^$¸sutGUXd}q¯adZfmy\ftG^ ±UNeIx]Uµmn`ÄUrw\]ZfU ^_U }nzNw1X[}neIX[´I}qU_mqb8\]tIU zµWIZ]a1wU„WGZfUrgSm…p­\fmqwU_a[b8\ftGUlZfU  x]WRad`Ix]Uw1X[ZfmnXd´G}qU¸¹o>U®p)ad`Ygˆ†A±UtGaWYU\faº}qU®X[Zf`ÇXp p eGZfX[\]U WYX[Z¢X[^_U)\fU Z¢x_bkadZ$Xdx^_Xd`>zÇWIZ]U®gSmnp)\]aZfxÀXxWRaxfx]mq´G}nUµx]mq`Yp)U \ftGm…xB±mq}n}Smn`Ip)ZfUrXx–UA\]tGUWIZ]a´IX[´Gmn}nmy\{z adbYtIXmn`G¯Íad`GUuadZB^_aZ]U WImqU®p)Urxa[bBU w>m…gSU `Yp)U'\faÀeIx]U mq`µ^_XdÂ>mq`G¯ÀXÀWGZ]U®gSm…p­\]mnad`$bkaZX \fUrx–\_x]U)\_ad´Yx–UrZ]w1X1\fmqa`:¸»sutIUiÅuXdÅm…ÆAoOp Z]mq\]UrZ]mnad`¶prX[WS\feGZfUrx ´Ra[\ftKa[b°\]tIUrx]UºÜ>Û­Ú]ã Ü> Û *­ÙSpâ Z[/ß .$\]tGUµWGZ]mnadZ ¢ gSU)\fU Zf^_mq`GU®x \ftGU´RU `IU)ÉG\a[bx]U U mn`G¯NXWIZ]U®gSmnp)\]aZ bkadZ„\]tGU¬ £ ÉIZ¢x{\ \fmq^_Umq` \ftGU$\]Z¢X[mn`Gmn`G¯ºx–U \r†\]tIU`3eI^UrZfX[\]aZ!\fU Zf^ ¯mqwUrx_tGmn¯dtÇbkZfU  å3eGUr`Ip)zWGZfUrgSm…p­\fadZ¢x¯dZfUrX[\]U ZBWIZ]mnadZfmy\{z†d±tGmn}nU\]tGUbkZfXp­\]mnad`YX[} Ur}qUr^_U `3\fx_ZfU WGZfUrx]U `3\_XÈgGmq^_mn`Gmnx]tGmn`G¯OZ]U \]eGZf`Ixmn`Çx]U Urmq`G¯OX WIZ]U®Ù gSmnp)\]aZ^aZ]UV\]tYX[`$ad`Ip Ud¸ éê ê3z¸ sutIU ƒ U eG^_U ZfmnprX[}Fx–\]Zfmq`I¯x ^lXdWGWYU®gÈ\]a$\ftGUx]WYU®p)m…X[}B\fadÂdUrá` ^†5 ¸ _‡p)aeG`3\„p)eS\  a 7¶a[b ÉYwdU°aSp p)eIZ]ZfU `Ip Urxm…xueIx–U®gˆ†Ix]a„\ftIX1\X±adZ¢g  \{z3WRUÍ^„eIx–\aSp  p eGZX1\i}qU®Xdx–.\ 4¾\]mn^_Urximq`\]tGUOgGX[\fXÇx–U \imn`aZfgSUrZ\fa·´RU mn`Ip }qeIgGUrgKmn`§\]tGUµwdaSp Xd´GeG}…X[Zfzd¸ o3\]aWK±AaZfgGxX[ZfU$ÉI}y\fU ZfUrg bkZfad^ \ftGUÇwdaSp Xd´GeG}…X[Zfz¹eIx]mn`G¯¹eYx–mn`G¯¹\]tGU§ÓX[mn`>´Ya1± x{\fa  WI}qm…x{Y\ !˜aNp®cXd}q}neG^$†°‰r‹‹ Ž ))¸Khj`Ç\]tGUNprXdx]UiadbV\ftGUÈcAa^WF¸ ; gIX1\fXÍx]U)\fxr†[Xdxfp)mnm  Ur`Ip)aSgSU®g'´Gmn`IX[Zfz gGX1\¢X±uXdx.^lX[`>eIXd}q}nz x–UrW  XdZfX[\]U®g_bkZ]a^Á\]tGU'XdZ–\fmnp }qU®x ¸Æ8Xp¢tgSaSp)eG^_Ur`\±uXdxAU `Ip aSgSUrg Xx_X»´Imq`IXdZ]z  w1X[}neGUrg¶wdUrp)\]aZ±tIU ZfUiU®Xdp¢t·wdUrp)\]aZ_p ad^_WRa  `IU `3\ÀU `Ip aSgSUrgDZtGU \]tGUrZXOx–WRUrp myÉYpN±AaZfg  \{z>WYUiaSp p eGZ]ZfUrg mn`i\]tGU gGa>p eG^_U `3\r¸ s.a´RU)\–\fU Z eG`IgSUrZfx–\fXd`IgV\]tGU8mq^_WIXp­\:\ftIX1\.gGa>p eG^_U `3\F}nU `G¯d\]t tYXdxa`NÅuXdÅmnÆo$X[`Igi\ftGU„ad\]tGUrZU ²SXd^_WG}qU!x–Ur}qU®p­\]mnad`NXd}q¯a  Zfmq\]tG^lxr†:±AUlp Z]U®X1\]U\fZ]eG`Yp X1\fUrgÈgGX1\¢Xx]U)\¢xÍbkadZ'U®Xdp¢tÈa[b8\ftGU gIX1\fXx–U \fx:}nm…x{\fUrg Xd´Ya1wUd¸FsutGUC\]ZfeG`Ip X[\]U®g x–U \fx.X[ZfUBWGZ]UrWIX[ZfUrg Xd`IX[}nad¯adeIx]}nzl\]a_\ftGU„eG`3\fZ]eG`Yp X1\fUrgiad`IUrx # a`G}qzigSaSp)eG^_Ur`\¢x XdZ]UVZ]U®x{\fZ]m…p­\fUrgÀ\]a!\{±AUr`\{z  ÉIwdU\]aÂdU `Yx8mn`}nU `I¯[\]­t !à\ftGUÍÉIZfx–\ ( 4!eG`Gm…åeIU'\]aÂdU `Y†x )uXd`Ig$X[`>zgSaSp)eI^Ur`3\±my\ftibkU ±U Z\ftIX[` 3

\{±U `3\{z  ÉIwdUleG`Imnå3eGU\]aÂdU `Yx m…x!gSm…xfp X[Z¢gSU®gˆ¸µsutGUWGeGZfWYa3x–U adb\]tGU°\fZ]eG`Yp X1\fUrggSaSp eG^_U `3\Vx]U)\fxmnxu\falbkeG}n}qz$Xdprp)aeG`3\bkadZ gGa>p eG^_U `3\.}nU `I¯[\]tp¢tIX[Z¢Xdp)\]UrZ]m…x{\fmnprxˆmn`'\]tGUU w1Xd}qeIX[\]mnad` x]mq`Yp)U gGa>p eG^_U `3\Àp)}…Xdxfx–mqÉYprX1\]mnad`·tIXdx\ftGU$WGZfadWRU Z]\{zO\]tIX[\lgG8m 7¿U Z  Ur`3\ ad´Ix]U Zfw1X1\]mnad`Yx !?gSaSp)eI^Ur`3\f x )°tIXeG^„´RU Z¢x adb„gSm…x{\fmq`Ip)\i±adZ¢gGxr¸ c\ftGU Z$gSad^lXdmq`IxigSa·`Ga[\$tIXYeGU `Ip  mn`G¯_WRU Z]bkadZf^_Xd`Ip)U°adb.U)²GXd^WI}qU x]U }nUrp)\]mnad`$x{\fZfX[\]U ¯mqU®x ¸ sutIU8p)}…Xdxfx–mqÉIUrZˆeIx–U®g°mq`'Urw1X[}neIX1\fmqa`'±Xx:XÅuXadK} JKXd`Igmq`\]tIUp Xx–Uua[bˆeG`Yp)U Z  \¢X[mn`3\{z_xfX[^_WG}nmn`G¯„X[`YgÀå3eGU Zfz  ´>z  p ad^_^_my\]\]UrUd†3ad`GUVadZ8^_aZ]U p }nXx]x]mqÉIU Z¢x'\fZfXdmq`GU®gObkZfad^½\]tGU$p)eIZ]ZfU `3\„\fZfXdmq`Imq`G¯Èx–U \r¸Ovx  mn`G¯\]tImnxumn`SbkadZf^lX1\fmqa`À\]a_mn`SbkadZf^mq\fxp¢tGamnp Ud†3\]tIU x–Ur}qU®p­\]mnad` Xd}q¯adZfmy\ftG^(m…xu\ftGU `NXd}q}na1±AU®g\]alZfUrå3eGUrx–\\]tYX1\VX_WIXdZ–\fmnp eG}…X[Z gGa>p eG^_U `3\ÍbkZ]a^(\ftGU!WRa>ad} ´YU}…X[´RU }nUrgNXd`IgNXdgIgSUrgN\]aÀ\ftGU \fZfXdmq`Imq`G¯lx]U)\®¸ 7 þr0 üML1 mybR\]tGU\fa[\fXd}S`>eG^„´RU ZBadbRgSaSp)eI^Ur`3\fxmn`„\ftGUu\]Z¢X[mn`   mn`G¯ix–U \Vm…x}nUrxfx\ftIX[`¶‰ ‚d1‚ N \]tGU!`3eI^„´RU ZÍa[bAp }nXx]x]Urxr†Yx–\]UrWIx ‰'Xd`I.g (!XdZ]UÍZfU WRUrX1\fUrgˆ¸

1

0.9

0.9

0.8

0.8

0.7

Accuracy

Accuracy

0.7

0.6

0.6

Random BaBiES Uncertainty Sampling Query by Committee Entire Pool Length

0.5 Random BaBiES Uncertainty Sampling Query by Committee Entire Pool

0.5

0.4

0.4

0.3

0.3 50

100

150

200

250 300 350 Number of Articles Labeled

400

450

500

50

1

100

150

200

250 300 350 Number of Articles Labeled

400

450

500

1

0.95

0.95

0.9 0.9

Accuracy

Accuracy

0.85 0.85

0.8

0.8

0.75 Random BaBiES Uncertainty Sampling Query by Committee Entire Pool

0.75

0.7

0.65

0.65

0.6 50

100

150

200 250 Number of Web Pages Labeled

300

350

400

50

0.98

1

0.96

0.98

0.94

0.96

0.92

0.94 Accuracy

Accuracy

Random BaBiES Uncertainty Sampling Query by Committee Entire Pool Length

0.7

0.9

0.88

100

150

200 250 Number of Web Pages Labeled

0.84

400

Random BaBiES Uncertainty Sampling Query by Committee Entire Pool Length

0.88 Random BaBiES Uncertainty Sampling Query by Committee Entire Pool

350

0.92

0.9

0.86

300

0.86

0.82

0.84 0

20

40

60

80 100 120 Number of Articles Labeled

140

160

180

Õ

200

0

20

40

60

80 100 120 Number of Articles Labeled

Õ

140

160

180

200

Sñj Kî ör OXf Pòjë¢ò{ñ\ö®BrþRîúïû ñ r1ùY.ò{öÄô'ð{ç[+ö Zû3þýS dò{ö1îúö®0Sörùˆíë]ð°û îyí í ör ÷ ò{òjðë¢3íñ£ò8¢ñjëfû ò{ò{ñîúí 3ö ñ þd'ò{ê3ë¢ëí1ò{ð{ñRë¢ñ ò{ùö óò{îyí °í3íî 3þ3ëF0Sò{ëföðB> Ñ > S S ?    i10

Ë qsrmŸK:ªIÒpotK Î ©>d« LI}K.¨3uÑ Lj©G¨ / mn¯deGZfUŸ( x]tGa1±x8WRU Z]bkadZf^lX[`Ip UZfUrx]eG}q\fxAbkadZuZ¢X[`IgSa^ÁU)²GX[^  WI}qUÍx–Ur}qU®p­\]mnad`F†>ÅuXÅAm…ÆAoˆ†deG`Ip U Z]\fXdmq`3\{zlx]Xd^_WG}qmn`G¯Y†3å3eGUrZ]z´>z p ad^_^_my\]\]UrUd†}nU `G¯d\]tt !kad`Ä\]tGU»eG`3\]ZfeG`Ip X[\]U®gÄgSaSp)eG^_Ur`\¢†x ))† Xd`IgeIx]mq`G¯°\ftGUU `3\fmqZfUWYa>a} ¸ ƒ tGU `leYx–mn`G¯°\]tIUU `3\]mnZ]UWRa>ad}õ† \ftGU`>eG^„´RU ZBadbRa´Ix]U ZfwzXdprp)eGZ¢Xdp žz fu\]tGUWGZfadWRadZ]\]mnad`a[b gGa>p eG^_U `3\fxp adZfZ]U®p­\]}nz°}nXd´YUr}qU®g„´>z°\ftGU\]Z¢X[mn`GUrg„^aSgSUr} †X[`Ig ^_U®Xdx]eGZ]Ur^_U `3\fxXdZ]U°\¢X[U `a1wdUrZmq`Yp)ZfU ^_U `3\fxa[b\]tGU„\]Z¢X[mn`  mn`G¯x]U)\Íx–Um e |U ˆ y|ˆy¸usutIU„Xdprp)eGZ¢Xdp mqU®xuZ]UrWYaZ–\fUrg$X[ZfU XmnX[\]mnad`IxV¯dUr`GU Z  Xd}q}nz„x–\fXdZ–\fUrg_X1\ ‚ ¸ ‚Ž U®X[Zf}qz mn`\]tGUp eGZfwdUrxBXd`Ig±AUr`3\8Xdx}na1± Xx ‚ ¸ ‚ ‰XdxA^_adZfUgSaSp)eI^Ur`3\fx±AUrZ]UVXdgGgSU®gl\fa \]tIU\fZfXdmq`Gmn`G¯ x]U)\®¸ ÅuXÅAm…ÆAoWRU Z]bkadZf^_Urg$x–mn¯d`ImyÉYprX[`3\]}nzXd´Ya1wU'ZfXd`IgSa^(ad`$\ftGU eI`\fZ]eI`Ip X[\]UrgcAa^_W:¸ ;ÈXd`IgKX[}n}adbÍ\ftGUN\fZ]eI`Ip X[\]Urg§gSaSp e  ^_Ur`\¢x †Y±my\ftWYUrZ–bkaZ]^lXd`Ip)U'mn^_WGZ]a1wU ^_U `3\¢xumq`$\ftGU„}…Xdx–\ Žd‚ XgGgSU®g!gSaSp eG^_U `3\fxBadbI\ftGUeG`3\]ZfeG`IprX1\]U®g!ÓU eS\fU Z¢xBgGX1\¢XÍx]U)\®¸ hj`p)a^WYX[Zfmq`G¯\]tGUuWRU Z]bkadZf^_Xd`Ip)U8a[bYÅuXdÅm…ÆAo'Xd¯Xdmq`Ix–\.WImnp¢Â  mn`G¯Í\]tIU}qa`G¯dU®x{\BgSaSp)eG^_Ur`\¢x±AUx]U U\]tYX1\CÅuXdÅmnÆo aeS\]WRU Z  bkaZ]^lx\]tGU!}qUr`G¯[\ftNeS\]mn}nmy\{z$tGUreGZ]m…x–\]m…p[†¿gSU ^_ad`Yx{\fZfX[\]mn`G¯l\]tYX1\ ÅuXÅAm…ÆAo¾gSa>Urx`Ga[\lx]mq^_WG}nzºWGm…p¢Â»\ftGU$}qa`G¯dU®x{\_gSaSp)eG^_Ur`\®¸ a1±AUrwdUrZr†>p)}nax]UVU ²GX[^_mq`YX1\]mnad`Z]UrwdU®X[}…xC\ftIX1\ÅuXdÅm…ÆAolgGa3U®x bka}q}na1± \ftGUOWYUrZ–bkaZ]^lX[`Yp)UN\fZ]Ur`IgGxia[b}qUr`G¯[\fta`Ä\]tIUÈeI`  \fZ]eI`Ip X[\]Urg„gSaSp)eG^_U `3\¢x.x]eIp p U U®gSmq`I¯±tGU `„}qUr`G¯[\ft„gSa>Urx.X[`Ig WRU Z]bkadZf^_mq`I¯}nUrxfx±U }n}.±tIU `N}nU `G¯d\]tµb?X[mn}nx\]az3mnU }…gµXd´Ya1wU  Z¢X[`YgSad^ Z]U®x–eI}y\¢x ¸sutGU®x–U„Z]U®x–eG}q\fxVx–eI¯d¯dU®x{\XÀ^aSgSmqÉIU®gNÅuX  Åm…ÆAoÀ\]tIX[\Xdprp)adeI`\¢xbkadZ}nU `G¯d\]t:¸ ÅuXÅAm…ÆAoX[WGWIZ]awdU Z]\]mn`G¯Ç\]tIU / m…x–tIU Zmn`SbkadZf^lX  \fmqa`·^lX[\]Zfmy9² ¿ ª ¿ eIx–mn`G¯OgSm…X[¯da`IX[}umn`SbkadZf^lX1\fmqa`:†AXd`Ig¶mq\ ±adeI}ng·´YUmn`3\]UrZ]U®x{\fmq`G¯È\fa¾gSU)\fU Zf^mn`GUtGa1± ±AUr}q}ÅuXÅAm…ÆAo WRU Z]bkadZf^lxFmn`„p ad^_WIX[Zfm…x–a`°\]a°gS8m 7¿U ZfU `3\X[^_adeG`3\¢x.Xd`Ig°\{z>WYU®x adbBa 7  gSm…X[¯ad`IXd}:x{\fZ]eYp­\]eIZ]U'mn.` ¿ ª ¿¾¸‡\Vad`GU U ²>\]ZfU ^_Ud†G±U tYXýð{îúí¢îúý3ûrù¢ö®þ'ýSörí3ë¢í1ò{ñYë ñ >ûrù1ò{öøçlk·ö ÷3ò{êë örð{î Brîyí3ûrù í 0Sþë¢ùúd²ö eG^!´YUrZa[bWGZfUrgSm…p­\fadZ¢x †3bkadZXd}q}.gGX1\¢Xx]U)\¢x ¸ s X[´I}qU ('X1\]\]U ^_WS\¢xA\]a„Xd`Ix–±U Z8\]tGUÍå3eGUrx–\]mnad`Àadb:tGa1±Kp)aZ]ZfU  }…X1\fmqa`Xd^a`G¯ \ftGUÍWGZ]U®gSm…p­\]aZfxX 7¿Urp)\fxA\]tIUÍXdWGWGZfamn^lX1\fmqa` ´>z!\fXdÂ3mn`G¯°\ftGU !keI`\fZ]eI`Ip X[\]Urƒg )BgGX1\¢X x]U)\fxr†dWIZ]a ] Urp­\fmq`I¯'UrXdp¢t a´Ix]U Zfwzµ\]ZfeG`Ip X[\]mnad`:†.´GeS\!Xd}nx]atIXgµ\ftGUÀ±adZ¢x{\'a1wU Z¢X[}n} WRU Z]bkadZ  ^lXd`Ip)U†.b?Xdmq}nmn`G¯\]aµ¯dmnwdUÀX[´Ra1wdU  Z¢X[`YgSad^ WRU Z]bkadZf^lX[`Ip Ulad` Xd}q}.cAa^_W:¸ ;!Xd`Ig ƒ Urzit>eG^lXd`N}nXd´YUr}qmn`G¯adZ ^lXp¢tGmq`IUµ}nUrX[Zf`Gmn`G¯¶tYXdxa>prp)eGZfZfUrgˆ¸ŠsutGU»ÂU z§mn`G¯dZfUrgSmnU `3\

adbBadeGZV^U \]tGaSg$m…xXd`adWS\fmq^lX[}nmq\{zip Z]mq\]UrZ]m…X\]tIX[\ÍgSU®x]p Z]mn´RUrx ±mq\]t¾x]ad^_UXp p eGZfXp)z\ftGUÀU ²>WRUrp)\]U®gºU ZfZ]aZ°mq`IgGeIp)U®gº´>z»X WYX[Z]\]m…p)eG}…X[Z°\fZfXdmq`Gmn`G¯Nx]U)\®†.WGZfa1w>mngGmq`G¯X$^_UrXd`Ix°bkaZ ¯dZfU U®gSz aWS\]mn^_-m e®X1\fmqa`:¸ço>mq`Yp)U§p)ad^_^_ad` adWS\fmq^lXd}qmq\{z p)Zfmy\fU ZfmnXmq` \ftGUU)²SWYUrZ]mn^_U `3\fXd}3gGUrx]mq¯` }qmq\]UrZfX[\]eGZfUAXdZ]Up)a^WIeS\fX[\]mnad`IXd}q}nz mn`3\]Z¢Xdp)\fX[´I}qU_bkaZ„tGmn¯dt  gSmn^_U `Ix]mqa`IX[}8ad´Ix]U Zfw1X1\]mnad`Yx †.±AUÀgSU  wU }nadW„ÅuXÅAm…ÆAoˆ†®XVx–mn^_WG}qUuX[WIWGZ]a