University of Pennsylvania
ScholarlyCommons Technical Reports (CIS)
Department of Computer & Information Science
January 2004
Bayesian Example Selection Using BaBiES Andrew I. Schein University of Pennsylvania
S. Ted Sandler University of Pennsylvania
Lyle H. Ungar University of Pennsylvania,
[email protected]
Follow this and additional works at: http://repository.upenn.edu/cis_reports Recommended Citation Andrew I. Schein, S. Ted Sandler, and Lyle H. Ungar, "Bayesian Example Selection Using BaBiES", . January 2004.
University of Pennsylvania Department of Computer and Information Science Technical Report No. MS-CIS-04-08. This paper is posted at ScholarlyCommons. http://repository.upenn.edu/cis_reports/10 For more information, please contact
[email protected].
Bayesian Example Selection Using BaBiES Abstract
Active learning is widely used to select which examples from a pool should be labeled to give best results when learning predictive models. It is, however, sometimes desirable to choose examples before any labeling or machine learning has occurred. The optimal experimental design literature has many theoretically attractive optimality criteria for example selection, but most are intractable when working with large numbers of predictive features. We present the BaBiES criterion, an approximation of Bayesian A-optimal design for linear regression using binary predictors, which is both simple and extremely fast. Empirical evaluations demonstrate that, in spite of selecting all examples prior to learning, BaBiES is competitive with standard active learning methods for a variety of document classification tasks. Comments
University of Pennsylvania Department of Computer and Information Science Technical Report No. MSCIS-04-08.
This technical report is available at ScholarlyCommons: http://repository.upenn.edu/cis_reports/10
! #"$ $%&'( )* +,.-./1032541687:9=?,A@B7C6BDA0>-7FEG,.- H?03/EI,.-KJ8L:H?0NMO6P, QRES/ TVUWYX[Z]\]^_U`3\a[bCcAad^_WGeS\fUZXd`Igihj`SbkadZf^lX1\]mnad`oSpmqUr`Ip)U sutIU v`GmnwdUrZfx]my\{zladb|Ur`G`Ix]z>}qw1X[`ImnX ~FUw>mn`GU'X[}n} d X[}n`>eS\Vo3\]ZfUU\ |81YtGmn}n.XgSRUd}nWG[tGGm
GXG3G|C{>r>IS dd[.jddd>G 3R1dS ¢¡>3 £>¤G[>u j3d ¥§¦V¨©SªG«B¬R© x\]Z¢X1\fU¯mqU®xprX[`imq^_WGZfa1wdU°^lXdp¢tImq`GU°}nUrXdZ]`Gmn`G¯_Xdprp)eGZ¢Xdpzd¸ p\fmqwU}qU®X[Zf`Gmn`G¯°mnxB±m
gSUr}qz!eIxU®g \fa xUr}qU®p\C±tGm
p¢t Vp\fmqwU}nUrX[Zf`Gmn`G¯ X[WIWGZ]a3Xdp¢tGU®xÈ\]a*\]tGU¹WYa>ad}´IXdx]UrgÃx]U)\ U)²GX[^_WG}nUrx_bkZfad^³XOWYa>a}xtIadeG}
g§´YUµ}nXd´YUr}qU®g¶\fa f \ mq`I"¯ !£cAadtG`ÇU)\ÀXd} ¸n®d $ # ~FU±m
x %'&°X[}nUdVr $ # o>UreG`G¯ ¯dmnwdU´RUrx\§Z]U®xeG}q\fx·±tGUr`}qU®X[Zf`Gmn`G¯*WGZfUrgGmnp)\]mnwdU U \8X[}õ¸qIrd ( ):\fXdÂdUuX\fZfXdmq`IUrg!^_Xp¢tGmn`GUu}qU®X[Zf`Gmq`I¯ÍXd}q¯adZfmy\ftG^ ^aSgSUr}nxr¸$h£\m
x tGa1±UwdUrZr xa^_U)\]mn^_Urx gGUrx]mqZ¢X[´G}nU d X `Ig§WGm
p¢Â¶\ftGUµ`IU)²>\U²SXd^_WG}qUNbkZ]a^ \ftGUµWRa>ad}bkaZ}
X[´RU} \]a¹p¢tIa3a3xUºU²GX[^_WG}nUrx»´YUbkadZfU¶X[`>z¼}nXd´YUr}qmn`G¯aZ n m G ` ¯ºXdprp)aZfgSmn`G¯µ\]aOX»^_U®Xdx]eGZ]Uia[bÍU)²SWRUrp\fUrg·´YUr`GU)ÉI\r¸Çhj` ^_Xp¢tGmn`GUº}qU®X[Zf`Gmn`G¯KtIXdxNa>prp)eGZfZfUrg¸½sutGU¾adWG\]m f \ U f Z }qU®Xmq`G¯i^lXdp¢tGmn`GU_}nUrX[Zf`Gmn`G¯i±my\ftÈ}
X[´RU}nmn`G¯$mnx°mn`3\]U`YgSUrg ^_Xd}¿U)²SWYUrZ]mn^_U`3\fXd}:gSUrx]mn¯d`}nmq\]UZ¢X1\feGZfUVtIXx^_Xd`>z f \ » a WIZ]a1w>m
gSU´RU)\]\]UZ_^_UrXxeIZ]U®xadbU)²SWRUrp\fUrg¶´YUr`GU)ÉG\Àx]mq`Yp)U \]tGUradZfU)\]m
pXd}q}nz'X1\]\]Z¢Xdp\fmqwU8aWS\]mn^lX[}nmy\{z pZ]mq\]UrZ]m
XbkaZ _ ^ a Z]UÍmq`GbkadZf^_X[\]mnad`imnxuX`Ga1±`·Xdx ±tGU`Á±adZfÂ3mn`G¯¼±my\ftÃ}nXdZ]¯U·`>eG^!´YUrZfx¾a[b$WGZ]U Õ Ô ? â n ã ßÀÙ1ÖRÛ¢ÞdÔ +Û *]ãnßÛ¢ä¿âõÙ[ÖÜ>ÛÚ]-ã ,.ä !0/IUrgSaZ]a1w¿Sr 1( # cAtIX[}nad`IUZ gSmnp)\]mnwdU§bkU®X1\]eIZ]U®x¸ UÄWIZ]U®xUr`\º\ftGUÅuXdÅm
ÆAo %32BUZ¢gSmq`IU}n}qmõBr 4 )ubka>peIx]UrxÍad`\ftGUZ]Ur}nX[\]UrgµWGZfad´G}nU^ a[b p)Zfmy\fUZfmqa`:X[`ÄX[WIWGZ]aeG^´RUZladbÍ}
X[´RU}nUrg ^lX[\\]mn`G¯Èx]p¢tGUr^_UrxgGU^_ad`Ix\]Z¢X1\fUl\]tYX1\9X !k`Ga`IxU®å3eGU`3\]m
X[:} ) U²GX[^_WG}nUrx{æ\]t>eIxÍZfUrgSeYp)mn`G¯i\]tGUÀpax\°adb´IeGmq}
gSmn`G¯^lXdp¢tGmn`GU ÅuXm
gSUrx:X´IXdx]U}nmq`IUBbkadZF\]Urx\]mn`G¯±tIU)\]tIUZ ´GmnXxU®g'x]Xd^_WG}qmn`G¯ WI}qmn`G¯l^_U)\ftGaSgGxua[bBxUr}qU®p\fmq`G¯_U²SXd^_WG}qU®x¸ éûríê3ë ìí÷níörîyð{ï®þ ëfð{ûñjò{îóòõîúörôíör ÷A¢ø¿îúë¢ë¢í í¢í3ëñ£ôdéYùúëï®]ûêí3í3îyî ûûüuùë¢ Aý3ëfûýSðjörò{þ'ðjòë¢í1òö ö÷Aÿ ö®õÿþ'ý dò{ëfð öòC; ð{õí ë¢ýò{ùyê3û¢îú=ñ ë¢þ'< örëfí[?ð > ò+A[ûrð{ùúëùIB®ñ{ûûrðCþ'dùyýëfùyñjëfñAñ ö÷n÷:ð{örñ{þ ûrþ'ò{ý3ê3ùúëîúíýSBVö1ñ£öròjùIðûûð{ò{ëëBòôû >aw} x5mq`3\faÈgSm
x ] admn`\lx]U)\fx fzylC\ftGU\fZfXdmq`Imq`G¯Ox]U)\ Xd`I|g {NS\]tGU°ZfUrx]mngGeIX[}x]U)\\]a_^lX1²Smn^_-m erUV^lXdp¢tImq`GU°}nUrXdZ]`Gmn`G¯ ^_aSgSUr}WYUrZbkaZ]^lX[`Yp)Ud}¸ /Ga}q}na1±mq`I¯µ\ftGU`Ga[\¢X1\fmqa`·X[`Yg·gSU D E
p mnx]mnad`O\]tIUadZfzÈWGZ]U®xUr`3\fX1\fmqa`Oa[b@!õcAtIXd}qa`GUZ~%2BUZ¢gSmq`IU}n}qmõ ®d4 )µ±AUKgSUÉI`GUKX¼eS\]mn}qmq\{z bkeI`Ip\fmqa`3!0C
y L )»bkadZ ¯3X[eG¯mq`I¯^_aSgSU}8WYUrZbkaZ]^lX[`Yp)U_X1bà\]UrZ xUr}qU®p\fmq`G¯NX\fZfXdmq`Gmn`G¯ x]U)d\ yÄadbSÉG²SUrg!xUm eU¸.sutGUAwdUrp)\]aZ ip)adZfZfUrx]WYa`IgGx¿\fa\ftGUAWYX Z¢X[^_U\]UZ¢xCadb.X'\fZfXdmq`IUrgl^_aSgSU}õ _mnxuX wUrp)\]adZAa[b:ZfUrx]WYa`Ix]Urx bkaZl\]tIU\]Z¢X[mn`Gmn`G¯ºxU\rX[`Y}g ¶gSU`Gad\]U®xXOgSUrpmnx]mnad`Ç\]a¾´RU ^lXgSUX1bà\fUZA\]tGUV^_a>gGU}Im
x8\]Z¢X[mn`GUrg¸sutGUÍxUm eUadb\]tGU\]Z¢X[mn` mn`G¯xU\ ys[m
xC`Ia[\A¯mqwU`ÀX x]z>^´Rad}¿Xdx8my\mnxXdxfxeI^U®gÉG²SUrg \ftGZfadeG¯tGadeS\\]tGm
xU²SWYa3xmq\]mnad`:¸ÍsutGU!ÅuXWRUrp)\fX[\]mnad`Ix!mnx\]tYX1\ \ftGU_WGZfmqaZÍa1wUZÍWIX[Z¢X[^_U\]UZ¢xÍX[}n}qa1±xbkadZ'X[`ÈUrx\]mn^lX1\fmqa`µa[b ZfUrx]WRad`Ix]U°wdUrp)\]aZ L ±tImnp¢t$mn`\feGZf`$Xd}q}na1±xeIxu\faÀp)ad^_WGeG\]U U²SWYU®p\]U®gieGWRgIX1\]U®xubkadZ 8¸ cVeIZ'¯daXd}mnx°\fagSUwU}nadWÈU)²GX[^_WG}nUlxUr}qU®p\]mnad`ºxfp¢tGU^_U®xÍbkadZ aWS\]mn^lX[}Ô *)Û®Üdã árâ ãàÕ1ä»Ù>árá Z*Ù>Cá T$adb8XÅuXeG^´RUZa[bWGZ]U®gSm
p\]aZfxr¸Chj`$WGmn}qad\x{\feIgSmnUrx±U bkaeG`IgÀ\]tIX[\´>|z ( d WIZ]U®gSmnp)\]aZfxrd\ftGU°mq`>wUZ¢xU±uXdx8\]a>aU)² WRU`YxmnwdU\]apad^_WGeS\fUa`µX|Ur`\fmqeI^ h]hhpad^_WGeS\fUZͱtGU` Ur^_WG}qa1zUrgmn`Ixm
gSUX°¯Z]UrUrgSz!U)²GX[^_WG}nUx]U}nUrp\fmqa`lX[}n¯dadZfmq\]tG^$¸ sutIU'`3eI^´RUZ¢xAadb WIZ]U®gSmnp)\]aZfxeIx]Urgmq`$aeGZU^_WGmnZ]m
pXd}Uw1X[} eYX1\]mnad`iZ¢X[`G¯UbkZfad^ dd \]|a 14 3Xd`IgÀm
gSUrXd}q}nzÀ±U°±AaeG}ng }nmnÂdUV\]a±AaZ]Â_ad`igSad^lXdmq`IxA±mq\]tiUwdUr`¯Z]U®X1\]UrZ8`>eG^!´YUrZfxa[b WIZ]U®gSmnp)\]aZfxr¸ UµWIZ]U®xUr`\iX¾tGUreGZ]m
x\]m
pµX[WIWGZ]az>^bkadZ f ù E zUrx]mnXdÝ ` ù = `IXdZ][z ú²GX[^_WG}nU 7 U}nUrp)\]mnad`:¸ sutIUOÅuXÅAm
ÆAo XdWGWGZfamn`G¯lad`i\ftGU'\]tIU `>eG^´RUZadb\]mn^U®xubkUrX[\]eGZfUrø xË Xd`IÄg ª ûip)a aSppeGZ®IX[`Ig\]tGUgSmnXd¯da`IX[}:w1Xd}qeGU®x !0¿ ª ¿) ÊUÊ p)a` x]m
x{\fmq`G¯adbA\ftGU_\]ad\fXd}C`>eG^´RUZ'adbA\fmq^_Urx bkUrX[\]eGZfU ËaSppeGZfxr¸ U_WGZfadWRax]U!X[WIWGZ]amn`G¯mq\bkZ]a^ \]tGUuWRa3a} /10 þ!k±ý./1mq\], tIady eS\eGW¿gGX[\]mn`G¯_\]tGU padeG`3\¢x d)¸
Béê ë Að{ë¢+ë [ô û î ð A ù B®öð{îúò{êþ gGmnXd¯dad`YX[}õA}
Xdx\ÀZ]a1±'Xd`IgÇ}nXx{\pad}neG^_`:¸ sutGUµXdWGWGZfaad}p)aeG`3\fx Xd`Igl\fZfXdmq`Imq`G¯xU\8bkU®X1\]eIZ]U°p)aeG`3\fx gSU xfp)Zfmn´Gmq`I¯\ftGU_`3eI^´RUZ'a[bA\]mn^_Urx'XWGZ]U®gSm
p\]a!Z aSprp)eGZ¢xÍmq` \ftGUÀWRa>ad}AXd`Ig»\]Z¢X[mn`Gmn`G¯NxU\ZfUrx]WYU®p\]mnwdUr}qz¸sutGUZ]U®xeG}q\]mn`G¯ a´ ] Urp)\]mnwdUÍbkeG`Ip)\]mnad`i\]a_^_mn`Gmq^_Um eU m
x f ÅuXdÅm
ÆAo !0y~)È Î #s"À !{¡ ( ) $%'( &
±tIUZfU \]tIUx]eG^) m
xVa1wdUZVWGZfUrgSm
p\fadZ¢x!+*0°-,°±AaZfgi\fadÂdUr`Ix mn`OXigGa>peG^_U`3\ p)}
XdxfxmqÉYprX1\]mnad`»gSad^lX[mn`)VZ¢X1\]tIUZÍ\]tIXd`»a´ x]UZfw1X1\fmqa`Ix¸Ov`I}qmnÂdU\]tGU aWS\]mn^lX[}nmy\{zOad´ ] Urp\fmqwU ½ C\ftGU ÅuXÅAm
ÆAoÇeG\]mn}qmq\{zKpX[`´YUÈp)ad^_WGeG\]Urg¹å3eGmnp¢Â>}nzd^lX[Â>mn`G¯¶mq\ bkU®Xdx]mq´I}qUl\faµX[WIWG}qzOX¯Z]UrUrgSzÈX[}n¯dadZfmq\]tG^$¸sutGUXd}q¯adZfmy\ftG^ ±UNeIx]Uµmn`ÄUrw\]ZfU^_U}nzNw1X[}neIX[´I}qU_mqb8\]tIUzµWIZ]a1wUWGZfUrgSm
p\fmqwU_a[b8\ftGUlZfU x]WRad`Ix]Uw1X[ZfmnXd´G}qU¸¹o>U®p)ad`YgA±UtGaWYU\faº}qU®X[Zf`ÇXppeGZfX[\]U WYX[Z¢X[^_U)\fUZ¢x_bkadZ$Xdx^_Xd`>zÇWIZ]U®gSmnp)\]aZfxÀXxWRaxfx]mq´G}nUµx]mq`Yp)U \ftGm
xB±mq}n}Smn`Ip)ZfUrXxUA\]tGUWIZ]a´IX[´Gmn}nmy\{z adbYtIXmn`G¯Íad`GUuadZB^_aZ]U WImqU®p)Urxa[bBUw>m
gSU`Yp)U'\faÀeIx]U mq`µ^_XdÂ>mq`G¯ÀXÀWGZ]U®gSm
p\]mnad`$bkaZX \fUrx\_x]U)\_ad´YxUrZ]w1X1\fmqa`:¸»sutIUiÅuXdÅm
ÆAoOpZ]mq\]UrZ]mnad`¶prX[WS\feGZfUrx ´Ra[\ftKa[b°\]tIUrx]UºÜ>ÛÚ]ã Ü>Û *ÙSpâ Z[/ß .$\]tGUµWGZ]mnadZ ¢ gSU)\fUZf^_mq`GU®x \ftGU´RU`IU)ÉG\a[bx]UUmn`G¯NXWIZ]U®gSmnp)\]aZ bkadZ\]tGU¬ £ ÉIZ¢x{\ \fmq^_Umq` \ftGU$\]Z¢X[mn`Gmn`G¯ºxU\r\]tIU`3eI^UrZfX[\]aZ!\fUZf^ ¯mqwUrx_tGmn¯dtÇbkZfU å3eGUr`Ip)zWGZfUrgSm
p\fadZ¢x¯dZfUrX[\]UZBWIZ]mnadZfmy\{zd±tGmn}nU\]tGUbkZfXp\]mnad`YX[} Ur}qUr^_U`3\fx_ZfUWGZfUrx]U`3\_XÈgGmq^_mn`Gmnx]tGmn`G¯OZ]U\]eGZf`Ixmn`Çx]UUrmq`G¯OX WIZ]U®Ù gSmnp)\]aZ^aZ]UV\]tYX[`$ad`IpUd¸ éê ê3z¸ sutIU UeG^_UZfmnprX[}Fx\]Zfmq`I¯x ^lXdWGWYU®gÈ\]a$\ftGUx]WYU®p)m
X[}B\fadÂdUrá` ^5 ¸ _p)aeG`3\p)eS\ a 7¶a[b ÉYwdU°aSpp)eIZ]ZfU`IpUrxm
xueIxU®gIx]a\ftIX1\X±adZ¢g \{z3WRUÍ^eIx\aSp peGZX1\i}qU®Xdx.\ 4¾\]mn^_Urximq`\]tGUOgGX[\fXÇxU\imn`aZfgSUrZ\fa·´RU mn`Ip}qeIgGUrgKmn`§\]tGUµwdaSpXd´GeG}
X[Zfzd¸ o3\]aWK±AaZfgGxX[ZfU$ÉI}y\fUZfUrg bkZfad^ \ftGUÇwdaSpXd´GeG}
X[Zfz¹eIx]mn`G¯¹eYxmn`G¯¹\]tGU§ÓX[mn`>´Ya1± x{\fa WI}qm
x{Y\ !aNp®cXd}q}neG^$°r ))¸Khj`Ç\]tGUNprXdx]UiadbV\ftGUÈcAa^WF¸ ; gIX1\fXÍx]U)\fxr[Xdxfp)mnm Ur`Ip)aSgSU®g'´Gmn`IX[Zfz gGX1\¢X±uXdx.^lX[`>eIXd}q}nz xUrW XdZfX[\]U®g_bkZ]a^Á\]tGU'XdZ\fmnp}qU®x¸Æ8Xp¢tgSaSp)eG^_Ur`\±uXdxAU`IpaSgSUrg Xx_X»´Imq`IXdZ]z w1X[}neGUrg¶wdUrp)\]aZ±tIUZfUiU®Xdp¢t·wdUrp)\]aZ_pad^_WRa `IU`3\ÀU`IpaSgSUrgDZtGU\]tGUrZXOxWRUrpmyÉYpN±AaZfg \{z>WYUiaSppeGZ]ZfUrg mn`i\]tGU gGa>peG^_U`3\r¸ s.a´RU)\\fUZ eG`IgSUrZfx\fXd`IgV\]tGU8mq^_WIXp\:\ftIX1\.gGa>peG^_U`3\F}nU`G¯d\]t tYXdxa`NÅuXdÅmnÆo$X[`Igi\ftGUad\]tGUrZU²SXd^_WG}qU!xUr}qU®p\]mnad`NXd}q¯a Zfmq\]tG^lxr:±AUlpZ]U®X1\]U\fZ]eG`YpX1\fUrgÈgGX1\¢Xx]U)\¢xÍbkadZ'U®Xdp¢tÈa[b8\ftGU gIX1\fXxU\fx:}nm
x{\fUrg Xd´Ya1wUd¸FsutGUC\]ZfeG`IpX[\]U®g xU\fx.X[ZfUBWGZ]UrWIX[ZfUrg Xd`IX[}nad¯adeIx]}nzl\]a_\ftGUeG`3\fZ]eG`YpX1\fUrgiad`IUrx # a`G}qzigSaSp)eG^_Ur`\¢x XdZ]UVZ]U®x{\fZ]m
p\fUrgÀ\]a!\{±AUr`\{z ÉIwdU\]aÂdU`Yx8mn`}nU`I¯[\]t !à\ftGUÍÉIZfx\ ( 4!eG`Gm
åeIU'\]aÂdU`Yx )uXd`Ig$X[`>zgSaSp)eI^Ur`3\±my\ftibkU±UZ\ftIX[` 3
\{±U`3\{z ÉIwdUleG`Imnå3eGU\]aÂdU`Yx m
x!gSm
xfpX[Z¢gSU®g¸µsutGUWGeGZfWYa3xU adb\]tGU°\fZ]eG`YpX1\fUrggSaSpeG^_U`3\Vx]U)\fxmnxu\falbkeG}n}qz$Xdprp)aeG`3\bkadZ gGa>peG^_U`3\.}nU`I¯[\]tp¢tIX[Z¢Xdp)\]UrZ]m
x{\fmnprxmn`'\]tGUUw1Xd}qeIX[\]mnad` x]mq`Yp)U gGa>peG^_U`3\Àp)}
XdxfxmqÉYprX1\]mnad`·tIXdx\ftGU$WGZfadWRUZ]\{zO\]tIX[\lgG8m 7¿UZ Ur`3\ ad´Ix]UZfw1X1\]mnad`Yx !?gSaSp)eI^Ur`3\fx )°tIXeG^´RUZ¢x adbgSm
x{\fmq`Ip)\i±adZ¢gGxr¸ c\ftGUZ$gSad^lXdmq`IxigSa·`Ga[\$tIXYeGU`Ip mn`G¯_WRUZ]bkadZf^_Xd`Ip)U°adb.U)²GXd^WI}qU x]U}nUrp)\]mnad`$x{\fZfX[\]U¯mqU®x¸ sutIU8p)}
XdxfxmqÉIUrZeIxU®g°mq`'Urw1X[}neIX1\fmqa`'±Xx:XÅuXadK} JKXd`Igmq`\]tIUpXxUua[beG`Yp)UZ \¢X[mn`3\{z_xfX[^_WG}nmn`G¯X[`YgÀå3eGUZfz ´>z pad^_^_my\]\]UrUd3ad`GUVadZ8^_aZ]U p}nXx]x]mqÉIUZ¢x'\fZfXdmq`GU®gObkZfad^½\]tGU$p)eIZ]ZfU`3\\fZfXdmq`Imq`G¯ÈxU\r¸Ovx mn`G¯\]tImnxumn`SbkadZf^lX1\fmqa`À\]a_mn`SbkadZf^mq\fxp¢tGamnpUd3\]tIU xUr}qU®p\]mnad` Xd}q¯adZfmy\ftG^(m
xu\ftGU`NXd}q}na1±AU®g\]alZfUrå3eGUrx\\]tYX1\VX_WIXdZ\fmnpeG}
X[Z gGa>peG^_U`3\ÍbkZ]a^(\ftGU!WRa>ad} ´YU}
X[´RU}nUrgNXd`IgNXdgIgSUrgN\]aÀ\ftGU \fZfXdmq`Imq`G¯lx]U)\®¸ 7 þr0 üML1 mybR\]tGU\fa[\fXd}S`>eG^´RUZBadbRgSaSp)eI^Ur`3\fxmn`\ftGUu\]Z¢X[mn` mn`G¯ixU\Vm
x}nUrxfx\ftIX[`¶ d1 N \]tGU!`3eI^´RUZÍa[bAp}nXx]x]UrxrYx\]UrWIx 'Xd`I.g (!XdZ]UÍZfUWRUrX1\fUrg¸
1
0.9
0.9
0.8
0.8
0.7
Accuracy
Accuracy
0.7
0.6
0.6
Random BaBiES Uncertainty Sampling Query by Committee Entire Pool Length
0.5 Random BaBiES Uncertainty Sampling Query by Committee Entire Pool
0.5
0.4
0.4
0.3
0.3 50
100
150
200
250 300 350 Number of Articles Labeled
400
450
500
50
1
100
150
200
250 300 350 Number of Articles Labeled
400
450
500
1
0.95
0.95
0.9 0.9
Accuracy
Accuracy
0.85 0.85
0.8
0.8
0.75 Random BaBiES Uncertainty Sampling Query by Committee Entire Pool
0.75
0.7
0.65
0.65
0.6 50
100
150
200 250 Number of Web Pages Labeled
300
350
400
50
0.98
1
0.96
0.98
0.94
0.96
0.92
0.94 Accuracy
Accuracy
Random BaBiES Uncertainty Sampling Query by Committee Entire Pool Length
0.7
0.9
0.88
100
150
200 250 Number of Web Pages Labeled
0.84
400
Random BaBiES Uncertainty Sampling Query by Committee Entire Pool Length
0.88 Random BaBiES Uncertainty Sampling Query by Committee Entire Pool
350
0.92
0.9
0.86
300
0.86
0.82
0.84 0
20
40
60
80 100 120 Number of Articles Labeled
140
160
180
Õ
200
0
20
40
60
80 100 120 Number of Articles Labeled
Õ
140
160
180
200
Sñj Kî örOXf Pòjë¢ò{ñ\ö®BrþRîúïûñ r1ùY.ò{öÄô'ð{ç[+ö Zû3þýSdò{ö1îúö®0Sörùíë]ð°ûîyí í ör ÷ ò{òjðë¢3íñ£ò8¢ñjëfûò{ò{ñîúí 3ö ñ þd'ò{ê3ë¢ëí1ò{ð{ñRë¢ñ ò{ùö óò{îyí °í3íî 3þ3ëF0Sò{ëföðB> Ñ > S S ? i10
Ë qsrmK:ªIÒpotK Î ©>d« LI}K.¨3uÑ Lj©G¨ / mn¯deGZfU( x]tGa1±x8WRUZ]bkadZf^lX[`IpUZfUrx]eG}q\fxAbkadZuZ¢X[`IgSa^ÁU)²GX[^ WI}qUÍxUr}qU®p\]mnad`F>ÅuXÅAm
ÆAodeG`IpUZ]\fXdmq`3\{zlx]Xd^_WG}qmn`G¯Y3å3eGUrZ]z´>z pad^_^_my\]\]UrUd}nU`G¯d\]tt !kad`Ä\]tGU»eG`3\]ZfeG`IpX[\]U®gÄgSaSp)eG^_Ur`\¢x )) Xd`IgeIx]mq`G¯°\ftGUU`3\fmqZfUWYa>a} ¸ tGU`leYxmn`G¯°\]tIUU`3\]mnZ]UWRa>ad}õ \ftGU`>eG^´RUZBadbRa´Ix]UZfwzXdprp)eGZ¢Xdpz fu\]tGUWGZfadWRadZ]\]mnad`a[b gGa>peG^_U`3\fxpadZfZ]U®p\]}nz°}nXd´YUr}qU®g´>z°\ftGU\]Z¢X[mn`GUrg^aSgSUr} X[`Ig ^_U®Xdx]eGZ]Ur^_U`3\fxXdZ]U°\¢X[ÂU`a1wdUrZmq`Yp)ZfU^_U`3\fxa[b\]tGU\]Z¢X[mn` mn`G¯x]U)\ÍxUm e|U y|y¸usutIUXdprp)eGZ¢XdpmqU®xuZ]UrWYaZ\fUrg$X[ZfU XmnX[\]mnad`IxV¯dUr`GUZ Xd}q}nzx\fXdZ\fUrg_X1\ ¸ U®X[Zf}qz mn`\]tGUpeGZfwdUrxBXd`Ig±AUr`3\8Xdx}na1± Xx ¸ XdxA^_adZfUgSaSp)eI^Ur`3\fx±AUrZ]UVXdgGgSU®gl\fa \]tIU\fZfXdmq`Gmn`G¯ x]U)\®¸ ÅuXÅAm
ÆAoWRUZ]bkadZf^_Urg$xmn¯d`ImyÉYprX[`3\]}nzXd´Ya1wU'ZfXd`IgSa^(ad`$\ftGU eI`\fZ]eI`IpX[\]UrgcAa^_W:¸ ;ÈXd`IgKX[}n}adbÍ\ftGUN\fZ]eI`IpX[\]Urg§gSaSpe ^_Ur`\¢xY±my\ftWYUrZbkaZ]^lXd`Ip)U'mn^_WGZ]a1wU^_U`3\¢xumq`$\ftGU}
Xdx\ d XgGgSU®g!gSaSpeG^_U`3\fxBadbI\ftGUeG`3\]ZfeG`IprX1\]U®g!ÓUeS\fUZ¢xBgGX1\¢XÍx]U)\®¸ hj`p)a^WYX[Zfmq`G¯\]tGUuWRUZ]bkadZf^_Xd`Ip)U8a[bYÅuXdÅm
ÆAo'Xd¯Xdmq`Ix\.WImnp¢Â mn`G¯Í\]tIU}qa`G¯dU®x{\BgSaSp)eG^_Ur`\¢x±AUx]UU\]tYX1\CÅuXdÅmnÆo aeS\]WRUZ bkaZ]^lx\]tGU!}qUr`G¯[\ftNeS\]mn}nmy\{z$tGUreGZ]m
x\]m
p[¿gSU^_ad`Yx{\fZfX[\]mn`G¯l\]tYX1\ ÅuXÅAm
ÆAo¾gSa>Urx`Ga[\lx]mq^_WG}nzºWGm
p¢Â»\ftGU$}qa`G¯dU®x{\_gSaSp)eG^_Ur`\®¸ a1±AUrwdUrZr>p)}nax]UVU²GX[^_mq`YX1\]mnad`Z]UrwdU®X[}
xC\ftIX1\ÅuXdÅm
ÆAolgGa3U®x bka}q}na1± \ftGUOWYUrZbkaZ]^lX[`Yp)UN\fZ]Ur`IgGxia[b}qUr`G¯[\fta`Ä\]tIUÈeI` \fZ]eI`IpX[\]UrggSaSp)eG^_U`3\¢x.x]eIppUU®gSmq`I¯±tGU`}qUr`G¯[\ftgSa>Urx.X[`Ig WRUZ]bkadZf^_mq`I¯}nUrxfx±U}n}.±tIU`N}nU`G¯d\]tµb?X[mn}nx\]az3mnU}
gµXd´Ya1wU Z¢X[`YgSad^ Z]U®xeI}y\¢x¸sutGU®xUZ]U®xeG}q\fxVxeI¯d¯dU®x{\XÀ^aSgSmqÉIU®gNÅuX Åm
ÆAoÀ\]tIX[\Xdprp)adeI`\¢xbkadZ}nU`G¯d\]t:¸ ÅuXÅAm
ÆAoX[WGWIZ]awdUZ]\]mn`G¯Ç\]tIU / m
xtIUZmn`SbkadZf^lX \fmqa`·^lX[\]Zfmy9² ¿ ª ¿ eIxmn`G¯OgSm
X[¯da`IX[}umn`SbkadZf^lX1\fmqa`:AXd`Ig¶mq\ ±adeI}ng·´YUmn`3\]UrZ]U®x{\fmq`G¯È\fa¾gSU)\fUZf^mn`GUtGa1± ±AUr}q}ÅuXÅAm
ÆAo WRUZ]bkadZf^lxFmn`pad^_WIX[Zfm
xa`°\]a°gS8m 7¿UZfU`3\X[^_adeG`3\¢x.Xd`Ig°\{z>WYU®x adbBa 7 gSm
X[¯ad`IXd}:x{\fZ]eYp\]eIZ]U'mn.` ¿ ª ¿¾¸\Vad`GU U²>\]ZfU^_UdG±U tYXýð{îúí¢îúý3ûrù¢ö®þ'ýSörí3ë¢í1ò{ñYë ñ >ûrù1ò{öøçlk·ö÷3ò{êë örð{î Brîyí3ûrù í 0Sþë¢ùúd²ö eG^!´YUrZa[bWGZfUrgSm
p\fadZ¢x3bkadZXd}q}.gGX1\¢Xx]U)\¢x¸ s X[´I}qU ('X1\]\]U^_WS\¢xA\]aXd`Ix±UZ8\]tGUÍå3eGUrx\]mnad`Àadb:tGa1±Kp)aZ]ZfU }
X1\fmqa`Xd^a`G¯ \ftGUÍWGZ]U®gSm
p\]aZfxX 7¿Urp)\fxA\]tIUÍXdWGWGZfamn^lX1\fmqa` ´>z!\fXdÂ3mn`G¯°\ftGU !keI`\fZ]eI`IpX[\]Urg )BgGX1\¢X x]U)\fxrdWIZ]a ] Urp\fmq`I¯'UrXdp¢t a´Ix]UZfwzµ\]ZfeG`IpX[\]mnad`:.´GeS\!Xd}nx]atIXgµ\ftGUÀ±adZ¢x{\'a1wUZ¢X[}n} WRUZ]bkadZ ^lXd`Ip)U.b?Xdmq}nmn`G¯\]aµ¯dmnwdUÀX[´Ra1wdU Z¢X[`YgSad^ WRUZ]bkadZf^lX[`IpUlad` Xd}q}.cAa^_W:¸ ;!Xd`Ig Urzit>eG^lXd`N}nXd´YUr}qmn`G¯adZ ^lXp¢tGmq`IUµ}nUrX[Zf`Gmn`G¯¶tYXdxa>prp)eGZfZfUrg¸sutGU»ÂUz§mn`G¯dZfUrgSmnU`3\
adbBadeGZV^U\]tGaSg$m
xXd`adWS\fmq^lX[}nmq\{zipZ]mq\]UrZ]m
X\]tIX[\ÍgSU®x]pZ]mn´RUrx ±mq\]t¾x]ad^_UXppeGZfXp)z\ftGUÀU²>WRUrp)\]U®gºUZfZ]aZ°mq`IgGeIp)U®gº´>z»X WYX[Z]\]m
p)eG}
X[Z°\fZfXdmq`Gmn`G¯Nx]U)\®.WGZfa1w>mngGmq`G¯X$^_UrXd`Ix°bkaZ ¯dZfUU®gSz aWS\]mn^_-m e®X1\fmqa`:¸ço>mq`Yp)U§p)ad^_^_ad` adWS\fmq^lXd}qmq\{z p)Zfmy\fUZfmnXmq` \ftGUU)²SWYUrZ]mn^_U`3\fXd}3gGUrx]mq¯` }qmq\]UrZfX[\]eGZfUAXdZ]Up)a^WIeS\fX[\]mnad`IXd}q}nz mn`3\]Z¢Xdp)\fX[´I}qU_bkaZtGmn¯dt gSmn^_U`Ix]mqa`IX[}8ad´Ix]UZfw1X1\]mnad`Yx.±AUÀgSU wU}nadWÅuXÅAm
ÆAo®XVxmn^_WG}qUuX[WIWGZ]a