Designing Agent Collectives For Systems With Markovian ... - CiteSeerX

Designing Agent Collectives For Systems With Markovian Dynamics Paper ID:546 David H. Wolpert

John W. Lawson

NASA Ames Research Center MS 269-2 Moffett Field, CA 94035

NASA Ames Research Center MS 269-2 Moffett Field, CA 94035

[email protected]

[email protected]

ABSTRACT


     !"$#%'&)(+*-,/. 012 ,34 ,5 
76859 *) :51 ;*
.;=. 5=
 57.;5 5?,:@ A0B.DC01E1
,= 6F 6GH.;$GI'JK*LG 5M 
I, ,/.; NO.;G55P.@Q, DFR68O6S"2 ;,R687GI'JT*LG A U ,K
FI, :T 4VWA.;5 90B.DC8 01E6YX[Z%, 2 75
 O2\
 D2] ACF6K
.;=*R,/. 012 ,3N >5H.D, 5B
H.OU K^7.;,3 ;F . _68J8. 015+2`
9 T,U": G8 _ ;*%
a5?J85?U 0b*-, 0c G,Ud*e.;GH.;W IR.;f 6I 5%d. @ *-aC8QW8U 5:AO. GR.D W(g5gQW,01IO6YXh#%G,T.;QQ8, :. /
S,/.;5?*- ,015 
gL01U"CFH68O6H(%.D,G01i *$O.;
4.;Oj 5PGJT*LGU  _VW*- ,7.; G.;T
H.D=*LG kX9
5l,/.;5?*- ,0B.; 
. 5=VWm5l_5H.D, 51 ;18 F >^7.;,3: DF . _68J8H.;0=U 5MnNQ.;,G .D,=5.;, 5+2`
,B ;=.; *`
a.D,G015 ;*o.;p.;Oj 5$GI'J1*LG p.D,% V5, . VXrqAsF:5?:.D 
5t,/. 5?*L ;,0B.; BB5 0+G .; 5 F:  FŒ?ŸT—-•a›H£Œ ¡O–e“•“_tŒ £’“9‘8—e–L’ –L—eœf¦§QH.;,G .;,OM)5A ,GIJpR.D,+5?J5?015s68/. O6g01 F6 @ ;*r
i5?J85?0\ 5sG5G8U . IJT01QW 55VMk
 O2¨. T2=.O:  65G
T01 F6 ©. T2 5?O. 6,/.  +
P501Q+. 55G01Q8 g
H.D) G8,)O.D,, .  ,I
015P.;,l 6F 6GH.;JT*L. I,IJA  F6N.;i2`
H.D
.D . /
, G,a,G.K5
 ;,5?7Q.;
.;  ,I
0 "y
f«)(Xt³% 3 2`I
®.Tq‚Y³sU"VH.;5O6K : M$!y Y«`U"VH. 56K, GT
, 5$ + ,@*- ,`68, G5$5R68U"‰f5$ ;*k, Gl6 5 5 _
p VH. ` :. PeX XM$ > ,_*- ,BVW /385/(X4qA ,/.;B5 0+G .; 56801 5?,/.;)
.;.,5G5iG5 7.a¢.J ;, y8, 5=01
 F6N .668,55+
p01 ;,a ,/. $ O.D,BO. 5 
H.;a
 u G.6F,/.;)  *$
1
 ,015i ;*s F: .  *-GIUd,/.; . I'J7. 01%
 ,JT| } ¯D€dX$#%B
% 
,`
H. 6YM. =.D 2`

4.; t 01QW 5i  * G   „B5+.7 O. 0B.DC8 0+G0ª < * ) e ;,z01 ,)Q8,5IJMW.l,O.;xQW  ` ;*o
E  )l+( 5G8,*e.;D(X "*2 O.;+/
  5 ]t5 s
.;¢
n
I,/6P H68 H. FQ8, VH.;VI'J 4
a ;,/. 695=QWO.;3:O64.;, G69: , 5 G  .; ` ;*%2`
 5  01QW 5.;,s 5) B„M8
12%
H.O:%3:IJBH68G6@ .;,  5Xn*YB.66I =
`5 H6l,0 5tQWO. 3:61. VW G8 G  Mn
  G  2`s.;5 AVWp .D,X_­¢H.; IJMtI*s
 G   u GH.;  m,5?t,0 5tQWO. 3:6=. VW G

)!2`
  G  5tR.D,M
1 G8, /
 i [* ]%2`x3:IJa,5GIz p

)+MH.;5%685I,O6YX GI:IJMO
 , u GI,01x
.;¢
 GI'J p
5P65I,/.;VB u GIU V,G0vVW
H.OF ,OMF*e.; ,O6a :5s. G8 0B.D . IJaQ8, D8 68 .;QQ8, Q, .D+ ;‰WU u G V8, G0\ :5` =
iQ .J,5+d. g5?U 5G),/.;,IJa 5 6,O6a p:. 01s
 ;,J 3+01
. 50v685H(X 2.2 Opacity

qAg O2 *L G5a .  ;,I
015@VH. 56! !G8 I'J9*LG 5

-@. / 1>
H.;a Q801 E
5 H.  3D 5A,/.D 4, H6!
 
I,/6,07Mk5GV ? P @
l, u GI,01)
H.D,/. nIi 5$C0?iG5M2`

1.D,$ *W
*L ,0 ++ ( K)l+D( VAnB!x+(( d°:( 2`
,C  AnB!W(i5l6QW6+  *  / X>«sJN68I‰f,aG8 I'JA5 *e.; ,6_| } °D€"M¢. 6TGH68,)VW>. QQ8, OC0B.D 5DM 81 /9E ];( 5`01 01 EO6a D:,$
)5z *r5G
aGI 5sVJB/
  5 AtB!x+(( F81Z)G 3 /W ]D( W "( GQ4 N.;® D:,/.;s. 668:a 5?/. OX®qApO.;z
a,5G/.; 6I‰f,gG8 I'J9
I  H %KJ@#@C0L-
#AGI'Jwd«$³O.;
Q 5 F68+
lI,+5?J85?0ª:  5)A.a^7.;,3 U F . 0B. ,s*-, 0´
PI . r01 D:5% ;* 
+QR.J,5XPqAl.D, ,5?6995G/
_. 015l2`
,75 01p *z
aQ .J,5 54.D,  i.;5%2`
 5= . r5?/.;+5VWi,U":  O6T @Q, F68GP
+65I,O6 0=U V.; YM l

6 /  % (X ³%8*L ;,GH.DIJ+6 s
5o01O.;5¢,U" F s
I,5?J85?U 07MD2`
/
+0B.JX? -
Y$( >D-,%'C J  $?> # X??
> X% 4 > ?> # %KJ J@? #7#9 O? 4 -#7# $? ? A %'>$>D%'>$A  8? -C ?$, %KJ@$? 8? J&!D #J ZB 9H .#@? ( 4? J&!D # J, BP ¯  J@?  - %''% # ( 9H . -
? # $?> N - ,.,  (X%'>$A6 #@ # $? 7J@#-J ?'-J J@?C0B>D#@?   J A % ? J # $?;!$#@%''% # ( . 8% # ? J&!D # JY$? ,D%'C0#@?F  -J 8.  K% J .  % J -B>D > -' ( ) # $? #9? -   A -  ? K% J 44B -7 J -4? #@  J -B #@ J@? ?





700

600

World Utility

500

400

300

200

100

0 0

 B
C 50

100

150 Time



200

250



300

% AB!$4? ( J@#@?  ,[? ZB  -B>DC? ZB -A?
> # J 8% #  %'>$?$-X($> - ,%'C J;%'>-L-B>$$2 ?> X% 4 > ?> # 7? ( %KJ J-   ? -J % A !X? $?J -B'X?A7- -#@%  >K%'> ,? ?ZB -B>DC0?F$!$?F#@ L-B>DX2,>$? J J@Z42 # $?F?> $% 4 >  ?>#

= 

4.2 Observability restrictions



A@

0

% A !$4?5F=H G 2,.-%KJ@ >K  -I ? L-AB?;>$B%KJ@?=ZB7B-BC0#9B4?
 #"%$%&KJ=!D,.,? ABL-,LEM -B>D >X > B-BC0#@? .#-/"%$%. & J  
?  ABL-,LE M !$#&%' % # (  !D>$C0#&%  > J  % # !N O   J@?   -  %''% # (PX? * 7J@#7
I #&%  ?J@#@?$,2J-4?F# $?;%'>D% #@% -B#@L-B% >D%'>$A ,? % XQ

300



 NQ,/.;M¢
=*LG n2 ;,R68 B *$
15?J85?0h0B.J ;PVW *-GIJ4 V5,.;Va TO.;
_.  OX4yFG/
901IO6K V5,.;VI'J ;*n.=QH.D, G .;,z 01QW F`0B.J@VWi6,01O6@VFJB
)Q, VU 07Xra ;
,`. 55M6G% l 010lGO.; @ 5?,/.;F5`O.;
.;  5t J=.  O2O61 < V5,:s.),/. BFG0+VW,r *k 01QW ;U 5Mo.;H6>0+G5?Q5 F6+VFJ7.    5WMf:TI*t
+ . G )l  $5%  VH.;J V8, :. 6O.;5?s 1.;x.;5X 
7¢«$³ . QQ, . /
_ GO69. VW D:@5=2IU"5GIO6K A. 68U 6F,55%5G
T5IGH.D 5X)­ ,0B.;JMW.@680B.;67R68I'J7 QW,U .D , [O. 9VWg68mHO6N2`
 576 .  H.;s 0151.;,;  -D† WO4„ 1 68QW6+ p2`

,z ;,s `
J@.;,i V5,.;VXt
$ %KJ@?  B  -C0#@B4? E M >$ > B-BC#@B4? .
%$%& J=  ?  )A;5O   J@?   -  %''% # ( -/.

% AB!$4? E2 ,.-
% J@ >   .
 =! ,.,?  ABL- , -B>$ A7- , ^!$#@%''% #&% ? JY8% #

%$%& J EM

100 Time

150

200

¯ X( . %$%& *L ;,t.s Ud*e.; ,6l5?J85?0ƒ2`I
1} ¯:w V5, . V I'Jf· °($
-. )O.;0v:. 01 M2`
, ) /  2   $ V, % / F  % A #& A / ?„²:( A > 2`
, # A /75 „ W = = W XBs5GI5P ;*$C8QW,015%*- ,P
5 . 5 , 5 50

100

150 Time

  200

  


250

% A !$4? ( J@#@?  ,?  5-B>DC0?ZB I-
AB?>#JE? ?CO!$# %'>$A X!D-B$L-#@%'C $(X> - '% C J $? ?> $% 4 >  ?># -J ??> %'>D% #&% -B % O? #9 O? 4 7? ( %KJ
J-;? -J %'> % A !$4?



 @

 C



C

300

x p.66F,55%
l01 ,+,/. r .;,iQ, V07Mf2l 8U 5 6,$.i5
IJ@6I‰f,Ft*-,/.;012 ;,3fXr1
5 O.;5M8O. /
@.   5P. 55O6K.@,O.;IU . GO6AFG0+VW, )4/FXB
15?/.D1 *
=5?J85?U 0    5).@: ,01 F68mO6KVJ -YW/2† 1X4&s  O.;,1GQf6.D@,G 5  2  G#  %O($.;,)*-G 5` *o
5%,O.;IU . GO6p .;, . V 5X  5?,G 7 *o*L.  ;,O6pGI5 .O/8   ( K)l& #  % (( V )l& #

6  % (&( = ?„O~( , u GI,5P
H.D  #: & %O(iVW@H68QWH68P * 5Wj 5l
 @ *%.  YX #%s2$.Ja l.  01Q5
p
 5$ + . 01QAd.;QQJ

6 t(   %  A >   L? TJ %$%& #"%%$ & ,M -/. C 150

160

170 Time

180

190

200

% AB!$4? (DJ@#@?  ,?ZB  -B>$C0? ZB   ¯;†F-AB?> # J ! J&%'>$A # $? - X(  B ? % ? J  ? # XX X?K$($>D-,%'C J %KJ A ? >$? (- ! -$L-#&%'C: !D>DC#&%  >  # X?
-AB?> # J
 , J&% #&%  .> J X? B  ' !$#&%' % # (M) %KJ- X!D-4#&%'C %'>  $% ;?
> J&%  .> J =! ,.,? ^  #   A7- , .J -4?Y. -B>D,. ,% DD ?  # C  -4? .
 ->D;.   
? # C-
? .
 -B >D-\#@?$-  A- ?;) $? % >D% &# % -#@L-B%'>D% >$A ,[? % K%KJ8>$B#MJ $
 >



 Q   

#"%$%& %$%& /- .







T>

 ) J #-/"%$%. &  %$%& ,M    -/. # HJ   160



170 Time

   #"%$%&  %$%&    

180

190

 I



M 



200



1

0.8

0.6

Intelligence

#





0 140

=P

150

% A !$4?  - ($  ?% ? J+ ? # XX  $? 4? # $? X! -$L-#&% C 0C  ? CO% ?> # J - ? ;B4? V # -> J&% AB>.J ABL- , .J ! , ,?  , -B%  -4? . ->D .  ,%'D$ ?FN # $4? ? -4? . .
 -B>$ # X?*#@?$- A-  ?   ?%KJ .
 9> # D% J C -J@? # $4? ?K Y# $? '%  % #@?" J@?  - %''% # ( !$#&%''% #&% ? J ->D # X? #@$? -  A - ? ,?ZB   B7J@? ?  #&%  ? =% ? # $?% #&%  ? % >#@?' % AB?>DC0? ZB -AB?> # JK!.J&%'>$A 9H . / =! ,.,?  ABL- , ? 7J&!.J -K#@?$- A-  ?  
?  ABL -,  X?
> # $? J&!$ -C0? ,$?1Z? ?> #&% -' ( #&!X> J $8> $? $? ABL - -#&%  > % > %'> #@?''% AB?>DC0? -JC02,.-4? #9 %A !$4? F4?  ?C0# J # X?;A?$-
#@? \$% CO!D # (   # $?,D4  ?  1

0.8

Intelligence

0.6

0.4

0.2

0

% AB!$4? N   ? C0?> #&%' ? %'>#9?
 '% AB?>DC0?  -
AB?># JK!.J&% >$A 9H . J=! ,.,? ABL-,LEM ? 7J&!.J - #@?$- A-  ? J=  ?  A7- , EM $B^#)$4?? -BC#&%  >.J - L-B>DX2 - J=>X  ? -
> % >$A= M  !D' AB%, ? -B>-I ? L-AB ? %'>#9?
 '% AB?>DC0?;  N5O 0

50

100 Time

150

EM 

?J





 

200



*L ;,015o2 ,5` D:,r01Xo+*L. OM n IJ=68 5n
 O.;0 . 01 :BQW  ;,lQW,*- ,0B.;MrVGiIP*e. 5=.;I 
,OX
1 O2 5?  5 < :9GID5 .i.  6 .#"%$%& 5?8:t, VG5?rQW,*L ;,0B. X x P*LG,
,$5?G68J=
 5`6F,/. -/0B. .D %6I‰f,)@QW,*L ;,0B. M 2` 01QH.D,O6+
`.O:,/. z  z *W
`.  5t*L ;, .+.;H6 
=. 0º:.;01Xg
B,5GI5l.D,B5
 O2cK­¢G8,@~8XgA
 O.;5z ;*W
O.;0¨:.;01M.;:. YM
,$5r
 
,= p
.  
sO. 0v. 01i. VWIz O2,z
H.;a
XH^7.;,5eMf. 6 lXHsXf
. 8XH« Q, ; JQW%01 F6Y *o5? /3@C8
.  X$‘F£Œ/›Fœ;Ž/–e Ž ¢•—e—•£/ŽMW…:†FŠ …: ~…:²;…Mr„O~ ~:X | °D€ lXH
H.; %. 6 +XHsXf
. XH#%a
%01  ,I'J@. 01Š «s. IJFO. k. 6pFG01,O.;Y5?G65X œ Ž–d ˜  MH}¯;°8Š ¯F„…M „O~ ~²FX | O€PsXf  .;G5z. 6gsX t G,OXW
i68J8H.;015$ * ,*- ,01sO.D,l  QW,/.D:i0lGI .   5?J85?015X8'$£?ŒO ••“ –L”¤ Ž+Œ?Ÿ)—-• t– Ÿ—••”—-!˜: /–L”f•¢•˜ £”–L”F¤ MQH.;5s};…:}F}¯ †FM&G=„O~ ~²8X |I„ „€ sX«PXZ%GVW,0B. a. 6p%XHZs  X
XZsGeX n  .D I'Jp. 6p.; .6. Q8/. VI'Jag.l5I*-U" ;,:. EB0B.;,3:OXQ8,Q,  H68U"0B.D  3 ~² †:}„DMf­V8,GH.D,J„~~ ²8X

|I„…;€P‚nXoX Di.  VMH^>XW‚rX‚kI0B. YM.;H67«iXFqµXW^p  ;,X s8*L ,01$O.;,Šn«5G,JXFŒ ‘F£/”W˜’xŒ?Ÿ  £/—e–   –d˜ ’ ”—•’L’ – ¤•”H /• `•Ž•˜ £? /FMW…8Š };‡: 8} ²¯FMo„~~°FX |I„D¯O€+^>XH‚rXH‚Y0B.;kX^7.;,3 ;@:.;015z. 5z.+*R,/. 012 ,3B*L , 0lGIIU.  $,*L ;,01`O.D,8XH'$£?ŒO /•/•“ –-”F¤ Ž+Œ?Ÿ —-•)+,+D—- ”—•£/”W˜;—e–dŒ ”W˜ ’ " Œ ”OŸ•£•”W /•lŒ;”#>˜: /F–L”W• ¢•˜ £”–L”F¤ MQH.;5i„D¯ H„°‡FM¢„O~ ~ …X |I„O°D€P%XWy8. H68
  0\.;H6@PXt,I5XY^pG .;`,*- ,01F  .;,1 @
)I,/.DO6pQ8, 5 ,Oj 5s6010B.8X -`–dŒ Žœ;Ž/—•  ŽMW‡:Š„… H„°°8Mr„~~:¯X |I„DO€P%XWy8. H68
  07M D=X‚x.D,5 kMH^>XW«zH68,5 YMH#+XHy8

 ;,JM . H6a­ XHk 
01 XH«zFJF01z :.; I T5?,GG,) ,/.;  2`
a2 ,5?zO.;5iGH.D,/. 5X'$£?ŒO ••“ –L”¤ Ž+Œ?Ÿ)—-• t– Ÿ—••”—-!i˜;—e–dŒ ”W˜ ’ " Œ ”ŸO•£•”H •+Œ ”  £—e– t –d˜’ ”—•’L’ – ¤•”H /•MQ. 5s…°8¯;‡8M¢„~~ ²8X |I„O²D€+yfXHy8YXg‘’ —e–L  ¤•”—.¢•˜ £/”–-”F¤0/1`˜›•£ŽnŸ/£Œ  —-•2+43,365   #7 Œ £98;Ž?Œ/›;:365D@?BA Xf«%«s«s r,55MW^p8 o.D,3WMW«iMk„~~:X |I„O~D€iPXyfXHy8G8 YX‚YO.;,l =Q8,O6zVJ1
˜ /F–L”f•¢•˜ £/”–L”¤ MW‡8Š ~……8M¢„O~ ²²FX | } †D€iPXyfXHy8G8 p. H6a«PX D+X  .;, X=z•–-”OŸŒ £? /•  •”— ¢•˜ £”–L”F¤0/  ” ”—e£ŒO“ ‘ —e–dŒ ”Xf^p' o,55MW. 0lV8, 6 M ^7«PMY„O~~ ²8X | }F„€+D=XWyJ.;,/.8XH^pGI .  z5?J5?015X  ˜O¤:˜O™–L”f•M „O~"} (Š  ~~:}Mo„O~~ ²8X | }}O€+D=XHxG01,`.;H6 +XZPXqA QW,OXH :iF  . H6 t,/.;55jQ.;,/. 6 OCfXF'#$£?ŒO ••“ –L”¤ ŽPŒ?Ÿs—-•Cf•¡ •”—•/•”—- !i˜ —e–eŒ ”W˜’ " Œ ”OŸ•£•”W /•lŒ;”  £—e– t –d˜’ ”—•’L’ – ¤•”H /•MQ. 5 „O† …H„†~8MY«zG5?kM
DPMW};†††FX | } ‡D€+sXqN.;3F5z. 7 6 ¢X).OJ. kXEsU"O.;,X>˜: /–L”f• ¢•˜ £”–L”F¤ MH²d‡03;…(Š }  ~F} ~}FMr„O~ ~:}FX | };…;€+^>BX ¢X8qN0B. YX«S0B.;,3:U" ;, F6@Q8,  ,/.;0101 F8I, 01`.;H6aI5z. QQO.; 7 165?,VG8O6 0lGI 0101 F6I'J  O2SQ8, V 015XFFŒ ‘F£/”W˜’¢Œ?Ÿ  £—e– t –d˜’ ”—•’L’ – ¤•”H /•(`•Ž•˜;£ /FM¢„~~ ‡8X | }¯O€ lXZPXFqN QW,OX  GH68O68Ud,/.D H.; IJp. 01%
 ,JX Q,U"Q,OM} † †8„X | } °D€ lXZPXFqN QW,OX