Repeated votes and a majority rule are properties of the ... Your observation results in z> c3 so your first vote is yes - .... is best & the simple majority worst. 0.5.
Effect of Majority Rule and Initial Bias on Information Aggregation by Groups R. D. Sorkin, S. Luan, & J. Itzkowitz University of Florida This research was partially supported by the Air Force Office of Scientific Research.
The problem: You (an art dealer) must decide whether to buy a possible ‘find’; a sketch by an early Impressionist. Your decision is ‘assisted’ by 3 experts who examine the picture (further tests/opinions are impractical)-
1
Louise
Expertise: High Correlation: with Louis Bias: vy conserv Vote: NO
Louis
Willy
Yourself
Medium
High
Medium
with Louise less conserv. YES
none very liberal YES
none medium -
Expertise Louise high Louis med Terrence high Yourself med
Bias vy consrv less cons. vy. liberal neutral
Correlation 1st vote with Louis N with Louise Y uncorrelated Y uncorrelated -
Is there an optimal way to combine their opinions? You hold a brief teleconference to obtain their final recommendations about purchasing the art… -
2
Optimal integration of a group’s information is a potentially difficult problem. In some cases, there could be person-by-person optimization of criteria and responses. Most people just use a majority rule to aggregate the information (which may describe some common ‘biases’). Suppose that repeated votes are taken… Repeated votes and a majority rule are properties of the standard American jury. How do the accuracy & bias of decisions depend on the parameters of such a system? -
Quick review of signal detection theory:
µN
Normalized separation = detection index, d’
µSN
z
zc z is a likelihood ratio statistic based on the input, x, e.g., z = log [λ(x)] = log [f(x|sn)/f(x|n)] The detector should respond ‘yes’ if z ≥ zc where zc is a function of the payoffs & prior probabilities. -
3
Receiver Operating Characteristic 1
All
Hit Probability
P(Conviction|Guilty)
0.8
on pts
e= hav C RO
d’
Negative zc Higher d’ & percent correct
0.6
Positive zc
0.4
0.2
0 0
0.2
0.4
0.6
0.8
1
False Alarm Probability
P(Conviction|Innocent)
A Distributed Detection Model of the Jury σ 12
di' =
µS1 − µS 0
+
s0 or s1 event
2 σ COMMON
σ i2
x1
+
2 σ COMMON
σ N2
2 σ COMMON
+
xN
xi
2 σ i2 + σ COMMON
d’1
d’i
d’N
lnλ ( xi ) = ln[ f ( xi | s1 ) / f ( xi | s0 )] lnλ(x1)
βi = V ⋅
p( s0 | r1, j r2, j ⋅⋅⋅ rN , j ) j ≠i p ( s1 | r1, j r2, j ⋅ ⋅⋅ rN , j ) j ≠ i
LDM 1
λ(x ) i) lnλi(x
lnλ(x1) ≥ lnβ1 ?
r1,0 or r1,1
LDM i
lnλ(x λ(xNN))
lnλ(xi) ≥ lnβi ? ri,0 or ri,1
lnλ(xN) ≥ ln βN ? rN,0 or rN,1
Decision Center ln[λ( r1,j , r2,j ,…, rN,j)] ≥ ln βC or K of N rule
Optimal rule “yes if n1 and y2 but not n3”
LDM N
Feedback Information about {d’i , βi}
Jury rule r0 or r1 response
4
How does the vote feedback influence your vote? You are detector number 3. You have a detection index d´3 that depends on (a) the signal-to-noise properties of the display, and (b) your own detection expertise You have an initial criterion c3 that depends on (a) the prior odds ratio, (b) the payoff matrix for the vote, Your observation results in z > c3 so your first vote is yes
-
Having heard the other members’ votes, you now need to make a second vote. Members 1, 2, and 4 voted { y1, n2, n4 } Calculate a new value for the ‘prior’ odds ratio and use that ratio to calculate a revised criterion. From Bayes’ theorem: p ( y1 | n) p (n2 | n) p(n4 | n) p(n) p (n | y1 , n2 , n4 ) p ( y1 , n2 , n4 | n) p(n) = = p ( sn | y1 , n2 , n4 ) p ( y1 , n2 , n4 | sn) p( sn) p ( y1 | sn) p(n2 | sn) p(n4 | sn) p( sn) The values for p(yi|n), p(yi|sn), p(ni|n), etc. can be calculated from your knowledge of the members’ { d1´, d2´, d4´, } and { c1, c2, c4, } -
5
Given several no votes, and few yes votes, the criterion would shift right
to a more ‘conservative’ value of zc.
Bayesian Jury Updating Strategy
f(x|sn), f(x|n) input (x)
λ(x)=
f(x|sn) f(x|n)
β=
(Vcorrect − no + V false − alarm ) p(n) ⋅ (Vhit + Vmiss ) p ( sn)
logλ(x)>logβ?
initial vote (yes, no)
6
Bayesian Jury Updating Strategy
f(x|sn), f(x|n) input (x)
λ(x)=
β=
f(x|sn) f(x|n)
(Vcorrect − no + V false − alarm ) p(n) ⋅ (Vhit + Vmiss ) p ( sn)
logλ(x)>logβ?
initial vote (yes, no)
other votes {y1,n2,n4}
βF =
(Vcorrect − no + V false − alarm ) p (n | y1 , n2 , n4 ) ⋅ (Vhit + Vmiss ) p( sn | y1 , n2 , n4 )
logλ(x)>logβ?
next vote (yes, no)
d´1, d´2, d´4, β1, β2 , β4
repeat
(knowledge of other voters’ expertise & bias)
Delphi Updating Strategy
f(x|sn), f(x|n) input (x)
λ(x)=
f(x|sn) f(x|n)
β=
(Vcorrect − no + V false − alarm ) p(n) ⋅ (Vhit + Vmiss ) p ( sn)
logλ(x)>logβ?
initial vote (yes, no)
other votes {y1,n2,n4}
βF =
(Vcorrect − no + V false − alarm ) p (n | y1 , n2 , n4 ) ⋅ (Vhit + Vmiss ) p( sn | y1 , n2 , n4 )
d´1 = d´2 = d´4 = µd’ β1 = β2 = β4 = µβ (knowledge of average voters’ expertise & bias)
logλ(x)>logβ?
next vote (yes, no) repeat
7
Conforming Strategy
f(x|sn), f(x|n) input (x)
λ(x)=
f(x|sn) f(x|n)
β=
(Vcorrect − no + V false − alarm ) p(n) ⋅ (Vhit + Vmiss ) p ( sn)
logλ(x)>logβ?
initial vote (yes, no)
other votes {y1,n2,n4} If sum of yes votes > no votes, decrease logβ else increase by some amount d´1, d´2, d´4, β1, β2 , β4 (no specific knowledge of other voters’ expertise & bias)
logλ(x)>logβ?
next vote (yes, no) repeat
Review of Possible Strategies Rational Strategy Revise your decision criterion according to the Bayes’ rule that optimally incorporates knowledge of other members’ sensitivity, criterion, and vote. Limited Knowledge (Delphi) Rational Strategy Revise your decision criterion according to the Bayes’ rule but use other members’ votes and average member sensitivity and criterion. Conforming Strategy Always shift your criterion in the direction of the majority vote; if tie, go with the majority of other members.
8
Expectations About Juror Parameters Jury size More jurors -> higher performance More jurors -> effect on hang rate? Juror expertise (incl. weight of evidence) Higher expertise -> higher performance Initial bias of Jury Deviation from neutral -> effect on performance? -> effect on decision bias Juror diversity: Between-juror correlation lower correlation -> higher performance Variance in member expertise higher variance -> higher performance Variance in member bias higher variance -> effect on performance?
Expectations About Jury Process Variables Majority required for decision ½, 2/3, ¾, unanimous -> effect on accuracy of initial (predeliberation) vote -> effect on accuracy of final vote -> effect on bias of initial vote -> effect on bias of final vote -> effect on hang rate
9
Decision accuracy as a function of group size & majority rule (Ideal combines continuous ests.) Group percent correct
1 0.95 Majority rule 0.9 0.5 0.85 0.666 0.8 0.75 0.75 0.7 0.995 0.65 ideal 0.6 0.55 With deliberation, the unanimous majority is best & 0.5 the simple majority worst. 3 4 5 6 7 8 9 10 11 12 mean c=0 Group size
µd' =1.0 σd' µc σ c =µd' σd'
= 0.33
G ro u p p ercen t co rrect
Comparison of optimum (Bayes’) updating and non-deliberation group groups (0.5 is the same). 1 0.95 0.9 0.85 0.8 0.75 0.7 0.65 0.6 0.55 0.5
Majority rule 0.5 0.666 0.75 0.995 0.5 0.666 0.75 0.995 3
4
5
6
7
8
9
10 11 12
With NO deliberation, the unanimous Group size majority is worst & the simple best.
mean c=0
Bottom three curves are for initial vote only.
10
Group percent correct
Effect of extreme initial juror bias… 1 0.95 0.9 0.85 0.8
Majority rule 0.5 0.666
0.75 0.7 0.65 0.6 0.55 0.5
0.75 0.995
3 4
5 6 7
8 9 10 11 12
Group size
mean c=-3 or +3
A unanimous rule reduces the negative effect of juror bias.
Group percent correct
Comparison of optimum and Delphi updating rules. 1
Majority rule
0.95
0.666
0.9
0.75
0.5
0.995
0.85
0.5
0.8
0.666
0.75
0.75 0.995
0.7 3 4 5 6 7
8 9 10 11 12 mean c=0 Group size The Delphi and Conforming rules are below opt andthe delphi
performance of the Bayes’ updating rule.
11
Hang prob. as a function of size, majority rule, & juror bias. -3
(Hangs defined as More than x ballots)
-2
m=12
0.015
-1.5 -1
0.01
-0.5
0.005
0
0
0.5 0.5
0.6
0.7
0.8
0.9
1
1
Majority rule
1.5 2 -3
0.02 Prob. of Hang
Few hangs; more hangs with smaller juries and unanimous rules. Parameter is initial µ c
Prob. of Hang
0.02
-2
m=6
0.015
-1.5 -1
0.01
-0.5
0.005
0 0.5
0 0.5
0.7
1
0.9
1.5
Majority rule
2
Accuracy as a function of majority rule and juror bias. 4
Initial µ c
3.5
-2
Group d'
3
-1.5
0 ±0.5
2.5
-1 -0.5
±1
2
0 0.5
1.5
1
±1.5
1 Best performance is with neutral bias and strict majorities. 1.5 A strict0.5 rule mediates the negative effect of juror bias. 2 0 0.5
0.6
0.7
0.8
Majority rule
0.9
1 size=12
12
Decision bias as a function of majority rule and juror bias. Initial µ c
5
-3
4
-2
Group criterion Group criterion
3
3 2
2
-1.5 -1
1
-0.5
0 -1
0 0.5
0.6
0.7
0.8
0.9
1
1
-2 -3
-2 -3
0.5 1.5 2
-4
3
-5 Majority rule
size=12
Stricter majorities mediate the effects of non-neutral biases.
Juries & Strict Majorities The major consequence of a strict majority rule is improved accuracy (not a more conservative decision). A strict rule may accomplish this by fostering deliberation (consideration of other members’ opinions). The second effect of a strict majority rule is to reduce the effect of juror bias on the accuracy and bias of the jury’s decision. It appears to be a near optimal system, so long as (a) jurors employ a sensible updating strategy, (b) jurors’ estimates are not correlated (in a non-uniform way), and (c) jurors are motivated to make the most accurate decision. If it ain’t broke…
13
References Pete, A., Pattipati, K.R., & Kleinman, D.L. (1993). Optimal team and individual decision rules in uncertain dichotomous situations. Public Choice, 75, March, 205-230. Sorkin, R. D. & Dai, H. (1994). Signal detection analysis of the ideal group. Organizational Behavior and Human Decision Processes, 60, 1-13. Sorkin, R. D., Hays, C.J., & West, R. (2001). Signal detection analysis of group decision making. Psychological Review, 108, 183-203. Swaszek, P. E., & Willett, P. (1995). Parley as an approach to distributed detection. IEEE Transactions on Aerospace and Electronic Systems, 31, 1, 447-457. Viswanathan, R. & Varshney, P. K. (1997) Distributed detection with multiple sensors: Part I—Fundamentals, Proceedings of the IEEE, 85, 54-63.
Questions?
Effect of bias in juror variance on percent correct 1 0.95
Group percent correct
0.9
0.99 0.75
0.85 0.8
0.5
0.75 0.7
σc = 0.33 σc = 0.66
0.65 0.6 0.55 0.5 3
4
5
6
7
8
9
10
11
12
Group size
14
