The k-Ratio Multiple Comparisons Bayes Rule for the ...

The k-Ratio Multiple Comparisons Bayes Rule for the Balanced Two-Way Design Author(s): Gene Pennello Reviewed work(s): Source: Journal of the American Statistical Association, Vol. 92, No. 438 (Jun., 1997), pp. 675684 Published by: American Statistical Association Stable URL: http://www.jstor.org/stable/2965715 . Accessed: 03/11/2011 11:14 Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected].

American Statistical Association is collaborating with JSTOR to digitize, preserve and extend access to Journal of the American Statistical Association.

http://www.jstor.org

The k-Ratio Multiple Comparisons Bayes Rule forthe Balanced Two-Way Design Gene PENNELLO A comparisonwise alternative to thet testing ofmultiple is thek-ratio Thismethod comparisons Bayesrulemethod. is comparisonwisein thatthetestofa meandifference doesnotdependon thepriorchoiceofthenumber ofotherdifferences tested, but inthatfortheone-way thecritical t valueofthetestrisesas thebetween-means yetis F protective design, F ratiofalls.In this articlethek-ratio method is extended tothebalancedtwo-way is limited todifferences between columnentries design.Attention within a rowofthetwo-way meantable(orviceversa).Forsucha difference, thek-ratio ruleF protects of against homogeneity thecolumneffects anddependson thecorresponding difference overall rows.Dependence on themarginal marginal difference increasesas theinteraction F ratiodecreases. Whenpositive, themarginal difference causesthecriticalt valuefordeclaring in thepositivedirection to be less stringent thantheone fordeclaring in thenegative significance direction. The significance critical t valueF protects morestringent inbothdirections whenthecolumnF againstcolumn-level homogeneity bybecoming F ratiois accordingly ratiois loweredandtheinteraction loweredto maintain thesamedegreeof marginal In the dependence. 22 design, thecritical t valuesaremuchless dependent on themarginal difference dueto therelationship between themarginal andtheinteraction F ratio. difference KEY WORDS: Adaptive Additive FullBayes; quadrature; losses;Comparisonwise; Exchangeability; Experimentwise; Familywise; Hierarchical model;Numerical integration; Shrinkage.

CarmerandWalker(1982)gavea compelling reasonfor not resorting to the familywise approach. They argued that andpractically One ofthemoretheoretically interesting if it is appropriate to apply the comparisonwise approach to is thatofwhether important dilemmas inscientific inference in separateexperiments, thenit followsthatit to use a comparisonwiseor a familywiseapproachto mul- differences is just as appropriate to apply this approach to differences tiplecomparisons. Thatis, shouldone limitthemaximum in if that are placed a single experiment, this weredone ortype1 errors, offalsesignificances, expected proportion merely to conserve experimental units. Similar arguments or to ae per familyof comparisons to ae per comparison have been made by O'Brien (1983), Rothman (1990), and Researchers oftenapplythecomparof similarstructure? Saville (1990). literature has isonwiseapproach, yetmuchofthestatistical A moreformalargument comesfromdecisiontheory. beenconcerned withmaximizing powerunderthefamilyas thejoint The multiple comparisons problemis defined is thek-ratio wisecriterion. One notableexception Bayes one of solvingmultiple whereeach component problems, rulemethod (DuncanandDixon1982),whichis a comparis tomakeinference ona difference uncomponent problem comparisons. isonwisealternative tothet testing ofmultiple dertest.If thedecisionlossesforthecomponent problems Thek-ratio method yieldsa jointBayesrulefordetermining are additive, thenthejointBayesruleis thesimultaneous theindividual ofdifferences between observed significances to eachcomponent application problemof theBayesrule meansnotonlyfromthe means.The theory encompasses alone(Duncan1955;Lehmann forthatproblem considered For 1957).In otherwords,if thecomponent butalsofrommorestructured designs. one-way design, losses are addithe tive,thentheoptimaltestfora difference instance, Brant, Duncan,andDixon(1992)havederived (in thesenseof k-ratio rulefortesting severaltreatments againsta control. minimizing thejointBayesrisk)is optimalwhether or not In thisarticleI extendthemethod tothebalancedtwo-way anyotherdifference theoptimal is beingtested.Therefore, designwithinteraction. is thecomparisonwise oneofchoosing approach aeon a per methodanditsre- comparisonbasis irrespective To morefullyappreciate thek-ratio of m. tothecomsults,ithelpstoreviewthestandard objections modelsthecomponent The k-ratiomethodspecifically The standard andfamilywise ob- losses as additiveand is therefore parisonwise approaches. a comparisonwise apjectionto thecomparisonwise approachis thatif a large proach.Component loss is specified by a ratiok of the m of meandifferences is tested,thenan lossesduetotype1 andtype2 errors, enoughnumber whichtakestheplace is expected of specifying unacceptably largenumber maeoftype1 errors oa.Popularvaluesof k are50, 100,and500, Thisobjection has becausethesearecomparable whenthetruemeansarehomogeneous. to thetraditional aevaluesof a .10, .05, and .01 in terrms led to thefamilywise approachforwhich,unfortunately, criticalregions of theresulting similarstatement maybe madein termsof type2 errors, in thedegenerate two-means case or thesingle-difference or falseinsignificances, whenthetruemeansareheteroge- case (WallerandDuncan1969,1972). neous. The k-ratiomethodmodelsthelevelswithineach factheuse of torof a designas exchangeable a priorithrough model usedfor the random-effects appropriate commonly Pennellois Cancer andBiostatistics 1. INTRODUCTION

Gene TrainingFellow, Epidemiology NationalCancerInstitute, Rockville,MD 20892. This workis based on the author'sPh.D. thesis,whichwas completedunderthedirectionof David B. Duncan at OregonState University. The authorthanksDavid B. Duncan, RobertE. Tarone,the editor,associate editor,and two refereesfor their helpfulcomments.

675

In the Public Domain Journal of the American Statistical Association June 1997, Vol. 92, No. 438, Theory and Methods

676

Journalof the AmericanStatisticalAssociation,June 1997

Table 2. Analysisof Varianceof Table 1 variancecomponent modelanalysis.Thisexchangeability ingyieldsfora difference betweenone-waymeansa comSource df F Mean square p value ponent Bayesrulethatadoptstheliberalism ofthecomparReagent 3 40.0 10.0 .001 isonwiset testwhenthemeansappearheterogeneous (i.e., 2 Catalyst 24.0 6.0 .016 whentheF ratioforthesemeansis large),butadoptsthe 6 Interaction 14.0 3.5 .031 conservatism ofa familywise t testwhenthemeansappear Error 12 4.0 homogeneous (i.e.,whentheF ratiois small)(Wallerand Duncan1969,1972).ThisF protection makestheone-way rulenotsubjectto thestandard objections to thecompar- betweenthereagentand catalystfactors(p value > .01) andclearheterogeneity amongthemarginal reagent effects isonwiseandfamilywise approaches. k-ratio in these rulefigures Thetwo-way rulederived hereis fordifferences between (p value= .001).Thetwo-way by makingthesignificance of a difference meansatdifferent levelsofonefactor butatthesamelevel considerations meansat different reagent levelsbutthesamecatofthe,otherfactor. Thesedifferences aretypically ofmost between of interest wheninteraction is nonnegligible. k-ratio rulesfor alystleveldependinlargepartona liberalinterpretation ofthemarginal difference overall catalysts. othertypesof differences weregivenin thethesisof Sun thesignificance difference is less varying andthuscan lead (1981).A program forcomputing k-ratio critical t valuesfor Thismarginal to a significant result that otherwise notbe obtained, would thesedifferences is inthejasasoftware collection onStatlib. Theprogram alsocomputes interval andpointestimates for evenfroma .05-levelt test. In contrast, the two-wayfamilywise rulesof Johnson thesedifferences, whichwillnotbe discussedhere.For a andWallace(1965),andMiller of suchestimates, discussionof thederivation see Dixon (1976),Kurtz,Link,Tukey, behaviors, be(1976)andDixonandDuncan(1975).Forsucha difference,(1966,pp. 56-58) do nothavesuchadaptive andconsequently treatfactor thecomponent BayesruleF protects againsthomogeneouscausetheyarenon-Bayesian as fixed,notrandom, as in thek-ratiomethod.It effects forthefactorwhoselevelsarebeingcompared, but effects is the treatment of effects as random thatenablesomnibus it also dependson thecorresponding marginal difference like F measures ratios and other relevant datalikemarginal overall levelsof theotherfactor, as the so increasingly differences tobecomeinvolved inthek-ratio decisionrule. F ratiodecreases. interaction Section2 briefly the the previews resultsthat two-way 3. THE ADDITIVE, LINEAR LOSSES MODEL rulegivesforan example.Section3 reviewsthek-ratio adis defined as thejoint comparisons problem ditiveloss structure. Section4 beginsdeveloping thetwo- Themultiple one 6 one of the of for each true difference tested making then the and wayrulebyderiving studying Bayesrulewhen three decisions: following thevariance areknown. Thissectionalsogives components an empirical are D_: 6 < 0, Do: 6 unranked Bayesrulewhenthevariancecomponents withrespectto 0, unknown. Section5 fullydevelopstheruleforunknown or D+: 6 > 0. (1) variancecomponents withtheuse ofJeffreys' indifference thek-ratio methodappliesthedepriors.Section6 appliesthefullydeveloped ruleto theex- In solvingthisproblem, fora largeclass cision of Lehmann theory (1957) developed theresults tothoseobtained from ample,compares two-way of multiple-decision problems. ofclassicalone-waymultiple extensions comparison rules, and modifies theexampleto studychangesin therule's 3.1 The AdditiveLosses Model forthe behavior.

ComponentProblem

A keyfeature ofthek-ratio method is thatitbreaksdown the the into component problem (1) mini-multiple comparBeforederiving thetwo-way derule,itmayhelptofirst HL: 6 > 0 against isons problem of simultaneously testing scribeitsgeneralcharacteristics an example.Conthrough 0 < 0 againstAR: 6 > 0. Thisbreak siderthe resultsgivenby Draperand Smith(1981, pp. AL: 6 < and HR: 6 downintoleft-andright-tailed is interval testing problems four rateexperiment 439-440) of a production involving doneto avoidtesting H: 6 = 0, thepointnullhypothesis the combinations of which reagentsand threecatalysts, I calltheseprobas falsea priori. whichis oftenrecognized werereplicated twice(Table 1). The anal-ysis of variance lemssubcomponents ofthecomponent anddenote problem, (Table2) indicatesless thanhighlysignificant interaction thenulldecisionsto notrejectHL andHR (i.e.,to accept all valuesof 6) by DL and DI, and thealternative deciTable 1. Draper and Smith's Production Rate Observations 0 HL HR AL: sions to 6 and and to < reject (i.e., accept withMeans in Parentheses AR: 6 > 0) byDL andD1R. Catalyst The pairsof subcomponent decisions(DL, DR), (DL, 1 2 Mean DR), and(DL, DR) correspond 3 tothecomponent Reagent decisions and of (1). The of the of 6 intersection values D_, Do, D+ A 4 6 (5) 11 7 (9) 5 9 (7) 7 acceptedbythecoordinate decisionsofeachofthesepairs B 6 4 (5) 13 15 ( 14) 9 7 (8) 9 C 13 15 (14) 15 9 (12) 13 13 (13) 13 is preciselytheset of valuesof 6 acceptedby a compoD 12 12 (12) 12 14 (13) 7 9 (8) 11 nentdecision.Forexample, theintersection of valuesof 6 Mean 9 12 9 (10) acceptedbyDLi andDORis D_: 6 < 0. 2. A PREVIEW

Pennello:k-RatioMultipleComparisons

677

theintangible In theterminology of Lehmann(1957),thefourth pos- former seemto morecloselyapproximate without beingoverly in manyproblems sibledecisionpair,(D , D+R), is inconsistent, becauseits trueloss function thedeciForexample,whenmakingerroneously coordinate decisionsacceptno commonvaluesfor6. The specific. toscorethelosswhen Bayesrulesforthesubcomponent problems are compati- sionD+, itis oftenmoreappropriate thanequalto theloss when makethese 68 -2 as twicethelossrather ble if and onlyif theycannotsimultaneously andthecommon coordinate decisions.I use theseterminologies throughout.6 = -1. Moreover, linearlossesfortesting yieldequivalent inlossesforestimation If theBayes rulesforthesubcomponent problemsare modelofquadratic (Dixon1976;DixonandDuncan1975). thentheBayesruleforthecomponent compatible, problem tervalestimates oftesting HL: 6 > 0 is verysimpleif theloss due to eachcomponent decision Thelossesfortheleft-tailed problem D is thesumofthelossesdue to thecoordinate The loss due decisions againstAL: 6 < 0 aredefined symmetrically. ofthedecisionpair(DL, DR) thatcorresponds toD. Under to a decisionDL is thismodelofadditive subcomponent losses,thecomponent LL(DL, 8) = LL(6)I(DL = DL) + L (6)I(DL = DL), BayesruledecidesD ifthesubcomponent Bayesrulesdecide (DL, DR) (Lehmann1957). whereLL(6) - LR(-6) and LL(6) = LR(-6). esti-, For example,supposethatd is theGauss-Markov mateof 6 withknownvariancegrdand theBayes critical 4. THE TWO-WAY RULEWHENVARIANCE and d > 1.969d. regionsforHL and HR are d < -196Jd COMPONENTSARE KNOWN becausetheyaredisThesecritical regionsarecompatible randomeffects modelby Denotethebalancedtwo-way joint.Underadditivesubcomponent losses,thecomponent ab independentsamplemeans yij ,- N(uij ), wherei BayesruledecidesD_ ifd < -1.969d, Do if Idl < 1.96d, 17 . . . , a and j = 1, . . .,b, decomposition pij = 0 + ai + and D+ if d > 1.96Cd. Thiscomponent rulecombinesthe 0 and + N(Oo,or),aii N(0,02), 1j resultsof one-tailedintervaltestsintoa two-tailed rule 3j Yij, andpriors forall C72), independent N(0, C2), mutually N(0, Yijfor8. i andj. Let thesamplemeansbe averagesof r replicates withvariance 2, so that -2 = c2/r.LetA,B, andC denote 3.2 The AdditiveLosses Model forthe JointProblem }, {aj}, to thefamilies of effects factors {aS corresponding of sev- and {'yj }. Underthismodel,theeffects For a multiple comparisons problemconsisting within eachfactor an analogousstatement can be areexchangeable. eralcomponent problems, are made.If theBayesrulesforthecomponent problems I am concernedwith the true difference 612,j thenthe andthecomponent lossesareadditive, compatible - A2j betweenmeans at, withoutloss of genPlj is thesimultaneous Bayesruleforthejointproblem appli- erality,levels 1 and 2 of factorA and level j of cationto each component problemof theBayesrulefor factorB. Considerthe variancecomponentsknown theuse for now. In this case the sample mean vectory = alone.Thisresultsupports thatproblem considered of a comparisonwise approachto multiple comparisons. forthetrue (Yll, Y12.. *Ylb. Y21, Y22, . ., Yab) is sufficient Generally, component Bayes rulesare compatible (i.e., means a = (tll, p12, ... ), lb,J 21i),22,. .... ,iab), and thus loss Bayesrulesfor612,jcanbe developed cannotmakeinconsistent decisions)ifthecomponent withy. bystarting I am explicitabout structures are definedsymmetrically. in Section4.5. conditions thesesymmetry 4.1 General Formulasforthe Right-Tailed a

SubcomponentBayes Rule

3.3 The LinearLosses Modelforthe HR: 612,j < 0 theright-tailed oftesting Consider problem Problems Subcomponent againstAR: 612,j > 0. Let o be a decisionrulethatmaps For the right-tailed subcomponent problemof testing from tooneofthe DR andD+ for612,j y possibledecisions HR: 6 < 0 againstAR: 6 > 0, theloss due to a decision The nulland criticalregionsof o are RR {y: p(y) in generalas DR canbe defined D } andR+ = {y: 9(y) = DR}. to showthatunderthelinearlosses It is straightforward = = = + L(DR, 8) LR(6)I(DR D+) LR(6)I(DR D-R) model(2), theBayescriticalregionis I call LR(6) and L2R(6) whereI is theindicator function. the type1 and type2 loss functions,because theyare the

j

j

d>k

(3) {Y: I[w(&Iy) r(1y) d/ ofthesetypes. lossesdueto errors o -~~~~~00 Inthek-ratio thetype1 andtype2 lossfunctions method, where r(.IY)is the posteriordensityof 612,j giveny. are linearin 6, with in (3) is theposterior oddsofAR relTheratioofintegrals LR (6) -k, 1611(6< O) ativeto HR weighted by thelinearterm8. Thisweighted oddsis thesmallestvalueof k forwhichy lies posterior and as the in the criticalregionand thuscan be interpreted to k. Lr(R) -k21i11(8> 0), (2) "p value"relative Supposethat812, jIY N(E, V). A changeofvariablein forsomek1 > k2 > 0. Theratiok-k1 /k2 givesthemethod (3) to thestandardized value(8 - E)/V'/2 yields its namesake.The linearlosses(2) are preferred overthe (4) constant lossesL1R(8) k1 and L2R(8)= k2,becausethe R+={z: M(z)/M(-z) > k}, -

RR

678

Journalof the AmericanStatisticalAssociation,June 1997

where z = E/V1/2 and M(z) = g(z) + zG(z) is a func- where c = ([SA/SC + (b - 1)]/[bSc])1/2 and m = andcumulative dis- (SA/SC - 1)/b1!2 are nonnegative.As the true interactionoftheN(0, 1) probability density posterior odds tionF ratio4c5increasesto IA, SC increasesto SA, and tribution functions g and G. The weighted withz; thus M(z)/M(-z) increasesmonotonically theslope m decreasesto 0. Thus an increasein interaction causesthecritical region(9) tobecomelessdependent (5) onthestandardized {z: Z > Zk}, R+ difference marginal Z12,. and,therefore, more dependent on the standardized Gauss-Markov differ= k. The number Zk iS whereZk satisfies M(zk)/M(-Zk) ence Z12jthecritical valueforthestatistic z. Forthecommonly used If SA andSc aredecreased bythesameproportion, then specification k= 100,Zk = 1.72. theslopem remainsunchanged buttheintercept CZk inthecritical creases,making valueCZk - mz12,. increaseuni4.2 The Two-WayRight-Tailed Subcomponent on set of formly the valuesforZ12,. Thisuniform possible Bayes Rule increaseinconservatism hasintuitive appeal,becauseas SA of 612,j giveny dependson The posterior distribution A effects decreases, the factor become morehomogeneous. theGauss-Markov difference d12J = Ylj - Y2j; thecorresponding marginal difference dl2,. = (Y. - Y2.)/b, where 4.3 The Left-Tailed SubcomponentBayes Rule the Yi. = Ebl yij; and,by wayof thesetwoquantities, Becausethelossesfortheleft-andright-tailed subcomA shows ponentproblems interaction contrast dc = d12,J- dl2,.. Appendix theBayescritaredefined symmetrically, that ical regionfortheleft-tailed problem is, from(9), 812,jlY -N(SAdl2,

+ Scdc, (SA + (b-1)Sc)u2/b),

(6)

whereSA = 1 - 1/IA and Sc = 1- l/lc are shrinkage 2 factorsthatdependon the"true"F ratiosIA = and 4c = u2/U2 of the expectedmeansquaresC2 ?U2 forfactors A andC to the br?2 + a 2and a2 = r 2 + expectedmeansquareo2 forerror,and whereCd = 2a2 iS the varianceof dI2,J.

R_ = =

Z{Z21,J:Z21,j > CZk - mZ21,.} {Z12,J:

Z12,j
Zk(SA + (b - 1)SC) 1/2 the left-tailed critical value.Fromthepreceding paragraph, where Z12,j = dl2,j/gd is the standardizedvalue of the leaststringent oftherequirements forpositive andnegative Gauss-Markovdifference d12,J,and z12,.= d12,./(ud/b1/2) significance is theone forwhichthesignof significance is thestandardized valueofthemarginal difference dl2,.. agreeswiththesignofthemarginal difference. The critical at thelimitsof 4c. For region(7) simplifies 4c = 1 (U2 - 0), whichis thecase of no interaction,4.4 The ComponentBayes Rule Sc 0, andtheregionreducesto If thesubcomponent lossesareadditive andthesubcomfrom rules are Section3.1, ponent Bayes compatible, then, RR = Z{z12,.:Z12,. > Zk/SA1}1 (8) thecomponent BayesruledecidesD_, DO,orD+ whenthe Bayesrulesdecide(DL, DR), (DyL,DoR) or For Fc = (A (9 = 0), whichis thecaseoffullinteraction,subcomponent if The (DyL, DR). subcomponent Bayesrulesarecompatible = SC SA, and theregionreducesto theBayescritical RL andRR aredisjoint. From(5), regions RR = {Z: Z > Zk}, andbysymmetry, RL = {Z: Z < -Zk}. RR+ ={Z12,J: Z12,j > Zk/SA1}1 ThusRL andRR+aredisjointif andonlyif Zk > 0. From TheBayesrulemakestheintuitive ofapplying thediscussion adjustment = k, which surrounding (5), M(Zk)/M(-Zk) thecritical dif- impliesz1 = 0, and M(z)/M(-z) increaseswithz. Thus valueZk/S112to thestandardized marginal ference case butto thestandard-Zk > 0 ifandonlyif k > 1. Fortunately, Z12,.in theno interaction k = kl/k2 setting ized Gauss-Markov difference Z12,jin thefullinteractionto a valuegreater than1 is reasonable, as thismeansthat case. Furthermore, thecriticalvalue zk/SA'2 risesas the theloss due to a type1 error(k161)is perceived as more trueF ratio'IJAfalls;thatis,itbecomesmoreconservativeseriousthantheloss dueto a type2 error(k26I). as thefactor A effects becomemorehomogeneous. ThisF Thusfork > 1, thecomponent BayesregionsareR_ protection is attractive, because812,j comparestwolevels RL noR= RL, Ro =RLn ROR,andR+ = RLn RR RR. offactorA. From(9) and(10),theseregionsare onemaywritethecritical Generally, regionR+Rin terms of a criticalvalueforZ12,; thatdependslinearly on Zl2,.. R-={z12,J: Z12,j K CZk-

mZ12,.

},

(9)

RO={z12j:

-CZk-

mz12.-

< Zl2j

?

CZk - mz2,.

},

Pennello:k-RatioMultipleComparisons

679

most j = 1,2,... , b. Therefore,a negativecriticalvalue

and

willnotbe utilizedon mostZ12,j. On theotherhand,if a negative criticalvalueis utilized on a negative Z12,j, thenthismaysignalone of twomodel A negative misspecifications. andlarge Z12,j anda positive 4.5 The JointBayes Rule thatzc = dc/(ad(b - 1)/b)1!2, thestandardized Z12,. imply Forthejointproblem ofmultiple ifthecom- valueoftheinteraction differences, contrast dc = dl2,j - dl2,., is large ponentlossesforthedifferences areadditiveandthecom- in absolutevalue.However,'bc is smallandis estimated ponentBayesrulesarecompatible, then,fromSection3.2, by the sum of squares of (a - 1)(b - 1) orthogonalconthejointBayesruleis thesimultaneous application ofcom- trasts, of whichzc is one.Thustheestimate of 'bc may ponentBayesrulesto thecomponent The com- be muchlargerthanbc itself.If not,thenz4 maybe far problems. ponentrulesare compatible if foreach inconsistent joint largerthanthesquaresoftheothercontrasts, whichis not theintersection decision, ofthecorresponding decisionre- expectedunderexchangeability of theinteraction effects. gionsfromthecomponent Bayesrulesis empty. In sum,although a negative criticalvalueforpositivesigIn thecase oflinearlosses,a condition sufficient forthe nificance makessensewhentheeffects arenearlyadditive, component Bayesrulestobe compatible is thatthesamek, utilization ofsucha critical valuemayimplythatbc is ungreater than1,be usedforall components. Thissymmetrydervalued orthattherandom-effects modeldoesnothold. condition is in stepwiththeprecedence of theclassical one-waymultiple comparison rulesto set thecomponent o levelsto the same value.SupposethatZ12,j and Z23,j 4.7 An EmpiricalBayes Rule arepositivesignificant; thatis, supposethatthedecisions Supposethatthevariancecomponents areunknown and and > are > made.Thesetwodecisions 0 0 612,J errorof d12,3.An empirical 623,J imply thatSd is thestandard Bayes that613,J > 0. To be compatible, thecomponent Bayesrule ruleforthestudentized difference t12, = dl2,j/sd replaces for613,J shoulddeclareZ13,j as positivesignificant. If the bA,i4,bc and Z12,. in theformula(9) by the observedF decisions612,J > 0 and 623,J > 0 are made by the same k, ratiosFA and FC andthestudentized marginal difference then,from(7), that tl2,. = dl2,./(sd/bl/2). Thisrulehastwoshortcomings thefullBayesruleofthenextsectiondoesnothave.First, (SA - Sc)zI3,. + b1/2Sczl3,J an FA orFC lessthan1 renders a negative, andthusout-offactor or bounds, therulefails shrinkage SA Second, SC. (SA - Sc)z12,. + b1/2Sczl2,J to accountfortheinaccuracies of FA, FC, andtl2,. in estimating ObA, 4oc, andZ12,.. The fullBayesruledilutesthe + (SA - SC)Z23,. + b1/2 SCZ23, on FA,FC, andtl2,. by accounting dependence forthede> 2zk(SA + (b - )SC) 1/2. greesoffreedom associatedwiththesestatistics. R+ = {Z12,J:

Z12,j > CZk - MZ12,.}.

If k > 1, thenZk > 0, andtheforegoing equationimplies that(SA - SC)Z13,. + b1/2SCZ13J; > Zk(SA + (b -)SC)1/2 (i.e.,Z13,j is indeedpositivesignificant).

5. THE K-RATIORULE FOR UNKNOWN VARIANCECOMPONENTS

Whenthevariancecomponents are unknown, thesufficientdataare (y,s2), wheres2 is theerrormeansquare. In theright-tailed Bayes criticalregion(9), thecritical Recallthats2 is distributed as -2. x2(f)/f, wherex2(-)devalueforZ12,j is negativewhenZ12,. > cZk/m. In other notesthecentral distribution andf = ab(r- 1) chi-squared words,Z12,j maybe negative yetbe declaredpositivesig- is theerrordegreesof freedom. nificant. a negative Although criticalvalueforpositivesigTo construct a fullBayes rulethatmapsfrom(y,s2) nificance can be alarmingat first, it has intuitive appeal to component thek-ratio methodintroduces the decisions, fromtheperspective aof theno interaction = case (4c = 1). Jeffreys , indifference for prior (oc,J22a2, al2). When ] = - c2 for all j = 1,2,... , b, and thevariance In this case, 612,J are mutually a pricomponents independent theGauss-Markov estimate ofa, - ca2 is dl2,..Thuswhen ori,the random-effects whichis proportional likelihood, Z12,. is positive it makessenseto declareZ12,j to theproductof theconditional significant, densitiesf(ylt,au) and as positivesignificant of its value;thatis, for f(S8210-2) and thepriordensity regardless p(4la), yieldstheJeffreys criticalvalueof -oo. Indeed,the priorp(a) = o' 2 0__B20C2(Box andTiao 1973,pp.252, Z12,Jtohavetheeffective Bayescritical region(8) forthiscase dependsonlyon Z12,.- 332).Thefinalvariance forthegrandmean0 component Jo-2 It standsto reasonthatin a mildinteraction case,a nega- is takentobe infinity at a convenient stepinthederivation tivecriticalvalueforpositivesignificance can makesense of theBayesruleto effectively 0 givelocationparameter if Z12,. is positiveandlarge. theJeffreys indifference priorp(O) oc1. a negative valueofCZk - MZ12,. is unlikely Nevertheless, tobe utilizedtodeclarea negative Z12,j positive significant. To producea negativecriticalvalue,thetrueinteraction5.1 The WeightedPosteriorOdds forthe Right-Tailed SubcomponentProblem F ratioJ0 mustbe smalland Z12,. mustbe positiveand large,and theseoutcomesimplythatZ12,j iS positivefor For theright-tailed subcomponent problem, the Bayes 4.6

NegativeCriticalValues forPositiveSignificance

680

Journalof the AmericanStatisticalAssociation,June 1997

critical regionis RI ) W(y,s2 WY SE)

{(y, S2): W(y, S2) > k}, where

6f(ylo, o)p(4jf)f (s2 102)p(o-) d1udo=f[5>0] d,udaf tk(t12,,i FA, FB, FC, qA1 qB1 qC, f)} tl2,. = dl2,./(sd/bl/2), and tc = dc/(sd(b-

wheresd= 2s /r. AppendixB shows thatthe numerator = k. of W(y, si2), apartfromtermsthatcancelwiththedenom- The valueof tk is thatvalue of t12,JsuchthatW

inator, evaluatesto F

JFBvC/FC

FAv

(VQ)1/2Mh(Z)Q-h/2vqA/2-1

A program thatcomputes tk is availablebyelectronic mail (sendtheone-line from request"sendkr2way jasasoftware" to [email protected]). The program uses an adaptive basedon Simpson'sruleto numeriquadrature algorithm callyintegrate (11).

dVAdVBdvc ( 11) 5.4 The Componentand JointBayes Rules As inthecase whenthevariance areknown, components where V = (SA + (b - 1)Sc)/b, where Si = 1 - vilFi of the intersection subcomponent rules Bayes yieldsthe for i = A, C; Q (qAVA + qBVB + qcvc + f)/h, and as as k is greater component joint Bayes rules long where h = abr - 1; z = E/(VQ)1/2, where 1 than and all In the same for differences tested. thiscase E = SAtl2,./bl/2 + Sctc((b - 1)/b)1/2;and Mh(z) = the for component Bayes regions 612,j are gh(z)((h + z2)/(h - 1)) + zGh(z), where gh and Gh are the probability densityand cumulativedistributionRFA, FB, FC,qA qB, qc, f)}, {t2,J: t12j < -tk(-tl2,, functionsof a Student'st variablewith h degrees RO = {t12,J: -tk(-tl2,., FA, FB, FC, qA, qB, qc, f) of freedom.The integrand in (11) is proportional to (VQ) 1/2Mh(Z) timesa three-dimensional F density with ? t12,J < tk(tl2,, FA, FB, FC, qA, qB, qc, f)}, qA,qB,qo, and f degreesof freedom.The noncancel- and ing part of the denominator of W(y,s2) is identical, exceptthatMh(z) is replacedwithMh(-z). R+= {tl2,j: t12,j > tk(t12,.,FA, FB, Fc, qA, qB, qc, f)}. x vqB/2-1 vqc/2-1

5.2

Comparisonwiththe WeightedPosteriorOdds for KnownVariance Components

6.

k-RATIORESULTS FOR THE DRAPER AND SMITH DATA

The weighted odds W(y,s2) is muchlikethe 6.1 CriticaltValues posterior odds forknownvariancecomponents, Forthetwo-way in Section2, I havecalweighted posterior datadescribed thequantity M(z)/M(-z) givenin (4) forwhichz is de- culatedforeach difference betweenreagentsat thesame finedby (6). ThroughSA and Sc, W(y, s2) dependson levelofcatalyst theobserved t statistic, the100-ratio critical t valuet100,andtheweighted FA and FC in thesamereciprocal way as M(z)/M(-z) oddsW (Table posterior dependson BA and bc. Unliketheempirical Bayesrule 3). Forthesedata,therewerenonegative valuesfor critical of Section4.7, however, SA and Sc are precluded from positivesignificance, or vice versa,suchas theones disif it becomingnegativeby thelimitson VA and vc whenFA cussedin Section4.6. Thusa t is positivesignificant and FC are less than1. W(y,s2) is a dilutedversionof exceedsthegivenpositivet100andnegative ifit significant dividing M(z) byM(-z), however, largely becausebefore is exceededbythegivennegative a t is t100.Equivalently, dividing, W(y, s2) averagesthesimilartermsMh(z) and significant whenW exceeds100. Mh(-z), multiplied by (VQ)1/2, overpartof therangeof The criticalvalue t1oovarieswiththe corresponding thethree-dimensional F distribution. suchthattherequirement forsignifit statistic marginal fromM(z)/M(-z) in thatW(y, s2) de- canceis less stringent W(y, s2) differs whenthesignof t agreeswiththe pendson FB, whereasM(z)/M(-z) does notdependon signofthemarginal t. Catalyst column2 revealsthatposithetrueF ratioforfactorB, ~BB (ara: ? ?c)/aft. Al- tivemarginal t statistics (1.7,3.5,5.2) resultin liberalposthough itis difficult toseefrom(11),itcanbe inferred from itivet100values(2.2, 1.7, 1.4) butconservative negative thestudyof Bayesianpoolingby Box andTiao (1973,p. t100values(-4.7, -7.1). As a result, thedifference between 264) thatpartofthesumsofsquaresforB is beingpooled reagents C andA inthiscolumnis positivesignificant even

Pennello:k-RatioMultipleComparisons Table 3.

681

Criticalt Values t100and WeightedPosteriorOdds W forthe Draper and SmithData Catalyst 1

Reagent Difference B-A C-D D-B D-A C-B C-A

2

3

t

t100

W

t

t100

W

t

t100

W

Marginalt

0 1.0 3.5 3.5 4.5 4.5

2.2 2.2 2.2 1.7 1.7 1.4

2 13 570 1,945 6,680 18,343

2.5 -.5 -.5 2.0 -1.0 1.5

2.2 -4.7 -4.7 1.7 -7.1 1.4

160 1 1 182 1 130

.5 2.5 0 .5 2.5 3.0

2.2 2.2 2.2 1.7 1.7 1.4

5 160 2 10 438 2097

1.7 1.7 1.7 3.5 3.5 5.2

thoughtheobservedt is only1.5,wellbelowthecomparable.05-leveltwo-tailed and evenone-tailed Student'st deviates2.18and1.78(notshown).Thissignificance is due to theverylargemarginal t valueof 5.2,whichcausesan inordinately highlevelofpower.

bythepositiveandnegative t100values(Fig.2). Thesecritical t boundaries are nonlinearfunctions of themarginal t statistic, whereasin thecase of knownvariancecomponents,thecriticalz boundaries givenby (9) and (10) are linearfunctions ofthemarginal z statistic Z12,. The solidlinesdepictthecriticalboundaries givenby 6.2 ComparisonwithClassical Rules theF ratiosoftheDraperandSmithdata.I havealso disboundaries aftermakingtwochangesto Figure1 illustrates thechangesin thedecisionsthatare playedthecritical posterior odds. givenby thek-ratio rulefork = 100relativeto two-way theinputsfortheweighted ofsevenclassicalone-way extensions rulesatthecomparaLowering Fc. WhenFC is loweredfrom3.5 to 1.5 levelo = .05.The 100-ratio ble significance rulefinds more (withtheinteraction p valueraisedfrom.03 to .26), differences thananyoftheotherrules,eventhe significant the boundary lines on flatten, makingthedependence comparisonwise t test.Thisis duetothehighsignificance of the t rise. This marginal increased condependence F ratio,coupledwiththelessthanhighsignifthereagents curswiththeabsolutedependence on themarginal t F ratio.Together, icanceoftheinteraction theseoutcomes in the of the value limiting case additive effects model, indicatea considerable amountof marginal heterogeneity whichis oftenusedwhenFC is insignificant. thatis relevant to theirsignificance. amongthedifferences * The 22 Design. Whenthedegreesof freedom forthe It is thismarginal heterogeneity thatcausesmanyof the factorsare changedto qA = qB = qc = 1, t1oono differences to be foundsignificant. t2. longerdependson FA and FC, becauseFA 6.3

ComponentCriticalt Regions

I havedelineated as a function ofthemarginal t statistic theboundaries of thecomponent criticalt regionsformed Catalyst 2

1 Reagent Mean 100-Ratio

A B D C

A C D B

5

9 12 13 14 l l

5 12 14

3

A B D C 7

8

l

S

I

)2(b - 1). These re-

-

8 13

3~ A

F=3.5 CB

Z ll

IzIzIzI

(bl/2

=

l

F, U, DIIllI B, T, NK

and FC = t2

lationships resultin criticalboundaries thataremuch onthemarginal t compared lessdependent tothecritical boundaries fortheothercases. Surprisingly, the critical boundaries areno longermonotonic functions of themarginal t. The criticalvaluespeakin magnit of0. After thispoint,thecritical tudeatthemarginal

R

ql1I

-l:

-

2

0+ 0

R

223-0

I

Figure1. Comparisonon theDraperand SmithData ofthe 100-Ratio Rule With.05-Level Two-WayExtensions of Classical One-Way Rules. The classical one-wayrules are the F-protectedrule (F) due to Fisher (1935, sec. 24), the unprotected-For comparisonwiset-testingrule (U) difference (also knownas the least significant rule rule),the Bonferroni rules of Scheff6 (B) also due to Fisher (1935, sec. 24), the familywise (1953) and Tukey(1951) (S and T), and the multiplerange rules of Duncan (1955) and of Newman (1939) and Keuls (1952) (D and NKJ. The two-wayextensiontreats the 4 x 3 two-waymeans as 12 oneway means, as suggested by Neter,Wasserman,and Kutner(1985, pp. 725-729). For example, Tukey'scriticalt values are computed on the basis of the studentizedrange distribution for 12 means. Means are grouped by all levels of reagentat a fixedlevel of catalyst.Withineach group,means are orderedincreasingfromleftto right.A mean leftof anotheris significantly less thanthe otherifthepair is not underscored by a bar.

-5

R

-1

\8

1

-2-

\\

2

3

4

R

-3-

Figure2. 100-RatioDecision Regions as FunctionsoftheMarginalt. The decision regionsR_, Ro, and R+ forthedifference 612,j = i1j - A2j are delineatedbyboundarylines thatrepresent100-ratiocriticalt values forpositiveand negativesignificanceat particularvalues ofthemarginal t. Boundarylines are shown fortheDraperand Smithdata at theoriginal value of the interactionF ratio(3.5), at a lower value (1.5), and at unit values forthe factordegrees of freedom(i.e., the 22 design). All other aspects of the Smithdata are unchanged.

682

Journalof the AmericanStatisticalAssociation,June 1997

valuefornegative significance, forinstance, declines ifthemarginalmeanscorresponding to theseeffectsappear inabsolutevalueas themarginal t riseseventhough a to clump,as clumpingsuggestsan unanticipated multigroup higher marginal t valueindicates lessevidenceofneg- structurewithinthe factor.Yet even if the random-effects ativesignificance. Thisdeclineis due to theincrease model is not preciselycorrect,it can resultin a multiple in FC thateventually resultsas marginal t continues comparisonsBayes rulethatin manycases providesappealto rise.Formarginal t equalto 0, thisincreasein FC ing adjustments forthestructure of thetwo-waydesignand decreases thedependence onthemarginal t enoughto forthe multiplicity involvedin testingseveraldifferences offset ofless evidencefornegative itsindication sig- fromit. nificance. Beyond0, anincreaseinthemarginal t proAPPENDIX A: DERIVATIONOF KNOWN ducesa slightly less stringent criticalt value.A crude VARIANCES POSTERIOR DISTRIBUTION FOR 612,j of thesecriticalboundaries the approximation is pair of verticallinesgivenby thecomparable.05-level, Heretheknown-variances posterior distribution for612,3 given without loss of generality, for two-tailed Student's t deviates?2.18. Thisfactlends y and o, givenin (6), is derived, = b. In vector matrix j and denote the x notation, balanced a b twoto the common of support practice usingcomparisonway randomeffectsmodel by y N(i, a 2Iab), decomposition wisecriticalvaluesin 2n designs. '

+ X,ao + Xe/ + -y,and priors0 N((,, a2), a N(O, UlIa), 3 N(O,9 2Ib), and y N(O,O2Iab), where1, is an s x 1 vectorof Is, Is is an s x s identity matrix, ,

7. CONCLUDING REMARKS

=lab0

lb [lb In thisarticleI haveappliedtheadditivelossesmodel anda random effect modeltotwo-way meanstotestmulti. XiB(ab x b) ple differences betweenmeansat different levelsof factor X?(abxa) A of thedesignbutat thesamelevelof factorB. These lbJ ~~~~~~~L lbJ modelsyielda jointBayesruleforthedifferences thatis a / = (/31,v2,... v,/b)', and -y ,. a?a)', = likeunprotected comparisonwise, t tests, butis F protective . * ,"'lb,Y21, ,Yab)("/I,"/I2, of thefactorA effects. againsthomogeneity Additionally, First,transformy and ,t by premultiplying by an orthogonal theBayesruleadjuststo themarginal differences overall "Helmert-like" matrix in H, whichI defineas theab x ab matrix levelsof B, as illustrated in Figure1. A pairof boundary TableA.1 afteritsrowshavebeennormalized. Forinstance, the fromFigure1 is thepairforwhichFA is low- toprowoftheH matrix linesmissing is 1ab/(ab) /2.The a - 1 rowsafterthe thea spaceofdifferences; theb - 1 rowsafter eredandthenFC loweredaccordingly tomaintain thesame toprowcomprise On thebasisofthediscus- that,the/3space;andthefinal(a - 1)(b - 1) rows,the-yspace. degreeofmarginal dependence. oftherowsofthea and sionin Sec. 4.2,theseboundary lineswouldbe parallelto Therowsofthe-yspacearedotproducts the(b- I)st rowofthe-yspace(theninth butlie outsideofthesolidboundary linedelineating thetk ,3spaces.Forexample, ofthefirst rowofthea space(the valuesfortheoriginal data.Theselineswoulddemonstraterowshown)is thedotproduct secondrow)andthelastrowof the,3 space(theseventh row). thattheBayesruleprovides. clearlytheF protection Un- Thesethreerowsareimportant to thederivation. it is notclearthatsucha valueof FC exists, The vectorw = Htt = [wo,w' ,w,w,]' consistsof mean elfortunately, andtherefore I haveonlydiscussedF protection in terms ementwo and ab - 1 orthonormalcontrastsw,, w,, and wy in of theknownvariances case. the a,,3, and -yspaces. Note that[wo,w,e ,, w]' = H(labO + Table3 indicates valuesof k andal, X,ac + Xa,g3 + y) = that,at comparable thetkvaluescanrangefrombeingmoreliberalthanar-level critical t valuestobeingmoreconservative comparisonwise 3 + Hay, ab)1/2 L?| + b1/2 Hc a +la L ? 1/2+b thanoa-level criticalt values.This broad experimentwise (a) 0 0 Hoo rangeof valuesoccursbecausethecontext eachdifference of thetwo-waydesignplaysan has withinthestructure (A.1) rolein thedetermination ofitssignifextremely important -

L

-

icance.

where [hcvo,H j' = Ha, the a x a orthogonalHelmertmatrix

A summary of theBayesresultfora difference is the thatis thematrix oddsW. If W exceedsk,thenthedifweighted posterior ... 1 1 I1 ference is significant. ThusW actslikea "p value"relative 1-1 1 1 -2 00*- O to k. Of course,unlikethep valuefromthet testof the W adjustsfortherelative sizesof theF ratios, difference, therelevant and thedegreesof freemarginal difference, -(a-i) domassociatedwiththesestatistics. In thisregard,W is after its been rows have and[h,o, H>B]' = Hb. orthonormalized, themorecomplete summary. From(A.1),thepriordensity ofw is p(w) = p(wo)p(w,c)p(w0) Theadjustments madebyW areintuitively but appealing p(w,y),wherewo N((ab)1/2 p,abU2 + bU,2+ ao2 + o2), Wc, arepredicated ontherandom effects thattheef- N(0, (boJ2+ 702)Ia-i ), wB assumption t- N(0, (a, + o2)Ibh1 ) and w fectswithin eachfactorareexchangeable a priori.The in- N(O, 0y2I(a1l) (b1l}). teraction effects maynotbe exchangeable iftheBayesrule Let w =Hy = [w,w/ ,wvg,w]j and note that wlw declaresa negativedifference positivesignificant, or vice N(w, OIab). By applyingthe standardresultthatw N,p N(ws,V) imply wIw N(,up + Vp(Vp + versa.Theeffects ofa mainfactor maynotbe exchangeableVp) and wlw

Pennello:k-RatioMultipleComparisons

683 Table A. 1. The Helmert-like MatrixH BeforeOrthonormalization

it

-

it

it

...

1b

*--

b

-1 b

?b

...

b

lb

-2b

*--

b

b-

a-i

?b

rows b

1b ?b-2

1)

-2, ?b3

1i1,

7

1

1 1

1b

I

b-2

1-10b

1,1, -2, ?3

2 :

1-10' -(-)1-110 t,

-1,

?b-2

1,1,-2,?,b3

V) -' (w

-

1p),Vp(Vp + V)-1V),

1,

.

-2, ?3

b-1

~~~~~~~~~~~~~~rows

(-)*1b

1, ?b-2

11, 1-2,

b-2

:

O(-h1-' -1,

I

b-2

?b

?3

b1 b*3

rows

-2, -2,.4.3

the posteriordensityof w

w is 7r(wlw)= r(woIw)r(w,tIw) Iw)r(w tIw),wherewoIw 7r(wa 2

, N(SMWo, SM ), W I|W N(SAW&,SA 21Ia-1), W6W and N(SBWp, SBoYIb-1), w,lw N(SCw7, SCUyI(-1)(b-1)), whereSi = 1 - 1/4i fori = M, A, B, C, and 4M = (abro2 + brao2+ arcra+ ra )/2. is a linear combinationof The difference612,b = lb-t2b the firstelementof w, and the (b - 1)st elementof wy because = (2/b)1/2W (1), 612,b- 612,. +?c, where(612,. - l/b(Ql. -[t2.) = where tti. = and t2. -1/b(i. (c -b(lb _b=It,u

and marginal density m(tlaf) satisfies ? thestandard f(tIr,,a)p(rIcr)= 7r(rIt,ac)m(tIo).By applying

in (11), u =

resultthatw N(w,V) implyw N(tp,Vp) and wIw for t is + the marginal density V), N(pp, Vp = m(to la)M(t,

m(tla)

l