How to increase your chances of obtaining a ...

Journal of Clinical and Experimental Neuropsychology

ISSN: 0168-8634 (Print) (Online) Journal homepage: http://www.tandfonline.com/loi/ncen19

How to increase your chances of obtaining a significant association between handedness and disorder D. V. M. Bishop To cite this article: D. V. M. Bishop (1990) How to increase your chances of obtaining a significant association between handedness and disorder, Journal of Clinical and Experimental Neuropsychology, 12:5, 812-816, DOI: 10.1080/01688639008401022 To link to this article: http://dx.doi.org/10.1080/01688639008401022

Published online: 04 Jan 2008.

Submit your article to this journal

Article views: 16

View related articles

Citing articles: 10 View citing articles

Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=ncen20 Download by: [Radcliffe Infirmary]

Date: 24 April 2016, At: 04:54

Journal of Clinical and Experimental Neuropsycholcgy 1990, Vol. 12, NO.5 , pp. 812-816

COMMENT

How to Increase Your Chances of Obtaining a Significant Association between Handedness and Disorder*

Downloaded by [Radcliffe Infirmary] at 04:54 24 April 2016

D.V.M. Bishop University of Manchester ABSTRACT Handedness inventory data are peculiarly prone to be used to classify subjects in a post hoc fashion. Data from a comppter simulation are used to show that if this procedure is adopted, significance levels can be seriously misleading: pnder certain con4tions the odds of obtaining “significant” results by chance are as Pigb as 2 3 . Specification of cutoffs after inspection of the data is probably @e most important source of spurious associations between handedness and disorder.

There is no shortage of studies reporting associations between hapdedness aud developmental disorder, but there is little consistency from one stpdy to another. For example, developmeqtal dyslexia has been linked to mixed hqpd preference (Harris, 19571,strong left-hapdedsess (Geschwipd & Behan, 1982), inconsistent right-handedness (Schachter, Ransil, & Geschwind, 1987) and both left- and strong right-haqdedness (Apnett & Kilsbaw, 1984). There are also maqy studies that report n o association (see review by Bishop, 1983). The range of discrepant findings concerning handedness and disorder is quite perplexing. W e are taught to use tests of statistical significance to tell which findings are likely to be robust, yet in this field failures to replicate statistically significant associations abound. Soper, Ciccbetti, Satz, Light, and Orsini (1988) discussed several reasons for lack of replicability of neuropsychological research. One factor is a publication bias that operates in favor of s[atistically significant results, which when coupled wiih the use of spa11 sample sizes leads to a high probability of spurious associations being published. However, such biases are not specific to handedness research. The impression is that inconsistent findings on this topic are disprqpor-

* Author’s address for reprints: Dorothy V.M. Bishop, MRC Senior Research Fellow, Department of Psychology, university of Manchester, Oxford Road, Manchester M13

9PL, Esgland.

Accepted for publication: September 19, 1989.


HANDEDNESS AND DISORDER

813

tjonglely commoq, raising the question of whether there is something special qbout handedness that generates spurious associations. One possible reason is the tendency to yse handedness as what Fairweather (1976) bas termed a *‘bonus factor”. Measurement of handedness by questionnqiqe i s quick and eqsy, so there is little cost involved in including an assessment of hand preference in psychological studies. A bandedness questionnaire may be “tbrowq in”, even if this is not the main focus of the investigation, just in case sowething interesting emerges. Results on spch an incidental variable are likely to be ignored unless there is a significant association with disorder. Thps researchers as well as journal referees may unwittingly conspire to the bias against publication of negqtive findings. A second characteristic of handedness studies is that handedness questionnaires give a measure that can be used to categorise individuals in a variety of wqys. Soper et al. (1988) noted that post hoc selection of cutoffs for subdjviding groups can give misleading results, and they presented illustrative examples from their own data. I want to take their argyment a step further by using sirnulqterd data to demonstrate that this is not a purist’s quibble, and that the conseqyences of setting cutoffs only after examiniqg the data are serious. It is ironical that while the aim of the hqndedness inventory was to move awqy from treating handedness as a categorical variable to measure it in a quantitative fashion (Oldfield, 1971), in practice, nearly every study that uses such qHestionnaires goes on io subdivide subjects into handedness groups and then apply contingency table analyses. Yet there is no agreement as to where cutoffs shoqld be placed to define groups. If we Used a five-item handedness inventory to count the npmber of actions performed wit0 the right hand. we could produce a jusljfication for (i) distinguishing extreme right-handers (scoring 5) from all the rest (0 to 4); (ii) distinguishing extreme left-handers (scoring 0) from all the rest ( I to 5); (iii) distinguishing left-biased individuals (scoring 0, 1 or 2 ) from righi-biased individuals ( 3 , 4 or 5 ) or (iv) distinguishing mixed-handers (scoring 1 to 4) from those with strong hand preference (scoring 0 or 5 ) . Indeed, tbere are very few ways of specifying cutoffs to +vide a group into two on the basis of a hqndedness inventory that would not fit some theoretical formulation. Some researchers are quite blatant about choosing cutoffs to maximize differences betweeq groyps before conducting statistic41 qnalyses. Others may adopt this procedyre without stating so explicitly. Yet post hoc selection of cutoffs contrav e t ~ e sthe underlying assumptions of statistical tests. I ran a simple simulation to generate bypolhetical data from a 5-item bandedinventory for two groups. For both gropps, there was a probability of .8 that a n individual would be right-biased, whicb y e a n t that for each item on the invenp r y , \be independent probability that the right band would be preferred was .9. There was a probability of .2 that the inqividual would be left-biased, in which Fase for each item the probability of preferring the right hand was .2. The aim was yo simulate distributions of inventory scores similar to those obtaine4 with reql spbjects: the extent of bias in left-biased individuals was less than that of


814

D.V.M.BISHOP

right-biased individuals to reflect the influence of an overall environmental bias in favor of the right side. The short BASIC program that ran this simulation is given in the appendix. The simulation was run 100 times, with a sample size of 50 subjects per group. An initial check was run on the output to confirm that the randomization did indeed produce the expected distribution of left- and rightbiased individuals. The overall proportion of left-biased individuals was .2043, very close to expectation. I then inspected the output from each simulation, and placed one or two cutoffs to divide each group into two handedness ranges (e.g. right- versus non-right-handers; mixed vs consistent handers, etc.), so as to maximize the difference between groups. The Fisher Exact test was then applied to compute the probability of the 2 x 2 contingency table being generated by chance. The first 10simulations are shown for illustration in Table 1. In 23 out of 100 simulations the associated probability was .05 or less, and in four of these it was less than .01.As the Fisher test is essentially a one-tailed test, the obtained proportions are approximately double what would have been found had the cutoffs not been set post hoc to capitalize on chance. If we regard group A as dyslexic and group B as control, 82% of these “significant” associations were

Table 1. Simulated distributions of hand preference scores for two groups N items performed with R hand

RUN 1

groupA group B RUN2 groupA group B RUN3 groupA group B RUN4 groupA group B groupA RUN5 group B groupA RUN6 group B RUN7 groupA group B groupA RUNS group B RUN9 groupA group B RUN 10 groupA group B

0

1

2

3

4

2 5 2 4 4 2 3

(4 (1

5)

(4

16 15 9

6

6 2 4 2 3) 1) 4

2 0 1 0 2) 6) 3 2 2 s 3 3 1

5 4 10) 5) 4 4 5 7 3 4 2

4

4

5

2 5 1 2

3 6 7 6

2 1 0 2

4

5 4 3 3 (6 (1

(1

4 2 (0 (3

1)

4

2 5 4 6 6 3 2 6

12

13 13 16 9 13 18 12 14 11 14 18 10 8

9 14 17

probability* 5 18

24 23 28 24 29 24 21 20 20 27 22 25 26

.026 .014

.026

.026

17

21 29 26 26 17

* Brackets denote cutoffs placed to maximize differences between groups. P values computed using Fisher exact probability test.


HANDEDNESS AND DISORDER

815

consistent with one of the theories linking dyslexia and handedness mentioned above. The simulation was rerun with an inventory length of 9 items. Significant results at the .05 level could be obtained for 41 out of 100 simulations by selection of appropriate cutoff points. In 17 cases the probability level was less than .01. Clearly, when cutoffs are set post hoc, probability levels given by statistical tests are seriously misleading. The longer the handedness inventory, the greater the opportunity for capitalizing on chance by appropriate selection of cutoffs. With a 9-item inventory and 50 subjects in each of two groups, there is a 4 in 10 chance of obtaining a “significant” result. The solution to this problem is quite simple. Investigators should specify the hypothesis to be tested in advance and specify both sample size and cutoff ranges before collecting the data. If, with hindsight, the data suggest an interesting association that had not been anticipated, a new hypothesis, based on the results, should be formulated, and tested in a replication study. These recommendations are hardly novel, but appear to have been overlooked in many studies concerned with associations between handedness and disorder.

REFERENCES Annett, M. & Kilshaw, D., (1984). Lateral preference and skill in dyslexics: Implications of the right shift theory. Journal of Child Psychology and Psychiatry, 25, 357-377. Bishop, D.V.M. (1983). How sinister is sinistrality? Journal of the Royal College of Physicians of London, 17, 161-172. Fairweather, H. (1976). Sex differences in cognition. Cognition, 4,231-280. Geschwind, N., & Behan, P. (1982). Left-handedness: Association with immune disease, migraine, and developmental learning disorder. Proceedings of the National Academy of Sciences, 79, 5097-5100. Harris, A.J. (1957). Lateral dominance, directional confusion and reading disability. Journal of Psychology, 44,283-294. Oldfield, R.C. (1971). The assessment and analysis of handedness: The Edinburgh inventory. Neuropsychologia, 9, 97-1 13. Schachter. S.C., Ransil. B.J., & Geschwind, N. (1987). Associations of handedness with hair color and learning disabilities. Neuropsychologia, 25, 269-276. Soper, H.V., Ciccetti, D.V., Satz, P.,Light, R., & Orsini, D.L. (1988). Null hypothesis disrespect in neuropsychology: Dangers of alpha and beta errors. Journal of Clinical and Experimental Neuropsychology, 10,255-270.

816

D.V.M. BISHOP

APPEIiPV BASIC program to simulate banq preference distributions 10 INPUT “HOW MANY RUNS OF SIMULATION?” ; NP 20 NG = 2: REM NLJMf3fiR OF FpOUPS 30 INPUT “HOW MAYV SPBJECTS PER GROUP?” ; NS 40 INPUT WOW MANY ITEMS IN INVENTOPY?” ; NT 50 FOP p = 1 TO NR 60 FOR G = 1 TO 4 G 7U FOR S = 1 T O N S


SPY=O 90 PR = .9: REM PROF) TQAT AN ITEM PERFORYEP WITH RIGHT loo IF P N ~ ( X