Feb 12, 2016 - Pete Cape. Director, Global Knowledge, SSI. In laying out an agree/disagree battery in a grid format the
Left to Right or Right to Left?: How to Order Scales for Best Results Pete Cape Director, Global Knowledge, SSI In laying out an agree/disagree battery in a grid format the researcher faces a choice of which order to display the scale: Agree on the left or Agree on the right? Does the choice matter? Researchers aware of a primacy bias (a bias toward choosing the first item presented visually) would counter this bias by flipping the scale such that half the sample sees ‘Agree’ on the left and half sees ‘Disagree’. If they were also concerned about recency bias (a bias towards choosing the last item when presented aurally) then the same solution would work. But might this action actually cause more problems than it solves? Research done in 2004 by Roger Tourangeau demonstrated that most participants expect the most positive category to be presented first -‐ so there appears a natural order in ordinal scales. (An ordinal scale is one in which the order of the scale points is important, but there is no precise measurement of the gaps between each point.) In order to prevent confusion therefore the scales should be presented in their natural order. Under this reasoning the scale for satisfaction (for example) ought to start with “extremely satisfied” and end with “not at all satisfied”). Scales presented the “wrong way around” may be biased towards the negative end. Any necessary mental correction that the participant has to make to the way the scale is being displayed (now “up means bad” rather than “up means good”) should result in slightly longer being taken to process the grid. Thus we might expect both time and data differences if the scale is presented in an un-‐natural order. This, however, is not what is found in the case of online surveys. According to another questionnaire design guru Don Dillman, in Internet, mail and mixed-‐mode surveys no data differences are found when the scales are reversed –although the “un-‐natural” version does take longer to do. An agree-‐disagree scale however (unlike an “extremely” to “not at all” scale) has no natural order. The “good” end of the scale depends firstly on the item being rated and secondly in the eye of the beholder (on the basis that you would think “good” that which you are congruent on – whether agreement or disagreement). Our small experiment also shows no difference in data on an agree/disagree scale resulting from the order the scale categories are presented, and showed no statistically significant differences when the item being rated tended to be agreed with or disagreed with:
February 12, 2016
[email protected] | surveysampling.com
% Agreeing Agree on left
Disagree on left
Base
155
154
Sports is an important part of your life
52%
47%
A sedentary lifestyle has no effect on your long term health You would like to be more active and fit by engaging in sports or physical exercise but you can’t find the time You always ensure you make time for yourself to engage in sports or physical activities average time
18%
21%
52%
45%
51%
44%
40 seconds
43 seconds
median time
36 seconds
40 seconds
However, we did see a slight increase in time taken when the scale is presented Disagree-‐Agree rather than Agree-‐Disagree. SSI therefore recommends that ordinal scales be ordered left to right (or top to bottom) in terms of best to worst. If flipping of a scale is called for the order of the scale should be kept the same throughout an individual’s survey in order not to cause confusion. About SSI SSI is the premier global provider of data solutions and technology for consumer and business-‐to-‐ business survey research, reaching respondents in 100+ countries via Internet, telephone, mobile/wireless and mixed-‐access offerings. SSI staff operates from 30 offices in 20 countries, offering sample, data collection, CATI, questionnaire design consultation, programming and hosting, online custom reporting and data processing. SSI’s 3,600 employees serve more than 2,500 clients worldwide. Visit SSI at www.surveysampling.com.
February 12, 2016
[email protected] | surveysampling.com
