gpeterson - CASRO Online - March 2013 - Final3

SOLVING THE UNINTENTIONAL MOBILE CHALLENGE Gregg Peterson, Joanne Mechling, John LaFrance, Janice Swinehart and Gina Ham Market Strategies International Abstract Web surveys are increasingly completed on small-format mobile devices (e.g., smartphones) regardless of the intention of the researchers or the design of the instruments. In this experiment, we learn whether “mobile friendly” designs improve completion rates, survey length, self-reported user experience and respondent engagement for survey takers who choose to do a survey on smartphones, while still generating the same response distributions of surveys completed on PCs and tablets. By “mobile friendly,” we mean designs that use larger fonts and larger interface elements, and still size to the available screen real estate on a smartphone. We randomly assigned respondents to complete the same survey using one of eight survey presentation alternatives—six smartphone alternatives and two PC alternatives. We found that mobile friendly designs improve completion rates, survey length and user experience, but only one mobile friendly design consistently matched the response distributions from the PC surveys. We found very few differences in measure of engagement such as straightlining, speeding and satisficing in any of the alternatives.

BACKGROUND There is no doubt that unintentional mobile survey taking happens regularly. Mario Callegaro from Google first brought this phenomenon to researchers’ attention three years ago. Indeed, more and more web surveys are being completed on small-format mobile devices such as smartphones regardless of researchers’ intention or the instrument design (Peterson 2012; Comer & Saunders 2012; Jue 2012; Kinesis 2012). This growth in mobile survey taking tracks well with Americans’ increasing use of cell phones to access the internet for any reason (Smith 2012), which in turn, coincides with the growth in smartphone penetration in the US. Should we be concerned? Should we encourage or discourage more mobile survey taking on our existing web studies? As researchers, we do have the option to disallow it. On one hand, we favor honoring respondents’ choice about how they participate in our surveys. For example, Olson et al has recently shown that, when asked in a telephone study, “mode preference” can predict future participation in web surveys (Olson, Smyth, Wood, 2012). We know from Pew tracking studies that nearly one-fifth of cell phone owners do most online browsing on their phones (Smith, 2012), and we presume that many of these people spend little or no time browsing on PCs or large-format devices. Therefore, allowing mobile survey

LEAD RESEARCHERS: Gregg Peterson, Joanne Mechling, John LaFrance, Janice Swinehart, Gina Ham

17430 College Parkway, Livonia, MI 48152 P 734.542.7600 • F 734.542.7620 www.marketstrategies.com

Page 1 of 24 © 2013 Market Strategies International

taking should improve coverage and maximize participation rates by providing these people with a convenient way to complete surveys. On the other hand, a post hoc analysis of unintentional mobile survey taking demonstrates that compared with survey taking on larger devices, survey lengths can increase by as much as 50%, break-offs are nearly double for those who begin on a mobile device, and mode effects are possible (Peterson 2012; Grenville 2012; Jue 2012). While most plain HTML surveys designed for PCs render fine on smaller devices, the surveys are often difficult to read and answer without frequent zooming and pinching. The increased survey lengths and break-offs probably relate to both the design and usability of the surveys and devices, along with the natural lag on most cell phone networks and the speed of the particular networks where respondents browse. When allowing or disallowing mobile phones in surveys, robust participation seems to suffer either way. Disallowing mobiles might reduce participation by younger, more urban, and more affluent and educated respondents who will likely never finish our survey. Allowing mobiles could lose these same people at higher than expected rates due to break-offs. Perhaps most concerning—independent of who chooses to participate or complete surveys— is that the devices themselves (and how our surveys render on them) may impact how respondents answer key questions. The displays on smartphones are growing in size, but on average, continue to harshly limit what can be displayed and readable within the real estate available. Designing instruments that remain readable and work reasonably well within these limitations can force more scrolling, compel changes in the orientation of response choices, or lead to the creation of novel input mechanisms for capturing survey responses. These challenges are particularly acute when attempting to accommodate the designs of an existing survey’s questions, particularly questions customarily displayed in grids and rating scale type questions with many data points. For example, a question with a 0-to-10 scale on a series of related attributes, displayed in a grid, is popular in many kinds of survey research, particularly customer satisfaction research. These designs, that can overwhelm even large displays, are nearly impossible in a mobile environment. Usability expert Jakob Nielsen talks about the cognitive challenges associated with mobiles in his latest book (2013). He cites research that demonstrates a decline in comprehension when reading detailed information in a small window compared with a larger window. These declines relate to the need to memorize and scroll in order to maintain context and make comparisons. Nielsen recommends the following to designers of mobile websites: • Cut features to eliminate non-core functionality • Reduce word count • Enlarge interface elements to accommodate the “fat finger” problem




These are also good recommendations for researchers creating new web survey instruments, particularly when the intention is to capture some or all of the data from mobile respondents. However, some of these recommendations are unavailable to researchers trying to accommodate mobiles on an existing project or instrument with well-established norms or ways of analyzing and reporting on data. For example, clients who have established benchmarks for customer satisfaction based on a series of 11-point scales won’t easily move to 5- or 7-point scales, or reduce word count on questions or response stems without a long period of experimentation and calibration. So here’s what we hope to begin answering with this research: Can we meaningfully combine data from mobile and non-mobile participants when the survey instrument uses grids, 11point scales, and similar survey elements? Is there a design that can take advantage of the potential benefits of mobiles while minimizing the risk of mode effects, measurement error, longer survey lengths, and lower participation rates?

THE RESEARCH RECORD Rich literature supports the notion that survey design (independent of the questions asked) can impact survey results. Redline and Dillman (1999) discuss the impact of complexity and possible cognitive overload when designing self-administered mail studies that require branching. When questioned why respondents make mistakes, he answered that “respondents must process much more than just the verbal language of a self-administered questionnaire. They must process the graphic paralanguage, numeric and symbolic languages as well as its physical structure. Furthermore, they must interleave the processing of the languages applicable to the questions and responses, with this applicable to navigation.” He concluded that various “languages” can be manipulated to reduce errors. In short, good design makes a difference. Couper, Tourangeau, Conrad and other colleagues (2004, 2007 and 2008) have conducted a number of web experiments in which various elements of the web survey interface have been altered or manipulated in test and control situations. Inconsistent column widths, uneven spacing of response options, contrasting images, background color variations, and slider bars as alternatives to radio buttons (among many others) all have been demonstrated to impact survey responses. “Respondents make inferences about the meaning of survey items based on visual cues such as the spacing of response options, their order, or the grouping of questions. These inferences affect how quickly respondents answer the questions, which answers they select, or both.” Their works suggest that a consistent visual survey design may contribute to more consistent survey results over time. In one of the first studies analyzing smartphones as a potential tool for data collection, Peytchev and Hill (2008) conducted a number of experiments with a randomly recruited group of panel members who agreed to take surveys in exchange for smartphones and prepaid mobile connectivity. The experiments were designed to replicate past studies conducted in other modes as well as experiments designed to learn more about the unique




characteristics of this new mode. The researchers learned that small screens can affect responding, particularly for questions that extended beyond the screen, and the researchers saw differences in respondents’ willingness to provide open-ended responses. The researchers also found that images had a much smaller impact on survey response compared with the results seen in similar web survey experiments. The average size of a mobile phone screen has more than doubled since 2008; however, the findings support the notion that size and the design decisions that flow from that size significantly impact survey results. More recently, Bailey and Wells (2012) conducted a randomized experiment with panel members who own smartphones. The researchers randomly assigned participants to an online survey and a mobile survey conducted via a mobile application that respondents were required to download before proceeding. Bailey and Wells were surprised to learn that 23% of the panelists assigned to online surveys chose to take the survey on smartphones. They found no difference in study-specific response rates, completion rates or survey lengths when comparing online and mobile app participants, but the “unintentional” mobile survey takers took 50% longer to complete the survey. They found that mobile app users typed longer responses than the online counterparts. They were able to replicate other experimental findings from previous research in other modes—namely, that higher frequency scales generate higher frequency responses and that larger text entry boxes elicit more information than smaller boxes do, and they concluded that mobile surveys share many characteristics with other survey modes like paper and web. Their published record does not include comparisons of response distributions on rating scale questions.

SURVEY DESIGN ALTERNATIVES We are primarily interested in understanding whether “mobile friendly” designs improve the user experience for survey takers who do our surveys on smartphones, while still generating the same response distributions that surveys completed on PCs garner. By “mobile friendly,” we mean designs that use larger fonts and larger interface elements, and size to the available screen real estate on smartphones. In this randomized experiment, we tested mobile design alternatives whose primary difference was in the way scale questions and grids are displayed and answered. We focus on these questions types because they are particularly challenging in a mobile environment. We further investigated the impact of an alternative to the design of our standard PC survey—one that uses the same visual style and interface elements as our favorite mobile design. Finally, we tested the same survey delivered on a device-optimized application, programmed and hosted by our partner on this project, Research Now Mobile.




SURVEY INTERFACE DESCRIPTIONS USED IN THIS EXPERIMENT Test cell 1: Mobile/legacy: This is a standard HTML survey that uses small radio buttons and large grid layouts for ratings questions with multiple items. We made no attempt to alter our standard layout. This is not a mobile friendly design. Test cell 2: Mobile/new: This is the first of our mobile friendly designs, and the one that most closely matches our new PC alternative. All 11-point scale questions and most fully labeled scale items are presented horizontally. Like all other mobile friendly designs, we avoid the use of traditional grid layouts that require more vertical scrolling on questions with multiple items. (Note: All mobile friendly designs also use JavaScript to enable interactive interface elements.) Test cell 3: Mobile/numeric: The second mobile friendly design, respondents use numeric input boxes to indicate responses to key ratings questions. We created this interface for its simplicity and minimal need for interactive interface elements. Test cell 4: Mobile/slider: The third mobile friendly design, respondents use slider bars to indicate responses to key ratings questions. On questions with a natural midpoint, sliders are anchored in the middle. On categorical questions with no natural midpoint, the bars are anchored at the left end on a “never” or similar response. Test cell 5: Mobile/drop-down: The fourth mobile friendly design, respondents choose response choices to key ratings questions from a drop-down box. We use the native dropdown capabilities of each device, which vary slightly across mobile platforms. We order all numeric response choices (0-to-10 scales) with the highest number at the top, following the “up is good” heuristic suggested by Tourangeau et al (2004). Test cell 6: PC/legacy: This is our standard HTML survey designed for PCs. It is identical to the Mobile/legacy design, but of course rendered on larger devices. Note: Our past research indicates that the user experience is nearly identical when rendered on popular tablets, so the two PC test cells include tablet users. Test cell 7: PC/new: This is like our first mobile friendly design, rendered for larger devices. The key difference from test cell 2 is that we use a more traditional grid layout on multi-item ratings questions where item descriptions are in a left column. Test cell 8: Mobile/app: This cell can be considered mobile-optimized as the tool is platformspecific (e.g., Android, iPhone). Note: Programmers were forced to work within the existing interface options and limitations offered within the app. Most 11-point scales were presented in a fixed vertical form with the highest value on top, whereas most categorical scale questions used a slider-bar interface.




Screen captures for each survey design interface used in this experiment Figure 1: Test cell 1: Mobile/legacy (screen captures from iPhone 4s)

Figure 2: Test cell 2: Mobile/new (screen captures from a BlackBerry Torch)




Figure 3: Test cell 3: Mobile/numeric (screen captures from an iPhone 4 and a BlackBerry Torch)

Figure 4: Test cell 4: Mobile/slider (screen captures from an iPhone 5)




Figure 5: Test cell 5: Mobile/drop-down (screen captures from an iPhone 5)

Figure 6: Test cell 6: PC/legacy (partial screen captures from a Windows 7 PC)




Figure 7: Test cell 7: PC/new (partial screen captures from a Windows 7 PC)

Figure 8: Test cell 8: Mobile/app (screen captures from an iPhone 4)




EXPERIMENTAL DESIGN To systematically test the impact of these different mobile and PC design alternatives, we conducted an online survey of US adults using the ResearchNow online opt-in panel. We selected panelists who had previously indicated smartphone ownership, and further screened within the survey for respondents 18 years or older who indicated they “own and use a smartphone on a regular basis.” Our goal was to minimize the impact that variability in smartphone experience might have on completing mobile web surveys. We estimated the survey to take 8 to 10 minutes on PCs. The survey content was drawn from parts of a survey used for previous experiments by Downes-Le Guin et al (2012). The study is part of an ongoing, bi-annual study conducted by Market Strategies to provide energy industry executives perspective on public opinion related to energy and the environment. In addition, we included engagement and survey experience questions at the end of the survey. Figure 9: Summary of the experimental design

Panelists for the first seven test cells were selected from the ResearchNow panel to be nationally representative by region and to reflect age and gender distribution of smartphone owners as estimated by the Pew Center in its May 2011 tracking study. Panelists were randomly assigned to complete the survey on either a smartphone or a PC/tablet, keeping in mind the goal of completing approximately 250 surveys in each cell (i.e., we assigned fewer respondents to PC/tablet cells). Survey invitations and the panel landing page indicated the device type to be used to complete the survey. Upon entering the survey for the first time, respondents were randomly assigned to one of the first seven survey treatments appropriate




to the device assigned. Respondents entering the survey with the wrong device were prevented from continuing but encouraged to come back on the correct device. For example, respondent assigned to the smartphone cell who entered on a PC/tablet were shown the following: It appears that you’ve started this survey on a tablet or on a desktop/laptop computer. For this particular research effort, we need you to take it on a smartphone. Please close your browser and click on the URL in your study invitation once you are on a smartphone to complete the survey.

HYPOTHESES This experiment was designed to better understand the impact of various survey design alternatives, particularly focused on improving the mobile survey experience while maintaining data quality. Our hypotheses are outlined below: H1: Mobile surveys will take longer than PC surveys, regardless of design. H2: All mobile surveys will have higher break-offs than PC surveys, regardless of design. H3: Mobile friendly versions on smartphones will have shorter survey lengths and lower break-offs compared to the Mobile/legacy version. H4: Mobile friendly versions on smartphones will elicit higher user experience ratings compared with the Mobile/legacy version. H5: Survey design alternatives, whether on mobile or PC, have no impact on measures of engagement. H6: Response distributions across platforms (PC versus smartphone) will be most similar when they use a similar design (i.e., look and feel, interface elements). There will be the fewest differences in response distribution when comparing Mobile/new and PC/new.

RESULTS The survey was fielded from December 17 to December 30, 2012. ResearchNow emailed 27,143 invitations for respondents assigned to cells 1 through 7, resulting in 1,821 completes. The overall participation rate was 7%, but respondents in the smartphone cells were only half as likely to participate as respondents assigned to the PC/tablet cells. This difference is driven almost entirely by respondents who started the survey on the wrong device, were asked to come back on their assigned device, but never did. 50% of respondents assigned to smartphone cells opened the survey for the




first time on a PC, while only 6% of respondent assigned to the PC cells opened the survey for the first time on a smartphone. (These 6% of mobile starters could be considered “unintentional mobiles,” although we assume the proportions would have been higher had we not explicitly requested these respondents to complete the survey on a PC.) Table 1: Summary participation metrics for test cells 1–7 Smartphone cells (1–5)

PC/tablet cells (6–7)

Total

Invitations sent

22,731

4,412

27,143

Opened invites

2,818

649

3,467

Open rate

12%

15%

13%

Wrong device/incomplete

1,410

37

1,447

Wrong device rate

50%

6%

42%

Screen-out

18

49

67

Qualified starts

1,390

563

1,953

Break-offs

101

31

132

Break-off rate

7%

6%

7%

Complete

1,289

532

1,821

Completion rate

93%

94%

93%

Participation rate

6%

12%

7%

We hypothesize that experienced panelists are conditioned to complete surveys on PCs and likely do not read survey invitations carefully. After the first day fielding, the panel’s landing page was altered to boldly highlight the device assignment but this had only a negligible effect on the wrong device rate. Our goal in pre-assigning panel members to their respective devices was to minimize the potential for self-selection bias, but clearly younger respondents were more likely to choose the right device the first time and were more likely to come back on a mobile phone when asked to (see Table 2). Table 2: Survey participation by age in smartphone cells (age estimate comes from panel) 18–34

35–54

55+

Started survey

994

997

667

Wrong device/incomplete

45%

56%

67%

Complete (right device 1st time)

39%

31%

22%

Complete (wrong device 1st time)

15%

12%

11%

Total completes

538

432

224




The young skew in mobile participation in this survey is not entirely surprising and reflects what we’ve seen in past analysis of unintentional mobile survey taking (Peterson 2012; Grenville 2012). It also mirrors estimates of general mobile cell phone usage by Pew. However, the goal in this experiment was to create equivalent test cells in attempt to disentangle differences that might be caused by who chooses to participate in a survey on a particular device, from differences caused by the device and survey design itself. The first three rows of Table 3 show the unweighted age group distribution by test cell. Eighteen- to 34-year-olds are significantly underrepresented in the PC cells, and respondents 55 and older are significantly overrepresented. Table 3: Summary of key demographics and internet usage. (Weighted by age except as noted).

Mobile/ legacy

Mobile/ new

Mobile/ num eric

Mobile/ slider

Mobile / drop dow n

Mobile / app

PC/ legacy

PC/ new

n=242

n=247

n=249

n=246

n=254

n=198

n=243

n=250

a

b

c

d

e

f

g

h

unweighted 18–34

45%

50%

47%

49%

49%

47%

36% abcdef

36% abcdef

35–54

35%

34%

40% d

31% c

33%

33%

36%

36%

55 or older

20% c

16%

13%

20%

18%

19%

29% abcdef

28% abcdef

18–34

50%

50%

50%

50%

50%

50%

50%

50%

35–54

35%

35%

35%

35%

35%

35%

35%

35%

55 or older

15%

15%

15%

15%

15%

15%

15%

15%

weighted

Male

52%

52%

54%

53%

56%

54%

53%

57%

Female

48%

48%

46%

48%

44%

47%

47%

43% f

$100K+

25%

23%

28%

30%

25%

40% abcdeh

32%

23%

HS or less

4%

6%

4%

5%

4%

5%

23%

34% acdgh

23%

24%

29%

27%

21%

21%

College

40%

37%

40%

43%

42%

39%

41%

44%

Grad/prof

33% b

23%

33% b

28%

26%

28%

34% b

30%

Some college

5%

7%

Avg intnt hrs

23

24

22

23

22

22

Daily cell intnt

86%

82%

82%

82%

87%

83%

3G

28%

30%

31%

31%

31%

30%

23 67% abcdef n/a

22 73% abcdef n/a

4G

17%

19%

19%

20%

19%

18%

n/a

n/a

LTE 4G

13%

10%

11%

11%

9%

9%

n/a

n/a

WiFi

42%

41%

40%

38%

40%

43%

n/a

n/a

colum n proportions and colum n m eans tests: p < .05




To minimize the impact of self-selection bias, we weight all test cells to our goal proportions for age group. After weighting by age, there remain only a few differences and there appears to be no systematic demographic bias. Perhaps the biggest exception is the high-income skew of the Mobile/app group. It should be noted that all cells in this experiment are much higher income and more educated than the US population as a whole, and this primarily reflects who owns and uses smartphones regularly. As noted earlier, the Mobile/app group was recruited from among panel members who had already downloaded the Research Now mobile app. Their previous interest in participating in mobile research was a necessary bias we chose to live with and it might help account for the income disparity. At the bottom of Table 3, we see that average time on the internet and the type of network that mobile responders reported using to take the survey are statistically identical across all groups. However, respondents in the PC groups reported significantly less daily internet access via their mobile phone. We considered weighting based on these criteria but could not find a suitable source for setting target proportions and did not want to risk further reducing the effective sample size. Regardless of the differences, we still feel confident about making comparisons across these groups and are particularly encouraged that the groups assigned to mobile are nearly identical across the criteria we examined. We implemented several techniques to measure lack of engagement by respondents. We used these measures both to test our hypotheses about the impact (or lack of an impact) of engagement on presentation design and to use some of these questions as criteria to determine who should be removed from the final analysis. Table 4 shows the results of this analysis among all respondents who completed the survey including those who we eventually removed. PC respondents presented with our traditional legacy design were most likely to straightline on one or more of our four grid-style questions. There were no other differences on straightlining in mobile or new PC presentations. Mobile app respondents were slightly more likely to provide a meaningful (and not null) response to the open-ended question. The only other differences were in the number of characters that PC survey takers provided in the open-ends (among those who chose to respond). These differences are small but are likely related to the ease of typing on a larger keyboard as well as differing user expectations when responding in a mobile environment. Respondents were equally likely to report that they would “remove or downgrade your home’s insulation so that more heat escapes.” This trap question was randomly placed in a grid with 7 other more sensible statements about saving energy. Treatments were not different on selfreported satisficing. When asked how much effort they put into completing the survey, 8% to 9% of respondents said, “I skimmed the questions and chose an answer that seemed close,” but this did not significantly vary across PC and mobile versions.




We removed an average of 5% of respondents from the final dataset based on standard criteria used in other studies, using a combination of all but the open-ended analysis. (The “considered response” and number of open-ended characters were not part of the criteria.) We chose to remove these respondents for much of the analysis because our goal is to focus on the impact of the survey design alternatives on response distributions. Since we routinely remove unengaged respondents from our typical studies, it made sense to do so for this experiment. Table 4: Engagement and satisficing measures Mobile/ legacy

Mobile/ new

Mobile/ num eric

Mobile/ slider

Mobile/ drop dow n

Mobile/ app

PC/ legacy

PC/ new

n=258

n=258

n=258

n=257

n=258

n=203

n=266

n=266

a

b

c

d

e

f

g

h

Considered reponse

59%

52%

57%

57%

60%

67% bcdgh

53%

58%

Straighlined 1+

26% e

21%

24%

20%

18%

23%

30% bde

26%

Trap Question

19% d

17%

15%

17%

13%

14%

19%

19%

Self-report satisficing

9%

5%

7%

8%

6%

8%

11%

Strikeout

6%

4%

3%

4%

2%

2%

9% l

69

69

68

72

73

63

84 f

Number of OE char

8% 6% 86 acf

colum n proportions and m eans test: p < .05




To analyze differences in survey length, we chose to include only respondents who reported using their mobile phones to access the internet on a daily basis. This was the most important bias that we were not able to correct with age weighting, and we wanted equally adept and equally connected respondents for this part of the analysis in order to further isolate the impact of the device and survey treatment. In Figure 10, we can see that respondents assigned to the Mobile/legacy cell took approximately 50% longer to complete the survey than respondents in either of the PC cells. All mobile treatments took significantly longer. Only the Mobile/new treatment stands out as a sizeable improvement over the legacy treatment on the mobile side. The Mobile/legacy cell was the only treatment with significantly higher break-offs. Despite the extra survey length, respondents using mobile friendly designs were no more likely to break off than PC survey takers. Figure 10: Median survey length (among daily mobile internet users) and break-offs by treatment.

We asked several questions at the end of our survey to measure respondents’ enjoyment and experience in taking our survey. We asked 5 questions each using a 7point scale with anchored end points. We show the mean responses to these questions in Table 5. Mobile/legacy respondents reported a significantly worse user experience compared with every other cell, on each measure. Mobile/app respondents were just the opposite. Their reports were uniformly more positive compared with respondents in every other treatment. Most other mobile treatments garnered similar mean ratings; differences are trifling. The mobile/numeric option was seen as a tad harder to read, harder to answer and slightly




slower than other treatments. Given the need to constantly shift to a keyboard to respond to ratings questions, this is not surprising. We also asked mobile survey takers whether they would be willing to do a survey on a smartphone in the future based on their experience doing this survey. One in ten survey takers on the Mobile/legacy platform said they would not. Table 5: User experience ratings Mobile/ legacy

Mobile/ new

Mobile/ num eric

Mobile/ slider

Mobile/ drop dow n

Mobile/ app

PC/ legacy

PC/ new

n=242

n=247

n=249

n=246

n=254

n=198

n=243

n=250

a

b

c

d

e

f

g

h

Interesting?

5.1

5.5 ag

5.3

5.4 ag

5.6 acg

5.9 abcdegh

5.2

5.4 ag

Easy to read?

5.1

6.3 a

6.2 a

6.5 ac

6.4 a

6.8 abcdegh

6.4 ac

6.5 ac

Easy to answ er?

5.5

6.3 ac

6.0 a

6.2 a

6.2 a

6.7 abcdegh

6.3 ac

6.4 ac

Fast?

5.0

5.8 ac

5.5 a

5.6 a

5.7 a

6.1 abcdegh

5.6 a

5.9 acdg

Enjoy completing?

4.8

5.3 a

5.3 a

5.4 a

5.5 ag

5.9 abcdegh

5.2 a

5.4 a

2%

5% f

3% f

4% f

0%

n/a

n/a

Won't do in future

10% bcdef

colum n proportions and m eans test: p < .05

To cap it off, we looked for significant differences in the response distributions across 27 key questions in our survey. These questions were presented differently in each of our design treatments and, thus, most likely to be different across test cells. They included 15 fully-labeled categorical questions and 12 items using a 0-to-10 scale with end points labeled. The majority of these questions were shown as traditional grids in both of our legacy designs. In our mobile-friendly designs, they were broken up into individual items, each on a single page, and required considerable vertical scrolling in many cases. For the categorical questions, we used chi-square tests to find significant differences, and for the 0-to-10 scale questions we used a one-way analysis of variance to search for differences. In each case, we focused on pairwise comparisons among our two PC treatments and each of the mobile treatments. In addition, we were curious whether the new PC treatment would differ from the legacy PC design. Lastly, we computed the mean of all the absolute differences between top 2-box responses on each of the questions, in each paired comparison: The lower the mean absolute differences between treatments, the more similar the results. Our findings are in Table 6. The numeric entry, slider, drop-down and app versions of our mobile cells all had differences on at least four of 15 categorical questions when compared with both PC versions of our survey. The slider version had by far the most number of differences. The 0-to-10 ratings questions had fewer significant differences, but the slider version still faired quite poorly. When comparing the two PC treatments, there were three differences on categorical questions and no differences on the 0-to-10 scales. Finally, the analysis of mean absolute differences follows a similar pattern although it suggests that the mobile/drop-down version




was most different, with the slider and app-based version next most different. Only the mobile/legacy and mobile/new treatments were consistently (but not perfectly) similar to the two PC treatments. Table 6: Summary of differences on key response distributions

Categorical questons

0 - 10 rating questions

Chi-Square

ANOVA

Top 2 box

(15 questions)

(12 questions)

m ean abs diff

PC/ legacy

PC/ new

PC/ legacy

PC/ new

PC/ legacy

PC/ new

Mobile/ legacy

1

2

0

0

4%

3%

Mobile/ new

1

3

0

0

3%

4%

4

5

0

4

5%

6%

10

9

7

9

6%

5%

Mobile/ drop dow n

6

5

0

2

7%

7%

Mobile/ app

5

4

0

3

6%

6%

Mobile/ num eric Mobile/ slider

3

PC/ legacy PC/ new

3

0 0

4% 4%

count of significant differences: p < .05

DISCUSSION Our experiment was designed to test six hypotheses related to mobile participation in web surveys. Our first hypothesis (H1) states that mobile surveys will take longer than PC surveys, regardless of design. We accept this hypothesis. We leveled the playing field by ensuring that all respondents in our survey length analysis were regular (daily) mobile internet users with the same distribution of high-speed network access (WI-FI, 4G, 3G, etc.) While there was a great deal of variability in survey length across mobile treatments, both PC treatments yielded consistently faster survey lengths than the fastest mobile option. The second hypothesis (H2) was that all mobile surveys will have higher break-offs than PC surveys, regardless of design. We must reject this hypothesis. Only the Mobile/legacy survey treatment had significantly higher break-offs. There was very little variation in break-offs across the other treatments. Of course, we reject this within the context of a relatively short survey completed by opt-in panel members who’ve agreed to participate in our survey in exchange for incentives. Our third hypothesis (H3) was that mobile friendly treatments would uniformly shorten survey lengths and decrease break-offs compared with our Mobile/legacy treatment. This hypothesis is partially accepted. Break-offs were indeed consistently lower among mobile-friendly




treatments but the survey length differences were only significant in two of the 4 mobilefriendly treatments. While all mobile-friendly treatments were directionally faster than the legacy treatment, only the Mobile/new cell was significantly faster (p