Field Methods - Semantic Scholar

Field Methods http://fmx.sagepub.com

Evaluation of Web Survey Data Collection Systems Scott Crawford Field Methods 2002; 14; 307 The online version of this article can be found at: http://fmx.sagepub.com/cgi/content/abstract/14/3/307

Published by: http://www.sagepublications.com

Additional services and information for Field Methods can be found at: Email Alerts: http://fmx.sagepub.com/cgi/alerts Subscriptions: http://fmx.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav

Downloaded from http://fmx.sagepub.com at PENNSYLVANIA STATE UNIV on April 11, 2008 © 2002 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.

FIELD METHODS Crawford / WEB SURVEY DATA COLLECTION SYSTEMS

REVIEWS

Evaluation of Web Survey Data Collection Systems SCOTT CRAWFORD MSInteractive Deciding on which software package to use for Web-based social science data collection can be extremely difficult because of the wide range of software products available. Costs can range from a few hundred dollars all the way up to thousands, so it is important for the researcher to understand the three major categories of Web survey software and their capabilities before focusing in on comparing any specific group of products. This review highlights three software packages, each representing a category of software that is currently available on the market. The review evaluates each software package on its ability to adhere to known Web survey standards, its flexibility, and its robustness.

T

he World Wide Web has emerged as a viable means of collecting survey responses, and software developers have jumped at the chance to design the gold-standard Web-based data collection system. Such a system, it has been thought, could bring Web-based surveying to the common researcher. A comprehensive, self-contained Web survey system that meets most needs has yet to be developed. This is for a good reason: The Web survey research industry has not developed well-documented standards for how to do good-quality Web-based research. Various researchers have begun to explore the intricacies of doing research online and how various designs affect data quality (Dillman et al. 1998; Couper 2000, 2001; Crawford, Couper, and Lamias 2000; Dillman 2000; Kennedy et al. 2000). However, until a basic framework for design standards has been developed, the specifications for a good-quality Web-based data collection product will not be set. Until then, software quality will hinge greatly on the ability of the researcher to use the tools provided by the software to create the best possible solution, given the resources available for each survey project. The three products selected for this review represent the three main types of Web survey packages available today: a basic off-the-shelf package, an intermediate application service provider package, and an advanced survey Field Methods, Vol. 14, No. 3, August 2002 307–321 © 2002 Sage Publications

307


308

FIELD METHODS

system. The terms basic, intermediate, and advanced will become clearer as the evaluation develops. They correspond positively with cost; however, cost should not be used as a sole indicator of capabilities or quality. With Web-based survey systems, as with most Web development environments, there are almost always additional products, services, and systems required to make the product fully functional. These additional pieces influence overall costs and resource requirements. Contact the software vendors for detailed costs as they pertain to your specific survey needs. Regardless of which package you choose to invest in for your research needs, I recommend that you work with a network specialist to make certain that your available network systems (including hardware, software, operating systems, and network connections) can accommodate the software for its intended use as a twenty-four-hour-a-day, seven-day-a-week platform. Each product in this review represents a category of product that exists in the marketplace today. Inclusion was primarily based on my experience using these systems. I conducted the review through actual product use, product documentation, and with any additional information made available through the product Web site. I reviewed each product as a researcher who implements Web surveys on a daily basis. While the review does not exhaustively explore every feature and capability, it gives a broad idea of the level of functionality each product and product group may offer. Researchers considering purchasing any of these products should thoroughly review the one they feel may best fit their needs to ensure that it actually does. Three products were selected for this review. The first is the following: SurveySolutions for the Web v4.0 Perseus Development Corporation Web site: http://www.perseus.com/ Sales contact: [email protected] (781-848-8100) List price: $229 + $65 annual support/upgrade fee

SurveySolutions for the Web v4.0 is the latest version of the low end of the Perseus product line. This package has received much attention for its ability to bring Web surveys to low-budget research. In many respects, it is a teaser product for the more comprehensive Professional and Enterprise versions also distributed by Perseus. SurveySolutions is clearly designed for the researcher who does not possess a high level of technical skills. The interface looks much like the interface in Microsoft Word. Surveys can be created in a word processor and uploaded into the survey system. Figure 1 shows the look of the design interface.



FIGURE 1 SurveySolutions for the Web Survey Design Interface

309

310

FIELD METHODS

SurveySolutions is designed so that surveys can be hosted on the researcher’s servers and may require support from a computing department to manage the software and its place within your organizational Internet infrastructure. The second product to be reviewed is the following: zTelligence 2.0 MarketTools, Inc. Web site: http://www.markettools.com Sales contact: [email protected] (415-462-2049) List price: $6,000 setup and training, plus $2–$9 per completed respondent (dependent on volume of use)

zTelligence is a product born of the dot.com tradition as an application service provider; that is, the vendor provides the software as well as hosting services for the surveys. Hosting services include network access, network security, hardware, system software, as well as 24/7 technical support. zTelligence is an entirely Web-based product. Figure 2 shows a typical question creation page within the zTelligence system. The survey design and reporting are available via the Web through most standard Internet browsers. Survey design is done through a graphical user interface, with minimal or no complex programming skills required. Without even referring to the online instructions, one could fairly easily navigate the survey design features. Fielding a survey takes a little reading of the online documentation, but it can be mastered in a few hours. Because it is hosted by MarketTools as an application service, the user is worry free when it comes to system compatibility, maintenance, and general support. The third product to be reviewed is the following: mrInterview SPSS MR Web site: http://www.spssmr.com Sales contact: [email protected] (513-552-8700) List price: Prices start under $13,000. Contact SPSS MR for details regarding your specific needs.

The longtime statistical software giant has just recently released its mrInterview Web-based data collection package. This product was designed under the SPSS MR Data Model, which is an open database model that allows for integration with other systems already in existence. mrInterview requires HTML, javascript, or similar Web-based interface programming skills to alter the appearance of the Web surveys being designed. A


Crawford / WEB SURVEY DATA COLLECTION SYSTEMS

311

FIGURE 2 zTelligence Question Design Interface

script-style programming language is required to program the survey logic and validation checks. Greater functionality comes at a price of lower ease of use with mrInterview. While not extremely difficult, it will require patience to self-teach or a formal training course. The design interface relies on extensive HTML knowledge. Bottom line: To fully master this product, one requires the knowledge typically found in a computer science department or a Web design firm. SPSS has created some tools to more easily interface with the scripting language, but these limit the flexibility of the product if relied on. SPSS is flexible with how mrInterview is implemented. It comes from a long-standing product line that has been developed for users to run on their own systems, requiring their own network support structure. However, SPSS


312

FIELD METHODS

is exploring offering an application service provider setup for this product at an additional cost. Exact cost and details surrounding such an ASP service have not yet been determined. A typical mrInterview user may have multiple screens open at once in the design of the survey—a text editor for the scripting code, another text editor for the design template, a compiler to convert the code into a usable survey, and a Web browser to review the finished survey page. Figure 3 shows what a typical mrInterview design station may look like. To effectively evaluate the categories of software products in this emerging market, I decided to focus on three broad themes within the three selected products: • adherence to known Web survey design standards (a measurement against standards established through literature and practice of current Web researchers), • flexibility (a discussion of how well the software will allow researchers to grow into larger and better research projects, as well as adapt to new standards as they are developed), and • robustness (much like e-commerce, Web surveys are expected to function 24/7 with frequent heavy usage, so in any evaluation, one must measure how well the survey system handles such pressures).

ADHERENCE TO KNOWN WEB SURVEY DESIGN STANDARDS As highlighted recently by some Web survey methods researchers (Couper 2000), Web surveys allow almost anyone to obtain responses from hundreds or thousands of respondents. However, as with any scientific data collection, this introduces the risk that poor research designs will be conducted. Much as early spreadsheet programs lacked the ability to guide users through complex accounting, today’s Web survey packages do little to ensure that users collect quality survey data. There are three key areas of Web survey design: screen design, logic design, and validation design. Screen design includes the use of color, fonts, layout, graphics, and placement of the various components of each question. Logic design includes the flow through the survey, calculations that drive which question(s) should be asked, and general survey navigation. Valida-



FIGURE 3 mrInterview Script, Template, Compilation, and Browser Interface

313

314

FIELD METHODS

tion design includes programming that checks user inputs. This includes checks for data format, consistency between responses, range checks, and ensuring that responses are adequately answered. Within each of these categories, I have evaluated how well the system handles each area with its default settings and standard features. Default settings are important in reviewing survey software because of the newness of online research. It is important that survey systems are designed to minimize poor survey design. Defaults that promote poor design could well lead unknowing researchers down a path toward poor-quality data. The standards detailed below are standards that have been developed by a leading Web research firm, MSInteractive, for their survey designers to follow in their survey production work. Where noted, standards have come from published works in the field of Web survey methodology or Web site usability; otherwise, they have been developed through experience in practice. Because of the newness of Web survey methodology, these standards should not be used as a gold standard for evaluation. Capabilities that go beyond the standards are also important and are not left out of the discussion. The review of system flexibility will address how these missing general capabilities may be achieved by other means. In Table 1, I present where each system did not adequately meet one or more of these capabilities. Capability Requirements to Adhere To: Web Survey Design Standards • Screen Design Capabilities — Use of Sans Serif font with control of type size (Nielsen 2000) — Use of contrasting color for background, text, and emphasis (Nielsen 2000) — Minimal use of graphics/multimedia — Question, response, and instruction screen placement flexibility — “Continue” button placement flexibility — Placement of multiple questions on a page (Couper, Traugott, and Lamias 2001) — Question types include single choice, multiple choice, matrix, numeric entry, text entry, memo, constant sum, and pull-down — Use of HTML to edit the design within a page or question — Create custom question styles (such as combined text and check box questions or other nontypical formats) • Logic Design Capabilities — Handle surveys with all questions on one page (for short surveys) and interactive (screen-by-screen) designs for long surveys — Behind-the-scenes variable calculations (including timestamps, user environment variables, or calculations involving previous responses) — Imported sample data (preloaded data)



315

TABLE 1 Adherence to Web Survey Design Standard Capabilities: System Faults SurveySolutions for the Web Screen design

Logic design

Validations

Font setting in system does not control browser font Text color changes don’t work in multiple column responses No standard interface for question instructions Matrix question defaults with borders visible No control over “continue” button placement Side-by-side questions not possible No custom question creation No logic capabilities No preload capabilities No behind-the-scenes calculation capabilities No text fill capabilities No randomization capabilities No ability to interact with outside systems No validation against preload or other responses No user authentication Minimal data format check

zTelligence

mrInterview

Uses graphics rather No constant sum than radio buttons, question type check boxes, and Custom question buttons creation is possible Does not allow but extremely side-by-side question difficult to implement, placement with few instructions Continue button text on doing so cannot be changed Very little design No custom question control within the creation mrInterview product, but the standard design template provided is good No hidden values NO FAULTS FOUND or internal flags No ability to interact with outside systems

No custom validations NO FAULTS FOUND

— — — —

Behind-the-scenes skip logic Fill text responses into later parts of the survey Randomization and rotation of responses, questions, and so on Dynamic creation of response options based on previously provided responses — Ability to connect with external applications (including other survey systems or other databases used to manage sample) • Validation Capabilities — Mandatory responses — Compare responses against other responses or preloaded data — User authentication — Range checks — Data format checks (text, numeric, date, etc.)


316

FIELD METHODS

FLEXIBILITY System flexibility is extremely important for a Web survey system to survive as a strong competitor. As Web survey methodologists begin to define the best standards for conducting Web-based research, it is important that the survey software be able to accommodate change. It is not realistic to expect that as soon as a standard is established, survey software vendors would provide the capability immediately. Rather, the survey software should be flexible enough for the researcher to devise a custom solution using existing Web technologies that could be used until the capability is implemented within the software. This is not a call for openly shared software code for all survey software (however, that may not be a bad idea), but it is a call for survey software providers to build enough flexibility into their systems to allow for users to integrate their survey system with other software products they use in their research. Any serious Web researcher would likely have the need for at least one custom addition to any survey done. Screen Design and Validation Capabilities As standards for question placement on the screen, question types, and other design elements to Web surveys become clearer, it will be important to be able to break free from the default question settings. The technologies used for screen design are frequently the same as those used for validation (HTML, javascript, etc.), so these two issues are grouped together here. Due to their use of HTML in question layout and design, all of the products were open to customizations that could be handled through the addition of HTML tags within the questions and responses. zTelligence, however, was not able to handle HTML tags that are used for more than simple text formatting, such as complex tables. mrInterview and SurveySolutions do allow for a higher level of control over screen design with HTML, as they provide access to the underlying template for each question. mrInterview and SurveySolutions also allow for advanced interactive survey design capabilities as well as any validation approach that uses a Web-based scripting language (such as javascript), while zTelligence does not. When we attempted to use javascript within a zTelligence instrument, we caused an error that only MarketTools technical support was able to fix before we could continue using the product. Overall, while zTelligence has the easiest to use and best built-in standard design interface, it is greatly lacking in screen design flexibility. SurveySolutions, with a fairly robust design interface, is strengthened by its



317

flexibility in working with external design solutions. mrInterview is a product that, by design, uses standard HTML and other Web-based design interfaces for its screen design, meaning that it is also extremely flexible. Logic Design Capabilities Because SurveySolutions for the Web provides only a single form-based survey design with no logic capabilities, there was no review conducted for that product. However, it should be noted that technically, it would be feasible to create multiple surveys with SurveySolutions for the Web and tie them together with custom Web programming into a fully interactive survey. However, this option would not be much better than programming a full survey from scratch with a custom programming product and thus was not considered as a “flexibility” of the system. zTelligence, like its screen design interface, provides no control over customizing the logic design capabilities of the system. With zTelligence, you are left with the standard logic capabilities of basic skip patterns, text fills, preloads, dynamic list creation, and some randomizations. Highly complex logic designs, requirements to access external data sources, and the ability to allow external sources access to its data are not possible with zTelligence. mrInterview is the only system that allows for full flexibility within the survey logic. While this may not be a requirement for all researchers, it can be extremely important for a research organization looking to conduct very complex research programs. Logic can be created to handle extremely complex conditions that can use external data sources or inputs. Custom Web scripts can be activated from within mrInterview surveys, allowing for no limit to the complexity of the logic implemented. The SPSS Data Model provides an excellent ability to integrate mrInterview into your other systems and survey needs.

ROBUSTNESS Any product intended for use on the Internet must be evaluated for its ability to handle multiple respondents (in some surveys, it may be thousands) in the survey at the same time, regardless of the time of day or day of the week. While robustness of the system may not be as important as the other aspects of this review, a system that does not meet the needs of the user in this regard can become useless very quickly.


318

FIELD METHODS

The increased cost of large sample sizes is minimal compared to other data collection modes. However, server and software capacity to deal with multiple respondents at once may provide a ceiling for how many respondents are asked to complete the survey at any given time. Hardware, connection speed, and other server capabilities will also affect the ability for a Web survey to handle multiple respondents. However, several features built into Web survey products can make large-scale surveying easier to manage. SurveySolutions for the Web maintains no additional features that allow for a high volume of respondents to the survey. Using this system makes you totally reliant on the capabilities of the underlying network. mrInterview provides for server load balancing (a feature in which a respondent is sent to the most available of a group of servers running the software) and failover (where a respondent is moved to another server if one stops working). Both of these features add to the complexity of using mrInterview. For serious, high-volume Web survey researchers, such features may be required. zTelligence gains the first prize for robustness, however, as the system handles load balancing, failover, and 24/7 support to ensure that the survey system is up and running at any time for respondents to complete their surveys. zTelligence benefits greatly from its application service provider setup.

CONCLUSION In reviewing Web survey software, it is clear that the old axiom “you get what you pay for” holds true not only with reference to the cost of the system itself but also the cost of the resources required to maintain and use the system. The most capable systems cost the most and require the most attention and training to use effectively. For a first-time Web researcher or for an organization looking to do few surveys, high costs may not be justified. The good news is that there are alternatives to the most comprehensive system that will allow for good-quality Web-based research to be done. Table 2 details each product’s main strengths, weaknesses, and the general profile of a researcher who would likely find the product most appealing. Given available resources, mrInterview stands out overall as the most capable, flexible, and robust system available. The long-term view that the SPSS MR team maintains with its Data Model, allowing for integration across all survey needs, gives its system the best possibilities and advantages for the future. It is likely that third-party vendors, as well as SPSS, will slowly develop easy-to-use front-end plug-in software to make creating and fielding surveys easier to do with mrInterview.



319

TABLE 2 Strengths, Weaknesses, and a Typical Researcher Profile for Each Product SurveySolutions for the Web Strengths

Weaknesses

zTelligence

Low cost Reliable and robust Easy survey creation Few technical skills Flexible screen design required Acceptable screen design and logic capabilities No logic capabilities Very low flexibility Not very robust

A researcher who Typical researcher A researcher new to has little support profile for each Web surveys, most product likely working from a in Internet infrastructure. This could be an university or other independent program setting with a fairly evaluator or possibly robust Web a researcher looking presence. Research budgets are funded by to collect pilot data internal funds or pos- for a larger grant. sibly by grants, where Long-term use of a Web survey system the primary purpose may be desired, but of the grant is not data collection. Such no commitment to purchasing researchers would hardware and other only conduct a resources has been couple of short made. surveys a year.

mrInterview Very flexible Extensive screen design and logic capabilities Robust capabilities

Very difficult to use Requires extensive training High cost A researcher who has extensive Web survey plans, with the need for complex designs. Such researchers likely have a staff of programmers at their disposal who have experience (or the ability to be trained) in creating computerized survey instruments with some script-based system. Such researchers are likely working in the commercial setting or at a university research center.

For researchers with more modest resources, zTelligence and SurveySolutions for the Web fulfill very different needs. zTelligence will allow researchers to design and implement Web surveys quickly and with minimal need for a researcher to have a strong technical background. The ease of use may overshadow the lack of flexibility for creating surveys that are outside of the current zTelligence capabilities. The added comfort of knowing that you can survey as many folks as you would like, without worrying about the system coming down, makes zTelligence a very safe choice.


320

FIELD METHODS

If your organization has technical personnel ready to handle the installation and support of your own survey system, the flexibility of SurveySolutions for the Web gives that system an edge. Its ease of use, once installed, allows for researchers to do their own survey creation. Be careful, though, that you do not exceed the resources your system has in fielding surveys with this system. It is a good idea to pretest thoroughly before spoiling a sample on a survey that crashes when overburdened. Unfortunately, the limitations in logic capabilities make SurveySolutions for the Web a nonviable product for many researchers who need to field more complex surveys. However, if you find it adequate for now and your survey needs increase, you can always graduate to one of the higher end Perseus survey products. Selecting a Web survey system is not an easy task with an easy solution. There is no gold standard yet for Web survey software. Every investment in Web survey resources has its rewards and pitfalls. And once a product is selected, keeping up with the changing technologies can become a full-time job of its own. The right product for you now may not be the right product for you next year, or next month, or even tomorrow. Knowing what resources you have available going into the marketplace to obtain a Web survey product is key. With that, you can maximize your Web survey software purchase so that you can have the most capable, flexible, and robust product available at the time. Applied broadly, these three facets of Web survey packages can be applied even as new products come to market.

REFERENCES Couper, M. P. 2000. Web surveys: A review of issues and approaches. Public Opinion Quarterly 64:464–94. . 2001. Web surveys: The questionnaire design challenge. Paper presented at the 53rd Session of the ISI, August, Seoul, South Korea. Couper, M. P., M. Traugott, and M. Lamias. 2001. Web survey design and administration. Public Opinion Quarterly 65:230–53. Crawford, S., M. P. Couper, and M. Lamias. 2000. Web surveys: Perceptions of burden. Social Science Computer Review 19 (2): 146–62. Dillman, D. A. 2000. Mail and Internet surveys: The tailored design method. New York: John Wiley. Dillman, D. A., J. Tortora, J. Conradt, and D. Bowker. 1998. Influence of plain vs. fancy design on response rates for Web surveys. Paper presented at the joint statistical meetings of the American Statistical Association, August, Dallas, TX. Kennedy, J. M., S. Kuh, J. Li, J. Hayek, J. Inghram, N. Bannister, and K. Segar. 2000. Web and mail survey: Comparison on a large-scale project. Paper presented at the annual conference of the American Association for Public Opinion Research, May, Portland, OR. Nielsen, J. 2000. Designing Web usability. Indianapolis, IN: New Riders.



321

Scott Crawford is a research director of social science research at MSInteractive, Livonia, Michigan. He holds an M.A. in applied social research from the University of Michigan, Ann Arbor. His experiences and research focus are on the use of the Internet for high-quality social science data collections. Trained as a research methodologist, he is continually exploring new ways of improving innovative data collection technologies. His background also includes extensive experiences with computers and Web site development, and his undergraduate focus was in sociology. He teaches a course in Web survey implementation each summer at the University of Michigan, Survey Research Center, Summer Institute for Survey Research Techniques and manages all MSInteractive clients in academia, government, and nonprofit evaluators.
