ORGANIZATIONAL Stanton, Rogelberg / INTERNET/INTRANET RESEARCH METHODSWEB PAGES
Using Internet/Intranet Web Pages to Collect Organizational Research Data JEFFREY M. STANTON STEVEN G. ROGELBERG Bowling Green State University
Wide availability of networked personal computers within organizations has enabled new methods for organizational research involving presentation of research stimuli using Web pages and browsers. The authors provide an overview of the technological challenges for collecting organizational data through this medium as a springboard to discuss the validity of such research and its ethical implications. A review of research comparing Web browser–based research with other administration modalities appears to warrant guarded optimism about the validity of these new methods. The complexity of the technology and researchers’ relative unfamiliarity with it have created a number of pitfalls that must be avoided to ensure ethical treatment of research participants. The authors highlight the need for an online research participants’ bill of rights and other structures to ensure successful and appropriate use of this promising new research medium.
Between 1981 and the present, organizations in the United States have moved from centralized computing facilities to networks of interconnected personal computers (Abbate, 1999). These technology changes can have far-reaching implications for organizational research, including reduced research costs, enlarged sample sizes, shortened data collection-analysis-presentation cycles, improved access to previously hard-to-reach populations (e.g., the disabled), and enhanced interactivity of research materials. With these technological opportunities, however, come new challenges associated with designing, conducting, and interpreting research results. This article can serve as a road map to guide organizational researchers through unique issues encountered when planning and implementing research using Web pages on the Internet or an intranet.
Authors’ Note: We thank Lilly Lin and Alexandra Luong for their assistance in the literature search conducted for this article. Development of the ethics section of this manuscript was supported in part by award SES9984111 to the first author from the National Science Foundation. Correspondence concerning this article should be addressed to Jeffrey M. Stanton, School of Information Studies, Center for Science and Technology, Syracuse University, Syracuse, NY 13244-4100; e-mail:
[email protected]. Organizational Research Methods, Vol. 4 No. 3, July 2001 200-217 © 2001 Sage Publications
200
Stanton, Rogelberg / INTERNET/INTRANET WEB PAGES
201
Scope In this article, we focus on the use of Web page–based instruments for data collection. This focus provides a complement to some of the valuable, previously published work on e-mail-based surveys (e.g., Simsek & Veiga, 2000) and on other uses of e-mail for research (Anderson & Gansneder, 1995; Coomber, 1997; Schaefer & Dillman, 1998; Swoboda, Muhlberger, Weitkunat, & Schneeweiss, 1997). Using Web pages and browsers for data collection in management and industrial-organizational research can occur either over the Internet or an organization’s intranet. The Internet, a consortium of interconnected public and private computer networks, is notable for its accessibility to millions of computer users around the world. In contrast, intranets, the other foci of our article, are private networks administered by organizations and usually accessible only to organizational members. Organizational research conducted via the Internet and via intranets share broad parameters while differing in certain details. Although the differences do have minor methodological implications (e.g., intranets typically serve known populations), the major methodological challenges discussed here are generally common to both modalities. It is also worth mentioning that the scope of this study does not include the Internet as a target of study in and of itself (e.g., as a set of sociological phenomena, see Jones, 1999).
Examples and Issues: Setting the Stage Networked, browser-based research is highly flexible and can take many different forms. Not only can any paper-and-pencil study appear on a browser, but the medium can also provide unique, new opportunities for a researcher. Consider the following three examples: Example 1. A human resources department needs to make a quick decision about a new benefits plan. To uncover concerns and preferences, they publicize a brief Web survey on the corporate intranet. Employee survey responses are automatically entered into a database. Periodically, the software on the server automatically sends statistics and response rate information to the project leader in the human resources department. After 3 days, all data have been collected, analyzed, and distributed to decision makers. Example 2. Researchers from a professional industry consortium are interested in the identification of workplace barriers confronting sight-impaired individuals. They develop a Web-based research instrument to be compatible with pwWebSpeak™ and similar products that enable Web browsing for the sight impaired. They advertise their research in newsgroups and listprocs for the disabled. In addition, the researchers send dozens of e-mail chain letters to colleagues and contacts, asking them to inform sightimpaired workers about the research. Results of the questionnaire will provide information for a white paper that the consortium will publish to assist its members with ADA (Americans With Disabilities Act) compliance and employee retention. Example 3. University researchers are contracted to conduct a field experiment examining how managers in a particular organization use and interpret 360-degree feedback results. The field experiment is implemented on a customized Web server that randomizes presentation of experimental stimuli to participants (e.g., different
202
ORGANIZATIONAL RESEARCH METHODS
Figure 1: Schematic Overview of Networked Research Procedures
360-feedback reports). Study instructions are presented to the participant via streaming video. Furthermore, based on the demographic profile reported by the participant, the server tailors the types of questions asked about feedback experiences. The researchers solicit participation by providing the address of the research Web site in the organization’s monthly newsletter. The researchers must ensure, however, that only managers respond to the study, so they capture the Internet protocol (IP) addresses of participants and verify them against a list of IP addresses for all managers in the organization. Although by no means exhaustive of the possibilities for networked research, these examples serve to point out the enormous potential of networked research such as rapid access to study populations, enhanced interactivity of research materials, and specialized access controls. At the same time, these examples also contain some of the challenges of browser-based research. The challenges generally fall into one of three interrelated categories: challenges of actually collecting data using new technology, validity concerns, and ethical considerations. The remainder of this article addresses these issues. Within each section, we examine available literature from the social sciences that addresses relevant concerns. As an overview of how the sections interconnect, as well as to provide the reader with the “big picture,” Figure 1 presents a schematic diagram of the networked research process.
Data Collection Challenges Data collection using a Web browser as the medium for presentation of stimulus materials can require extensive knowledge of client-server computing and related technological issues. A detailed treatment of these issues is beyond the scope of this article. Instead, we take a “breadth over depth” approach and attempt to sensitize
Stanton, Rogelberg / INTERNET/INTRANET WEB PAGES
203
researchers to some key challenges indigenous to networked research. Where possible, we provide references to more extensive treatments of the various topics. The topics covered in this section address three common issues: (a) constructing and posting materials; (b) controlling access, authentication, and multiple response; and (c) encouraging participation in networked research. Notably absent from this list is a discussion of sampling and recruiting. However, a detailed review of probability and nonprobability sampling and recruiting strategies for networked research recently appeared in Simsek and Veiga (2000) as well as Schillewaert, Langerak, and Duhamel (1998). Posting and Constructing Materials
Researchers who wish to present networked, browser-based stimulus materials need hardware and software to fulfill the role of an Internet/Intranet server (see Schmidt, Hoffman, & MacDonald, 1997). The server stores the research materials and transmits them over the network on request. Server software can also be programmed to customize research materials and to receive data back from the client. A researcher’s home organization may already maintain Web servers, but many “Web hosting” companies also exist to provide these services (e.g., Yahoo). Schmidt (1997) provided a general introduction to the hardware and software requirements of networked research, and Schmidt et al. (1997) described particulars of implementing one’s own Web server. In addition to server facilities, researchers need methods for authoring stimulus materials and creating Web sites that provide the context for presenting the materials. Books such as Sams Teach Yourself HTML 4 in 24 hours (Oliver, 1999) and Web sites such as WebMonkey (http://www.webmonkey.com) provide extensive tutorial and reference information on Web site development. Supovitz (1999) described a case study of Web-based research from which he extracted eight lessons pertaining to the design and implementation of an effective Web site for research. Many commercial platforms exist for creating and distributing online surveys (King, 2000, contains a recent review). Furthermore, Morrow and McKee (1998) discussed strategies for making online research materials interactive. When designing Web materials, it is of critical importance to pilot test research materials, in part because materials may function as expected on one “platform” but not on another. In addition, it is important to ensure that respondents’ expertise with browsers and Web-based materials matches or exceeds the sophistication and complexity of the research instrument (for examples of problems in this area, see Bertot & McClure, 1996; Tse, 1998). Access Control, Authentication, and Multiple Response
In traditional research, gathering participants consists of contacting potential participants by mail, telephone, or face-to-face meeting and requesting participation in the research. Often a copy of the research materials accompanies the invitation. To our knowledge, researchers rarely express much concern that a participant will photocopy paper research materials and send them to new respondents. In contrast, recruiting techniques available in networked research, together with the ephemeral, electronic nature of the research materials, facilitate substantial opportunities for unsolicited
204
ORGANIZATIONAL RESEARCH METHODS
responses in browser-based research. For example, recruiting messages sent by e-mail to potential participants can easily be forwarded to other individuals or lists of individuals. When the URL of a Web survey appears in an e-mail message, the problem worsens. Convenience or plausibility samples obtained through public advertising of a study have even larger potential to attract unwanted participants. As a result of how easily research requests and materials can be copied and redistributed, two key aspects of networked research design are access control and authentication. Access control refers to methods for controlling use of computer resources (such as files stored on a server). Authentication refers to verification of the identity of a particular computer user. Authentication is often paired with access control in situations where successful authentication grants particular access privileges. Access control and authentication can be preventative in the sense that unwanted research participants can be disallowed from gaining access to or submitting research materials. Alternatively, authentication can be used to identify and remove their responses from the data as a first step in data analysis. In the simplest case of research on and within an entire organization, the use of an organization’s intranet often precludes responses from those outside the organization (for an overview, see Gilmore, Kormann, & Rubin, 1999). In all other research scenarios, unless convenience sampling is in use, researchers must have an access control or authentication strategy to prevent unwanted responses. Passive controls may suffice in some instances: If the research instrument is long and pertains to a very specific audience, there may be little motivation for a nonsampled individual to respond. In this case, simply stating who is eligible to participate may be sufficient to eliminate unwanted responses. A standard strategy for authentication and access control is a respondent identifier or password. Schmidt (2000) provided a detailed technical introduction to the implementation and use of passwords on Web-based research materials. Note that unless participants perceive the processes involved in allocating passwords or identifiers as random, they may have substantial concerns about anonymity, which may, in turn, modify response patterns. Filtering can provide authentication after data collection by checking passwords, reported demographic characteristics (e.g., removing males), or IP addresses (unique identifiers of network locations). Responses not matching a master list of “legal” identifiers are set aside or discarded. Multiple responding. Another aspect of avoiding unwanted participation is preventing multiple responses from the same individual. When an individual is asked to participate in a research project, it is hoped that he or she will submit only one set of data for the study in question. Multiple response can be inadvertent or purposeful. In inadvertent multiple response, an individual may accidentally activate the “submit” control more than once. Alternatively, the individual may mistakenly submit data prematurely and thus may resubmit after making a more complete response. In purposeful multiple response, an individual knowingly responds multiple times to a research request. This type of multiple response may be done to skew the findings in a particular direction or to sabotage the research effort. Purposeful multiple responses provide a greater challenge for data screening and analysis because the malicious respondent may submit slightly different data each time to prevent simple filtering of duplicates. Overcoming purposeful multiple responding can be difficult, but the following strategies can be used during the data collection process to reduce multiple respond-
Stanton, Rogelberg / INTERNET/INTRANET WEB PAGES
205
ing. First, when recruiting participants for a study, avoid motivating individuals to engage in malicious response. Specifically, avoid extensive cross-postings of research announcements and frequent e-mail participation reminders that may be perceived as intrusive (Cho & LaRose, 1999; Michalak & Szabo, 1998). Second, include explicit instructions outlining the need for one and only one response to the research request. Stress the importance and meaningfulness of the research. Third, design the Web site so respondents receive a confirmation query (e.g., “Are you sure you want to submit your responses?”) at the moment they submit their data. The server should also provide immediate acknowledgment that data have been received. Fourth, using one of the authentication strategies described above, one can assign each respondent a unique identification code to submit with the research data. After data collection, one can filter out multiple responses as the first step in data analysis. An alternative to passwords is to use the IP address from the client computer to flag multiple submissions from the same address. This technique is subject to false positives because different people responding from the same computer may have the same IP address attached to their submission. Augmenting server software to record the time of day would allow filtering if data came from the same IP address and were submitted within a brief period. Without unique identifiers or IP addresses, one can still screen for identical submissions by matching demographics and/or research responses. Malicious multiple responses not containing identical data are sometimes noticeable as distributional anomalies (e.g., extreme outliers). When multiple identical responses are detected in the data, a researcher can typically take one of three actions: drop all data in the multiple-response group, accept only the first submission in the group, or accept only the last submission in the multiple-response group. Encouraging Participation in Networked Research
In contrast to concerns for preventing responses from nonsampled individuals and multiple responses from others, researchers always hope for the highest possible response rate among sampled individuals to mitigate nonresponse bias (Luong & Rogelberg, 1998; Rogelberg & Luong, 1998; Smart, 1966). Schwarz, Groves, and Schuman (1998) listed standard methods of boosting response rates in traditional research: advance notice, incentives, persuasive scripts or introductions, and reminder notices. Each strategy has a counterpart in networked research. Witmer, Colman, and Katzman (1999) uncovered evidence that advance notice of an e-mail-based study, combined with the opportunity to decline participation in the research, helped to enhance response rates. Anderson and Gansneder (1995) used a “mail-merge” strategy to personalize each recruiting message. Teo, Lim, and Lai (1997) used $2 phone cards as an incentive to encourage early response to their Web-based questionnaire instrument. In Musch and Reips’s (2000) review of the history of Web-based experimentation, they located 10 studies that offered lotteries from $10 to more than $1,000. Another possibility involves providing the respondents with immediate general or personalized feedback concerning their research participation, an option that clearly requires custom server programming. Researchers may also facilitate response by maintaining the novelty of online data collection by designing diverse, intriguing, and interactive Web pages. Finally, perhaps the best way to facilitate response to a present research effort is to develop a credible track record (data from past research efforts
206
ORGANIZATIONAL RESEARCH METHODS
have been acknowledged and used responsibly and appropriately) with potential respondents (Rogelberg, Luong, Sederburg, & Cristol, 2000). Before closing our discussion of response facilitation, it is worth briefly discussing the notion of calculating response rates. Even with a known list of sampled respondents, it may be difficult to know the number actually contacted because of deactivated addresses, server errors, and other problems that interfere with the delivery of online recruiting messages. Without knowledge of the number of individuals that received a recruiting solicitation, it can be difficult to ascertain the response rate obtained. Not knowing the response rate makes it difficult to assess potential nonresponse bias. Kaye and Johnson (1999) made recommendations for estimating the response rate for networked research when it cannot be directly calculated. First, servers keep a log of the number of times each Web page was requested. By comparing this log with the number of completed research responses actually received, one can make a reasonable estimate of active refusals: individuals who saw the research materials (or at least the first page of instructions) and then declined to participate. In using this technique, it is important to use the server’s counts of “sessions” rather than “hits” because a page might be requested multiple times (hits) by one browser in the course of that user’s total time reviewing the research materials (a session). See Bosnjak (2000) for an application of this technique. Another strategy when using e-mail as a recruitment medium is to explicitly ask refusals to decline participation by replying to the solicitation with a response indicating their refusal. The clear problem with both of these latter strategies is that they ignore the possibly large set of sampled individuals who simply ignore the solicitation and do nothing. One of the important messages we hope to have communicated by raising all of the issues described in this section is that the technology involved in browser-based research adds a layer of complexity to the research process that many investigators have not previously encountered. This added complexity clearly necessitates a learning period during which researchers become more comfortable and familiar with the application of the new technology. More important, however, we suggest that the added complexity of browser-based, networked research creates new threats to the validity of research results and new challenges for the ethical treatment of research participants. In the following sections, we highlight the implications of the technology challenges highlighted above for validity and ethics.
Generalizability of Networked Research Data Differential access to computers, method bias, and response environment effects may adversely influence the generalizability of a networked study (Krantz & Dalal, 2000). In this section, we discuss each of these issues. It is important to note from the beginning, however, that the best way to address generalizability concerns is by crossvalidating results using different research modalities. Cross-validation is especially important when the research results are “mission critical” (e.g., when legal defensibility is a concern). Although samples obtained through networked means may stand on their own in exploratory research, when results are mission critical, crossvalidation of networked results with more traditional research techniques becomes a necessity.
Stanton, Rogelberg / INTERNET/INTRANET WEB PAGES
207
Bridging the digital divide. To promote generalizability, researchers ideally want their sample to mirror, demographically, the theoretical population of interest (Oakes, 1972). The digital divide is a term referring to the relative lack of access of low-income individuals to the Internet. We borrow the term here to capture the idea that individuals in different occupations within an organization may have different degrees of access to networked computers at work or home. Researchers (e.g., B. Thomas, Stamler, Lafreniere, & Dumala, 2000) have reported substantial demographic skews when collecting data pertaining to specific populations. Yost and Homer (1998) found substantial differences in responses to a job attitude survey based on administration type (Web vs. paper and pencil) until they controlled for job type. These results indicate that organizational researchers must remain sensitive to the fact that within most organizations, there are employees who do not have sufficient access to technology to respond to networked studies. Table 1 reveals that one of the most common divergences between samples obtained over the Internet and samples obtained through traditional research techniques pertains to differences in the demographic composition of samples. In some cases, the effect sizes of these demographic differences are substantial. Within organizations, access to the Internet or the organization’s intranet appears to vary with occupational type (Graphics and Visualization Unit [GVU], 1998). Workers in manufacturing jobs, face-to-face customer service jobs, physically mobile jobs, and other positions not commonly including computer work may have limited access to computers, browsers, and the organization’s intranet. GVU surveys have consistently indicated skews toward overrepresentation of males, Whites, and North Americans as Internet users. It is noteworthy, however, that in Table 1, these typical age and gender skews among Internet respondents do not universally hold (also see Pettit, 1999). Recent data suggest that gender skew is gradually being mitigated and that individuals from Europe and other regions are gradually gaining in proportional representation (Nielsen Netratings, 2000). Nonetheless, these findings all argue for the collection of comparison samples whenever demographic variables are suspected to influence a study’s focal variables. When demographic variables are related to the focal variables, nonresponse by a particular demographic group will lead to nonresponse bias and thus generalizability concerns (Rogelberg & Luong, 1998). For research conducted within a single organization, testing for demographic skew can easily occur using databases of personnel information. In samples obtained from the Internet at large, testing for such demographic skews can be accomplished using databases of demographic information such as the General Social Survey or profiles of Internet users such as the annual survey conducted by GVU. Atchison (1998) used this strategy with some degree of success in his dissertation. Method effects: comparisons of networked and traditional research. Generalizability of networked data can be compromised when the modality used to collect the data systematically alters the data. A substantial literature exists that compares computer administration of various types of tests with paper-and-pencil administration. Importantly, a great deal of the literature in this area is quite old and substantially predates the use of Web browsers (e.g., Kiesler & Sproull, 1986; McBrien, 1984), which may limit its applicability to browser-based research. Nonetheless, method effects (such as faking and socially desirable responding) documented in previous research may be exacerbated by Web-based administration (e.g., Zickar, 2000). Responses to
208
Table 1 Effect Sizes of Demographic and Substantive Differences Between Internet and Non-Internet Samples Citation
Research Design
Birnbaum (1999)
Ran decision-making experiments with two samples (n = 1,224) Internet respondents and (n = 124) undergraduate respondents
Buchanan and Smith (1999)
Compared (n = 963) Internet respondents with (n = 224) paper-and-pencil respondents to a personality survey (self-monitoring) Obtained (n = 573) Internet responses and (n = 233) undergraduate responses to an experiment on body image and female attractiveness Obtained two samples (n = 429, n = 1,657) of Internet respondents and two samples of paper-and-pencil respondents (n = 760, n = 148) to a personality inventory
Krantz, Ballard, and Scher (1997) Pasveer and Ellard (1998)
Smith and Leigh (1997)
Compared (n = 72) Internet respondents with (n = 56) paper-and-pencil undergraduate respondents in a survey study of sexual fantasies
Stanton (1998a)
Conducted two-group confirmatory factor analysis of survey results on Internet sample (n = 50) and postal mail sample (n = 181)
Yost and Homer (1998)
Compared different conditions (paper only and paper with an option for Web response) resulting in 1,090 paper-and-pencil versus 405 Web respondents to an attitude survey
Differences Between Samples Age: φ2 = .19 (Internet: Older) Gender: φ2 = .01 (Internet: More males) Substantive conclusions identical No mean differences detected Confirmatory factor analysis fit better in Internet sample Age: φ2 = .47 (Internet: Older) Gender: φ2 = .00 Substantive conclusions identical Age: d = .51 (Internet: Older) Gender: φ2 = .01 to φ2 = .05 (Internet: More males) Marital status: φ2 = .29 (Internet: > % married) Mean difference (d = .16) on one study variable Age: φ2 = .12 (Internet: Older) Gender φ2 = .29 (Internet: More males) Dependent variables: d = .02 to d = .47 No differences in factor structure Age: d = .24 (Internet: Younger) Gender: φ2 = .06 (Internet: More males) Hours/week: d = .52 (Internet: More) Job tenure: d = .14 (Internet: Less) Unionized: φ2 = .10 (Internet: Fewer) Hourly: φ2 = .15 (Internet: Fewer) Dependent variables: d = .47, d = .37 After controlling for job type of respondent because of differential access to the Web, mean differences on 2 out of 13 items Web responses were returned significantly faster than paper-and-pencil responses Web respondents provided significantly longer openended comments
Note. Effect sizes calculated and presented following recommendations of Rosenthal (1991, pp. 15-25).
Stanton, Rogelberg / INTERNET/INTRANET WEB PAGES
209
some instruments may vary based on respondents’ beliefs about anonymity or expectations about the purpose of the test (e.g., Allred, 1986; Kantor, 1991; Lautenschlager & Flaherty, 1990). There may also be deeper implications of anonymity rooted in participants’ beliefs about the trustworthiness and reliability of computers and software. For example, Kiesler and Sproull (1986) ascertained that networked participants responded to a greater proportion of items, made longer open-ended responses, and responded in a less socially desirable manner than those completing a pencil-andpaper survey. Earlier work showed that people tended to be more “self-absorbed” and less inhibited when communicating using a computer (Kiesler, Siegel, & McGuire, 1984; Sproull, 1985). Note that the novelty of a computer survey in the early 1980s may have influenced individuals’ response styles. In contrast, the popularization of the Internet has raised public awareness of privacy issues, and this could adversely affect respondents’ openness when responding with a browser (GVU, 1998). In contrast to concerns raised by the older research, recent research comparing browser-based results with other modalities paints an optimistic picture. Table 1 summarizes studies in which a direct or indirect comparison was accomplished between samples of data collected through traditional means and networked samples. Most studies showed minimal substantive differences in interpretation of the results obtained from the different samples (cf. the many demographic differences). Where mean differences did occur on substantive variables, they were generally insufficient to influence the overall interpretation of results. Furthermore, of major concern for correlational research, the factor structures of attitudinal measures appear to be invariant across administration types (e.g., Buchanan & Smith, 1999; Stanton, 1998a). Of course, invariant factor structure does not guarantee that two versions of an instrument are equated with one another. Thus, we amplify the admonition offered above: When research involves between-subjects comparisons of means, particularly with a sample obtained at an earlier point in time or with a different mode of administration, we recommend that researchers obtain cross-validation evidence of their browser-based results. Uncontrolled response environments. Uncontrolled response environments can negatively affect data generalizability in that each participant may respond to the Web materials in a different context. Buchanan and Smith (1999) contrasted Internet research strategies to research using other computerized platforms. In particular, they highlighted the fact that researchers have no control over circumstances in which participants complete research tasks. These circumstances include equipment and software used by the participant and the environment in which that equipment exists. Researchers cannot ascertain whether others are present when research materials are completed, whether environmental distractions affected the research participant, or whether software and hardware incompatibilities changed the appearance or behavior of the stimuli. This latter point is particularly salient for tests that require control over the timing of stimulus presentation or participant responses. Because connections between servers and clients are “asynchronous,” such studies require specialized client software to control timing of stimuli and to measure and/or control timing of participant responses. Importantly, the circumstances of research administration also include the state of the respondent himself or herself. Although researchers working in the laboratory may notice if participants are sleepy, intoxicated, or distressed, researchers conducting net-
210
ORGANIZATIONAL RESEARCH METHODS
worked research have no knowledge or control of the mental and physical state of respondents (differing mental and physical status of respondents may introduce error variance or confounding bias into the data). For research conducted within organizations, this offers no worse a challenge than standard mail-return survey practices. Perceived generalizability. With online study results in hand, one of the most frustrating tasks for researchers may be convincing an audience that the data are representative of a relevant population and that the method has not otherwise skewed the results. This lack of credibility may manifest for applied researchers in client or audience rejection of their results and for scientific researchers in difficulty publishing their data. This credibility problem has two aspects: One aspect is real threats to generalizability as discussed above. The other is simply the newness of the research medium. Managers, editors, reviewers, and others who evaluate research may not have had ample time or experience to develop reliable heuristics for judging the value of networked organizational research. A set of studies conducted by Walsh (1998; Ognianova, 1997; Walsh, McQuivey & Wakeman, 1999) confirmed that information from the Internet is perceived as considerably less credible than traditional sources (e.g., printed media). Walsh also documented that the perceived credibility of online sources varies systematically with demographic variables of the audience. Age, gender, frequency of Web usage, and other demographics accounted for about 8% of the variance in perceived credibility of sources in a student sample and 21% of the variance in an adult sample. In particular, older individuals and those with less Web experience were more inclined to disbelieve information obtained there. One implication of these findings is that researchers collecting data over the Internet (and even over intranets) may have to work extra hard, at least for now, to demonstrate the rigor of their research methods. The obvious, though costly, way of accomplishing this is to collect data from a cross-validation sample that is obtained through more traditional research methods. Most of the studies described in Table 1 undertook this strategy with the specific intention of comparing Internet and traditional samples.
Ethical Implications of Online Research Rosenthal (1994) suggested that validity and research ethics were inextricably entwined because the quality and proper conduct of a study affect both issues. Thus, we would be remiss if our discussion of browser-based research did not include a discussion of ethics. To protect the welfare of research participants, all researchers must adhere to the ethical principles of their respective specializations (e.g., American Psychological Association; see Lowman, 1998). Although this imperative applies regardless of the modality in which research is conducted, browser-based research introduces new ethical challenges for researchers. Ethics Overview
The principal, overarching ethical challenge facing networked researchers concerns participant privacy and anonymity. Unfortunately, in networked research, regardless of the technological sophistication, it is very difficult to guarantee the anonymity of respondents. Cho and LaRose (1999) gave a comprehensive discussion of
Stanton, Rogelberg / INTERNET/INTRANET WEB PAGES
211
privacy concerns associated with soliciting participation in research studies over the Internet, as well as concerns for anonymity and confidentiality. Data from BartelSheehan and Grubbs-Hoy (1999) strongly suggested that privacy concerns could substantially and negatively influence response rates to online research. Cho and LaRose (1999) and Stanton (1998b) provided reminders that the collection of passwords, IP addresses, and other unique identifiers may affect the degree to which respondents trust the anonymity of the research. These techniques clearly place an extra burden on the researcher to maintain the secrecy of the identification data provided. Schmidt (2000) provided technical information on threats to data security for studies conducted over the Internet. In brief, unless the researcher’s Web server and the participant’s Web client use Secure Sockets Layer (SSL) or another interface to provide secure client-server connections, there is no way to ensure that the participants’ responses cannot be accessed by a third party. Therefore, in seeking informed consent from participants, it is incumbent on the researcher to indicate that although every effort will be made to secure responses, the Internet/intranet is too uncontrolled to be able to make perfect assurances of anonymity or confidentiality. Even with clientserver security, depending on how the participant’s computer is configured, it still may be the case that participant responses could be inadvertently cached on the client computer. Consequently, participant responses could be purposely or accidentally viewed by others (this problem is exacerbated when computers are shared). B. Thomas et al. (2000) and others reviewed the importance of security protection for researchers’ Web sites to avoid unintentional release of data and misuse of computer resources. We obtained a relevant case study through personal communications, in which we ascertained that a researcher had distributed survey materials to a large number of participants using a list processor or “listproc.” Unfortunately, the technology was incorrectly configured such that when participants responded, their completed surveys were distributed to the entire list. This simple case illustrates that the amplifying power of the technology and a researcher’s lack of mastery of that technology can enable serious breaches of anonymity and confidentiality. Finally, it is worth mentioning that if the organizational research project involves monitoring and analysis of interactions occurring in a listproc, discussion group, or chat room, privacy and anonymity issues take on a complicated new dimension. The principle of informed consent requires that in any research on identifiable individuals, those individuals must give at least tacit approval to participation in the research (Sieber, 1992). J. Thomas (1996a) reviewed a highly controversial case in which an undergraduate researcher from Carnegie Mellon University published research containing serious ethical violations involving unannounced researcher observation of such interactions. Due to space limitations, we cannot provide full discussion of this topic, but an entire issue of the journal, Information Society, was dedicated to analysis of ethical issues pertaining to research in “cyberspace” (see J. Thomas, 1996b). Informed Consent and Debriefing
Typically, organizational research projects avoid deception and are innocuous in content. As a result, informed consent and debriefing are not always salient issues. Passive consent through actual participation in the research is often taken as evidence of consent (assuming the participant is not a minor). Occasionally, however, organizational research is sensitive in nature (e.g., sexual harassment) or may involve deception
212
ORGANIZATIONAL RESEARCH METHODS
(e.g., false feedback). Thus, it is important to understand options for consent and debriefing in networked research. Consent forms. Consent can serve as a gateway to the experimental research materials such that the first element the prospective participant sees when accessing the research materials is a consent form (Schmidt, 1997; Smith & Leigh, 1997). The participant indicates consent by clicking a button and only then gains access to the study. The challenge of informed consent in networked research stems from the word informed. In networked research, it is unknown whether participants have read or truly comprehended the information on the consent form prior to indicating agreement. Moreover, if a prospective participant does not understand a part of the consent form, there is no immediate way of addressing questions. Some mechanisms exist to allay these concerns. First, to promote comprehension and avoid superficial reading of the information contained on the consent form, a multimedia explanation (e.g., streaming video) of the consent material can be provided. A second mechanism involves designing “quiz” items that tap whether the prospective participant understood the information presented on the consent form. Access to the research materials can be made contingent on correct responses. This has obvious implications for response rates and illustrates one of the ways in which ethical, technology, and validity issues are tightly interconnected. Debriefing. Organizational research occasionally involves mild forms of deception. Ethically, deception is deemed acceptable to the extent that it does not cause harm, that it serves important research goals, and that alternative methods are unavailable. A condition of using deception, however, is that the researcher will conduct some type of debriefing with participants after the study is completed. Debriefing in the virtual world is not straightforward. For instance, the participant may easily leave a research Web site prematurely without reading the debriefing. Alternatively, participants may not understand the debriefing, and yet the researcher is not present to clarify any confusion. Relatedly and most important, the researcher is not present to deal with adverse participant reactions. To address some of these concerns, we recommend the following: As above, consider using multimedia or at the very least have the debriefing statement “open” in a separate window. Second, contact information should be provided so participants can ask questions and share concerns. Finally, research that may elicit strong emotional reactions should simply not be administered using networked methods. Reporting Results
One final unique ethical challenge to researchers who use networked data collection lies in the reporting of research results. We have previously discussed problems with ascertaining response rates, assessing nonresponse biases, discarding responses from nonsampled individuals, and discarding multiple responses from single individuals. Each of these issues can affect the quality of the data reported and the validity of the conclusions drawn from the results. It is the researcher’s responsibility to make a complete report of anomalies encountered (and how they were resolved) throughout data collection and analysis processes to facilitate judgments about data quality and validity by reviewers and readers (Aguinis & Henle, in press).
Stanton, Rogelberg / INTERNET/INTRANET WEB PAGES
213
Conclusion Predictions always entail risk, but we feel confident in suggesting that use of networked research methods will continue to burgeon for the foreseeable future. We have reviewed many challenges that researchers currently face and provided some suggestions for dealing with these. Before closing, we would like to broach some future needs that we see as important for ensuring the continuing success of networked research efforts. Participants’ bill of rights. Some of the ethical issues we reviewed suggest that our ability to use technology in the service of research has grown faster than our ability to apply ethics to the resulting new scenarios. General principles for the treatment of human subjects could be mapped specifically onto some of these new scenarios to create a networked research participants’ bill of rights. This bill of rights would promote norms concerning what online research participants can expect and demand from researchers who seek their participation. These norms would pertain to the whole networked research experience—from receiving a recruiting solicitation all the way to obtaining reports of the research results. SPAM-free recruitment protocol. On a more detailed level than a participants’ bill of rights, researchers need specific, operational guidelines on how to use e-mail and other electronic contact methods to reach potential research participants without breaching privacy or violating the standard rules of “netiquette” (Davidson, 1999). For example, one wise but relatively unknown rule is to always contact the owner of a listproc individually and privately to ask permission before posting a recruiting message to that listproc. Ideas such as these could be codified into a publicly available protocol for contacting participants electronically. Study certification and registry. We have some concern that overuse of networked research strategies may eventually result in substantial reductions in response rates, more attempts to sabotage research projects, and lower quality responses. Cho and LaRose (1999) underscored the idea that as the popularity of Internet data collection increases, individuals may become more resistant to responding. Rogelberg and Luong (1998) documented a similar phenomenon (for paper-and-pencil surveys) within organizations and termed it oversurveying. Schwarz et al. (1998) indicated that response rates across all types of studies have been declining over time. Together, these pieces of evidence suggest that researchers must treat Internet research participants as a finite and depletable resource. A constant barrage of e-mail solicitations to participate in Internet research seems an assured way of angering potential research participants and discouraging their participation. In addition, the ease with which different stakeholders within an organization can create and field an intranet survey raises a similar concern for intranet-based research. At present, several clearinghouse sites exist where researchers can register and publicize their projects (see Azar, 2000). Thinking more broadly, the potential for overuse of participant resources argues for some type of networked research consortium. Such a consortium could provide a similar project registry function, but this could be paired with guidelines and assistance to help researchers avoid alienating potential research participants.
214
ORGANIZATIONAL RESEARCH METHODS
Professional respondent panels. In market research and other areas of social science, researchers often assemble a working panel of respondents who typically respond to research stimuli in exchange for modest stipends or perquisites. When thoughtfully realized, these panels can be composed of highly representative individuals whose responses collectively provide similar or superior generalizability to nonprobability samples of volunteers. Such professional panels also have the potential to mitigate the oversurveying problems mentioned above. More empirical research on networked methodologies. A great deal of methodological research is needed to truly understand how to design and interpret data collected from the Internet and intranets. First, more research is needed on differences in responses and respondents obtained by different sampling/recruiting techniques, particularly as access to the Internet and intranets continues to grow. Second, the base rate of malicious multiple responding is unknown. An experiment could be devised to explore the effects of the recruiting method and the content of the research instrument on the likelihood of multiple response. Third, statistical explorations are needed to ascertain the influence of multiple responding on data quality. In addition, techniques such as classification analysis could be applied to networked data in the service of detecting response patterns that signify multiple response. Fourth, numerous questions remain in the area of nonresponse. Do passwords for access control decrease participants’ sense of anonymity? Do response rates vary with the type of access control used? The use of complex access control or authentication strategies may inhibit responding because of the extra time and effort respondents must spend gaining access to research materials. Finally, an additional avenue of inquiry concerns the perceived validity (managers, editors, reviewers, and others) of organizational data collected via the Internet and intranet. A plethora of additional opportunities exists for methodological explorations of networked research. With structures in place to support ethical networked research practices and additional methodological research to understand and address validity concerns, networked research may well become the method of choice for organizational researchers in the near future. References Abbate, J. (1999). Inventing the Internet. Cambridge, MA: MIT Press. Aguinis, H., & Henle, C. A. (in press). Ethics in research. In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psychology. Malden, MA: Blackwell. Allred, L. J. (1986). Sources of non-equivalence between computerized and conventional psychological tests. Unpublished doctoral dissertation, Johns Hopkins University. Anderson, S. E., & Gansneder, B. M. (1995). Using electronic mail surveys and computermonitored data for studying computer-mediated communication systems. Social Science Computer Review, 13, 33-46. Atchison, C. (1998). Men who buy sex: A preliminary description based on the results from a survey of the Internet-using population. Unpublished doctoral dissertation, Simon Fraser University, Burnaby, Canada. Azar, B. (2000). A Web experiment sampler. Monitor on Psychology, 31(4), 46-47. Bartel-Sheehan, K., & Grubbs-Hoy, M. (1999). Flaming, complaining, abstaining: How online users respond to privacy concerns. Journal of Advertising, 28(3), 37-51.
Stanton, Rogelberg / INTERNET/INTRANET WEB PAGES
215
Bertot, J. C., & McClure, C. R. (1996). Electronic surveys: Methodological implications for using the World Wide Web to collect survey data. Proceedings of the ASIS Annual Meeting, 33, 173-185. Birnbaum, M. H. (1999). Testing critical properties of decision making on the Internet. Psychological Science, 10, 399-407. Bosnjak, M. (2000, May). Participation in non-restricted Web surveys: A typology and explanatory model for non-response. Paper presented at the 55th annual conference of the American Association for Public Opinion Research, Portland, OR. Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90, 125-144. Cho, H., & LaRose, R. (1999). Privacy issues in Internet surveys. Social Science Computer Review, 17, 421-434. Coomber, R. (1997). Using the Internet for survey research. Sociological Research Online, 2(2) [Online]. Available: http://www.socresonline.org.uk/2/2/2.html Davidson, S. (1999). From spam to stern: Advertising law and the Internet. In D. W. Schumann & E. Thorson (Eds.), Advertising and the World Wide Web (pp. 233-263). Mahwah, NJ: Lawrence Erlbaum. Gilmore, C., Kormann, D., & Rubin, A. D. (1999). Secure remote access to an internal Web server. IEEE Network, 13(6), 31-37. Graphics and Visualization Unit, Georgia Institute of Technology. (1998). 10th WWW User Survey [Online]. Available: http://www.cc.gatech.edu/gvu/user_surveys/survey-1998-10/ Jones, S. G. (Ed.). (1999). Doing Internet research: Critical issues and methods for examining the Net. Thousand Oaks, CA: Sage. Kantor, J. (1991). The effects of computer administration and identification on the Job Descriptive Index (JDI). Journal of Business & Psychology, 5, 309-323. Kaye, B. K., & Johnson, T. J. (1999). Taming the cyber frontier: Techniques for improving online surveys. Social Science Computer Review, 17, 323-337. Kiesler, S., Siegel, J., & McGuire, T. (1984). Social psychological effects of computer-mediated communication. American Psychologist, 39, 1123-1134. Kiesler, S., & Sproull, L. S. (1986). Response effects in the electronic survey. Public Opinion Quarterly, 50, 402-413. King, N. (2000). What are they thinking? PC Magazine, 19(4), 163-178. Krantz, J. H., Ballard, J., & Scher, J. (1997). Comparing the results of laboratory and WorldWide Web samples on determinants of female attractiveness. Behavior Research Methods, Instruments, and Computers, 29, 264-269. Krantz, J. H., & Dalal, R. (2000). Validity of Web-based psychological research. In M. H. Birnbaum (Ed.), Psychological experimentation on the Internet (pp. 35-60). San Diego: Academic Press. Lautenschlager, G. J., & Flaherty, V. L. (1990). Computer administration of questions: More desirable or more social desirability? Journal of Applied Psychology, 75, 310-314. Lowman, R. L. (Ed.). (1998). The ethical practice of psychology in organizations. Washington, DC: American Psychological Association. Luong, A., & Rogelberg, S. G. (1998). How to increase your survey response rate. The Industrial-Organizational Psychologist, 36(1), 61-65. McBrien, B. D. (1984, November). The role of the personal computer in data collection, data analysis, and data presentation: A case study. Paper presented at the ESOMAR Conference, Nice, France. Michalak, E. E., & Szabo, A. (1998). Guidelines for Internet research: An update. European Psychologist, 3(1), 70-75. Morrow, R. H., & McKee, A. J. (1998). CGI scripts: A strategy for between-subjects experimental group assignment on the World-Wide Web. Behavior Research Methods, Instruments, and Computers, 30, 306-308.
216
ORGANIZATIONAL RESEARCH METHODS
Musch, J., & Reips, U.-D. (2000). A brief history of Web experimenting. In M. H. Birnbaum (Ed.), Psychological experimentation on the Internet (pp. 61-87). San Diego, CA: Academic Press. Nielsen Netratings. (2000). Internet year 1999 in review [Online]. Available: http://www. nielsen-netratings.com/press_releases/pr_000120_review.htm Oakes, W. (1972). External validity and the use of real people as subjects. American Psychologist, 27, 959-962. Ognianova, E. V. (1997). Audience processing of news and advertising in computer-mediated environments: Effects of the content providers perceived credibility and identity. Unpublished doctoral dissertation, University of Missouri, Columbia. Oliver, D. (1999). Sams teach yourself HTML 4 in 24 hours (4th ed.). Indianapolis, IN: Sams. Pasveer, K. A., & Ellard, J. H. (1998). The making of a personality inventory: Help from the WWW. Behavior Research Methods, Instruments, and Computers, 30, 309-313. Pettit, F. A. (1999). Exploring the use of the World Wide Web as a psychology data collection tool. Computers in Human Behavior, 15, 67-71. Rogelberg, S. G., & Luong, A. (1998). Non-response to mailed surveys: A review and guide. Current Directions in Psychological Science, 7, 60-65. Rogelberg, S. G., Luong, A., Sederburg, M. E., & Cristol, D. S. (2000). Employee attitude surveys: Exploring the attitudes of noncompliant employees. Journal of Applied Psychology, 85, 284-293. Rosenthal, R. (1991). Meta-analytic procedures for social research. Newbury Park, CA: Sage. Rosenthal, R. (1994). Science and ethics in conducting, analyzing, and reporting psychological research. Psychological Science, 5, 127-134. Schaefer, D. R., & Dillman, D. A. (1998). Development of a standard email methodology: Results of an experiment. Public Opinion Quarterly, 62, 378-397. Schillewaert, N., Langerak, F., & Duhamel, T. (1998). Non-probability sampling for WWW surveys: A comparison of methods. Journal of the Market Research Society, 40, 307-323. Schmidt, W. C. (1997). World-Wide Web survey research: Benefits, potential problems, and solutions. Behavior Research Methods, Instruments, & Computers, 29, 274-279. Schmidt, W. C. (2000). The server side of psychology Web experiments. In M. H. Birnbaum (Ed.), Psychological experimentation on the Internet (pp. 285-310). San Diego, CA: Academic Press. Schmidt, W. C., Hoffman, R., & MacDonald, J. (1997). Operate your own World-Wide Web server. Behavior Research Methods, Instruments, & Computers, 29, 189-193. Schwarz, N., Groves, R. M., & Schuman, H. (1998). Survey methods. In D. T. Gilbert, S. T. Fiske, & G. Lindzey (Eds.), The handbook of social psychology (Vol. 2, 4th ed., pp. 143179). Boston: McGraw-Hill. Sieber, J. E. (1992). Planning ethically responsible research: A guide for students and internal review boards. Newbury Park, CA: Sage. Simsek, Z., & Veiga, J. F. (2000). The electronic survey technique: An integration and assessment. Organizational Research Methods, 3, 92-114. Smart, R. (1966). Subject selection bias in psychological research. Canadian Psychologist, 7, 115-121. Smith, M. A., & Leigh, B. (1997). Virtual subjects: Using the Internet as an alternative source of subjects and research environment. Behavior Research Methods, Instruments, & Computers, 29, 496-505. Sproull, L. S. (1985). Using electronic mail for data collection in organizational research. Academy of Management Journal, 29, 159-169. Stanton, J. M. (1998a). An empirical assessment of data collection using the Internet. Personnel Psychology, 51, 709-725. Stanton, J. M. (1998b). Validity and related issues in Web-based hiring. The IndustrialOrganizational Psychologist, 36(3), 69-77.
Stanton, Rogelberg / INTERNET/INTRANET WEB PAGES
217
Supovitz, J. A. (1999). Surveying through cyberspace. American Journal of Evaluation, 20, 251-263. Swoboda, W. J., Muhlberger, N., Weitkunat, R., & Schneeweiss, S. (1997). Internet surveys by direct mailing: An innovative way of collecting data. Social Science Computer Review, 15, 242-255. Teo, T.S.H., Lim, V.K.G., & Lai, R.Y.C. (1997). Users and uses of the Internet: The case of Singapore. International Journal of Information Management, 17, 325-336. Thomas, B., Stamler, L. L., Lafreniere, K., & Dumala, D. (2000). The Internet: An effective tool for nursing research with women. Computers in Nursing, 18, 13-18. Thomas, J. (1996a). When cyber-research goes awry: The ethics of the Rimm “cyberporn” study. Information Society, 12(2), 189-197. Thomas, J. (1996b). Introduction: A debate about the ethics of fair practices for collecting social science data in cyberspace. Information Society, 12(2), 107-117. Tse, A.C.B. (1998). Comparing the response rate, response speed and response quality of two methods of sending questionnaires: E-mail vs. mail. Journal of the Market Research Society, 40, 353-361. Walsh, E. O. (1998, August). The value of the journalistic identity on the World Wide Web. Paper presented at the conference of the Association for Education in Journalism and Mass Communication, Baltimore. Walsh, E. O., McQuivey, J. L., & Wakeman, M. (1999, November). Consumers barely trust Net advertising. Cambridge, MA: Forrester Research. Witmer, D. F., Colman, R. W., & Katzman, S. L. (1999). From paper-and-pencil to screen- andkeyboard: Toward a methodology for survey research on the Internet. In S. G. Jones (Ed.), Doing Internet research: Critical issues and methods for examining the Net (pp. 145-162). Thousand Oaks, CA: Sage. Yost, P. R., & Homer, L. E. (1998, April). Electronic versus paper surveys: Does the medium affect the response? Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, Dallas, TX. Zickar, M. J. (2000). Modeling faking on personality tests. In D. R. Ilgen & C. L. Hulin (Eds.), Computational modeling of behavior in organizations: The third scientific discipline (pp. 95-113). Washington, DC: American Psychological Association. Jeffrey M. Stanton is an assistant professor at Syracuse University’s School of Information Studies, where he conducts research on the organizational impacts of information technology. He is the director of the Syracuse Information Systems Evaluation project. He holds Ph.D. and M.A. degrees in industrial-organizational psychology and a B.A. in computer science. Steven G. Rogelberg is an associate professor of psychology and director of the Institute for Psychological Research and Application at Bowling Green State University of Ohio. His research focuses on improving organizational research methods and understanding and mitigating teamwork failures. He holds Ph.D. and M.A. degrees in industrial-organizational psychology and a B.S. in psychology.
