Online data collection in academic research ... - Wiley Online Library

Blackwell Publishing Ltd.Oxford, UKBJETBritish Journal of Educational Technology0007-1013British Educational Communications and Technology Agency, 20052006384574582ArticlesOnline data collection in academic researchBritish Journal of Educational Technology

British Journal of Educational Technology doi:10.1111/j.1467-8535.2006.00638.x

Vol 38 No 4 2007

574–582

Online data collection in academic research: advantages and limitations

Samúel Lefever, Michael Dal and Ásrún Matthíasdóttir Samúel Lefever and Michael Dal are assistant professors at Iceland University of Education. Ásrún Matthíasdóttir is assistant professor at Reykjavík University. Address for correspondence: Samúel Lefever, Iceland University of Education, Stakkahlid, 105 Reykjavík, Iceland. Email: [email protected]

Abstract Online data collection in academic research might be replacing paper-andpencil surveys or questionnaires in the near future. This paper discusses the advantages and limitations of online data collection, with particular reference to the conduct of two qualitative studies involving upper secondary school teachers and students in Iceland in 2002. Email was used for contacting the participants to ask them to visit a designated website in order to complete the questionnaire. Some problems arose with the use of an online web-based programme for data collection. Among them were the unreliability of the email address lists and the lack of willingness, particularly among students, to participate. The paper concludes that while online surveys can access large and geographically distributed populations and achieve quick returns, they may no longer be as universally appealing as was once believed. Reaching the population sample remains a problem in online as well as in traditional data collection.

Introduction In this paper, we discuss online data collection, by which we mean the use of a website to collect real-time data and store this in a form suitable for statistical analysis. We will discuss some of the advantages and limitations that occur in using online data collection. The discussion derives from the authors’ experience in conducting two quantitative studies involving upper secondary school teachers and students in Iceland in the autumn of 2002. The research is a part of the project LearnICT, which is a research project funded by the Research Council of Iceland under the Information Technology research programme. The purpose of the LearnICT project is to consider the opportunities presented when Information and Communication Technology (ICT) is used in teaching and learning. The research covers the use of ICT at preschool, primary, upper secondary and university levels (http://namust.khi.is). © 2006 The Authors. Journal compilation © 2006 British Educational Communications and Technology Agency. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.

Online data collection in academic research

575

Web surveys in educational research Collecting research data through traditional paper-and-pencil methods can be costly and time consuming. It often involves visiting institutions (field work) in order to ensure that data collection is done satisfactorily. In some cases, it demands that the researcher has access to staff and trainees who can help gather the data. The alternative is to conduct web surveys. These appear to have the potential, and indeed are used worldwide, to collect large amounts of data efficiently and economically within relatively short time frames. Though online data collection is relatively new among educational researchers, several benefits have already been identified. Mertler (2002) observes that while online data collection is currently little used for this purpose, it is an efficient and convenient alternative to the more traditional method of gathering information from students, teachers and parents. First of all, web surveys guarantee a rather short time frame for the collection of responses and are time and cost saving. The World Wide Web also gives researchers access to a large, diverse and global population with the potential of huge amounts of data. Furthermore, researchers point out that online data collection protects against the loss of data and simplifies the transfer of data into a database for analysis (Carbonaro & Bainbridge, 2000; Ilieva, Baron & Healey, 2002). Some researchers even argue that using a web survey guarantees a potentially better response rate (Ilieva et al, 2002; Matz, 1999). However, this is not accepted by all researchers. There are some important issues that researchers should be aware of. Carbonaro and Bainbridge (2000) point out that an easy access to surveys for all participants is essential in online data collection. Also, web surveys must be designed in such a way that they are simple to complete. Third, web surveys must have a built-in security system to ensure credibility and anonymity and finally, web surveys should require only a minimum of computer skills for their completion. To date, few studies have discussed the reliability and effectiveness of using online data collection. One factor having to do with the reliability of survey studies is that of response rates. Comley (2000) names three factors that affect response rates in online surveys: (1) style of the first page of the survey, (2) relationship with the website/brand and (3) respondent interest or relevance of the survey. Response rates as low as 30% have been considered reasonable in self-completed postal or mail surveys (Saunders, Lewis & Thornhill, 1997) but according to Comley (2000), most virtual surveys in 1999 showed a response rate between 15 and 29%. Other research studies report rates from 25 to 60% in email surveys (Moss & Hendry, 2002; Virtual Surveys Ltd, 2001). With regard to the type of responses, Matz (1999) and Saphore (1999) found in comparative research studies that there were no differences in the pattern of responses between a web survey and an identical survey in paper form. On-site data collection with paper and pencil presents a number of known problems. It is very bound to time and location, and although the number of participants can be © 2006 The Authors. Journal compilation © 2006 British Educational Communications and Technology Agency.

576

British Journal of Educational Technology

Vol 38 No 4 2007

controlled, for example, in a classroom-administered questionnaire, there is no guarantee that the respondents are truly willing to participate. In web surveys, however, respondents can participate at their own convenience and can decide when and where to complete the survey. The only limitation is that the respondents need a computer that is connected to the Internet. This freedom given to participants can be two-sided. On the one hand, participants freely choose to respond to the questions, but on the other hand, it can be a disadvantage if respondents postpone filling out the questionnaire and then—in many cases unwillingly—forget all about it. This freedom of choice regarding participation also applies to mail surveys, but the advantage of online surveys over mail surveys is the ease of sending the responses. It is much more convenient to send responses through the click of a button rather than through a trip to the post office. An area of concern regarding web surveys is that of fraudulent respondents. In online surveys, respondents may not answer truthfully or can pose as different people (nor can honesty be guaranteed in paper-and-pencil surveys.) It is advisable to incorporate some means of personal identification to gain more control over the response group, such as contacting the population through an email mailing list. To date, there is no conclusive evidence that indicates a difference in responses between self-completed mail surveys and online surveys (Ilieva et al, 2002). However, there is evidence that suggests that participant responses in email surveys can be more detailed and comprehensive than in paper-and-pencil surveys (Schaefer & Dillman, 1998). Other studies report receiving more complete information through email and online surveys (Bachman, Elfrink & Vazzana, 1996; Mehta & Sivadas, 1995; Stanton, 1998). The limitations and difficulties in using online data collection can also be of a technical nature. External technical problems can arise from the Internet service provided or the respondents may lack the ability to use the new technology or the willingness to participate in online computer surveys. Reaching the target population through online methods also posses certain difficulties. Email messages announcing surveys are frequently interpreted as junk mail and are deleted without hesitation from the mailbox or automatically diverted by screening programmes. Other online data collection limitations have to do with population samples and their nonrandom nature. The control of the population can be achieved through the use of population lists, but they are frequently very difficult to obtain. A common way of designing online surveys is to link the survey to a home page or home pages and ask for volunteers to fill out the survey. In such cases, researchers have little control over who and how many people fill out the questionnaire, jeopardising the reliability of the survey. However, nonrandomness is a problematic issue with other methods of data collection as well, and it must be remembered that online data collection is based on volunteer sampling rather than on probability sampling. © 2006 The Authors. Journal compilation © 2006 British Educational Communications and Technology Agency.


577

Research method One of the objectives of this research study was to investigate the feasibility of online surveying methods. Our purpose was to increase the knowledge base regarding online data collection in educational research, in particular at the secondary level. One of the research design issues discussed by Mertler (2002) is the benefit of using email for direct contact with members of the sample population. Being able to publicise a web-based survey and to encourage participation through email enables the researcher to determine the response rate, with a possibility of an increased confidence in the generalisability of the research results. Our research was performed in 15 upper secondary schools in Iceland during the autumn of 2002. It was based on two similar questionnaires, one sent to students and the other to teachers. The questionnaires were lengthy and divided into four sections beginning with a section of general questions about age, school, etc. The second section covered general questions about access to computers, the Internet, etc. The third section contained more specific questions concerning opinions about the use of computers and new technology in teaching and learning. Finally, the fourth section contained questions about the respondents’ experiences of distance and distributed learning. The questionnaires were digitalised and put online by Outcome (http://www.outcome.is), an e-survey system for conducting closed online surveys. Through this system, it was possible to send out an introductory letter to the potential respondents by email. In the email, the respondents were asked to visit a particular website in order to fill out the questionnaire. The e-survey system automatically generates a number for each participant that can only be used once, ensuring that the respondents can only complete the questionnaire once. The researcher does not have access to the user numbers and cannot trace the participants to the email addresses. This system of secured information guaranteed participant anonymity. On the other hand, because of the use of user numbers, the system could be used to send reminders to members of the population who had not completed the questionnaire. Finally, the system automatically gathered the data from the respondents and compiled them in an Excel file and a Word report. Reaching the sample population In this research study, we were faced with the challenge of reaching a large but defined population. Therefore, a web-based survey with direct contact through email was chosen for time- and cost-efficiency purposes. Permission to collect student and teacher email addresses was received from school authorities. In Iceland, many upper secondary schools provide students with email addresses and in those cases, the schools gave us a list with email addresses. In other schools, only teachers are provided with email addresses. Therefore, in some cases, we needed to collect student email addresses with help from teachers in the schools. In most of those cases we received only a limited number of email addresses and the ones © 2006 The Authors. Journal compilation © 2006 British Educational Communications and Technology Agency.

578


Vol 38 No 4 2007

we got were handwritten. This meant that in some cases, the email addresses were wrongly deciphered in the process of digitalising them. Also, many schools use the teaching programme WebCT. In this teaching programme, there is a closed email system reaching only students and teachers. The researchers did not have access to this closed system, which meant that the necessary contact information was not available for a certain percentage of students. In other cases, it was evident that students had more than one email address and did not necessarily use the one provided by their school. Therefore, the email message concerning the questionnaire did not reach all of the student respondents. They simply did not see the message. Another difficulty encountered in using the email addresses provided by the school authorities was of a technical nature. Because of the size of the target population and the questionnaire, the sending and receiving servers had trouble handling the volume of messages sent through the e-survey system. In an attempt to increase the response rate, letters were sent to school administrators asking them to encourage participation among faculty members and students. Also, a flyer was sent to schools for posting on bulletin boards and the survey was advertised on the schools’ closed-circuit televisions where applicable. In some cases, the school administrators were even contacted by telephone in order to create an interest for participation in the web survey. In addition, an email notice was sent to all potential participants through the use of an e-Plugs mailing list from Lavasoft (http://www.lavasoft.dk). Email messages were sent to 8575 students. The response rate was 24% or 2093 respondents. The number of emails in the student survey returned as incorrect addresses was 673 or 8%. In the teacher survey, 906 teachers were asked to participate and 423 filled out the questionnaire. Thus, the response rate was higher among teachers (47%). In the student survey, 65% of the respondents were female and 35% were male, although gender distribution for the upper-secondary students across the country is only slightly in favour of females (52% vs. 48%). The majority of the respondents (70%) were between 16 and 19 years old. These figures suggest that female students between 16 and 19 years of age are more likely to respond to online questionnaires than male students. In the teacher survey, the responses were much more equally distributed between female (48%) and male (52%) teachers and once again, gender distribution for teachers at that level across the country are quite equal (females—51%, males—49%). Findings Participation rates The figures above illustrate the need for caution when using an online data collection. The participation rates are low compared to those generally seen in traditional pencil© 2006 The Authors. Journal compilation © 2006 British Educational Communications and Technology Agency.


579

and-paper surveys. But the respondents freely choose to participate, which is a positive factor with regard to the quality of the responses. In the student survey, the females who participated were almost twice as many as the males who did (ages 16–19). This gender difference was not seen in the teacher survey, but the overall response rate was considerably higher among the teachers. This suggests that respondent characteristics such as gender, age, areas of interest or personal sense of duty may be important factors regarding participation in online surveys. The population lists used in the surveys were not complete or infallible. In particular, the percentages of student email addresses collected per school varied considerably, making a random sample impossible. Generalising the results to a larger target population is questionable, but we feel confident with many of the results because of the large total number of responses and the fact that they mirror the findings from other research performed in Iceland and in other countries (Mathiasen, 2002; Reid, 2002; Stefánsdóttir, 2001).

Reactions from respondents The voluntary nature of the surveys was reflected in remarks from both students and teachers. Some teachers and students expressed opposition to participation. Also, some of the respondents expressed that there were too many reminders (three), which caused aggravation. In some cases, the comments from students were even rude and formulated as ‘threats’. Many respondents complained that the survey was too time consuming and too long. The data reliability can be influenced by this factor if the respondents lose their concentration or interest before finishing the questionnaire. In these surveys, for example, the latter section was only intended for persons who had experience with distance learning. Nevertheless, some of the respondents filled out this section even though they said that they did not have any experience with distance learning. This kind of mistake is probably a sign of a lack of concentration by the respondents and might have been avoided had the questionnaire been shorter.

Data entry The use of a paper-and-pencil format demands the time-consuming manual procedure of entering the responses into a data file. The chances of human error during this procedure are obvious. The e-survey system that was chosen for the study eliminated this step because the respondents automatically entered their responses into the data file, which was compiled as an Excel document and a Word report. This, however, caused some technical problems because the Excel file generated by the system was not compatible with the software chosen for statistical analysis. © 2006 The Authors. Journal compilation © 2006 British Educational Communications and Technology Agency.

580


Vol 38 No 4 2007

Monetary savings Using the e-survey system resulted in considerable monetary savings despite having to pay for the services of the company. A traditional paper-and-pencil survey would have included costs for questionnaire administration including paper, printing, postage, training, travel costs and costs for data entry. All together, the costs for the online data collection were estimated to be less than half of what traditional data collection methods would have cost. Discussion Both practical and ethical questions arose from the use of the web-based survey. The research design of the online questionnaire presented issues that call for a closer examination. They include issues of representative sampling, response rates and the generalisability of the findings. It is important to improve the ways of making contact with potential participants. We chose to publicise the web survey and reach our target population through direct email messages and publicly posted notices. This permitted us to determine a response rate, but the method was not without drawbacks. Unused and incorrect email addresses diminished the accuracy of the population lists. In addition, not all schools had complete email lists for their student population and therefore, a random representative sample could not be obtained. However, the relatively large number of volunteer respondents gives confidence to the general reliability of the data. There is also an ethical issue involved in the use of email addresses that has to do with ownership. Is the email address the personal property of the individual or the institution that issues it? Some responses received from students indicated that they felt that the ownership of the email address was theirs. But the official position of the schools was that student and teacher email accounts were the property of the institution. This question of privacy and ownership of email accounts is under current debate within several sectors of society. A further concern regarding educational research in Iceland is the increasing number of requests for access to school populations for research purposes. Schools are inundated with requests for surveys and school administrators have begun to limit access to the schools for outside researchers in order not to disturb teaching. Often, connections are needed to get permission to conduct a survey in an upper secondary school, particularly in the case of paper-and-pencil surveys that require class time. However, web-based surveys have the advantage of causing less disturbance in the daily school routine. This is one of the reasons why online data collection is likely to replace more traditional data-collecting methods in the future. Another issue regarding web-based surveys has to do with encouraging good participation and improving response rates. Although paper-and-pencil surveys may provide higher response rates, certain measures can be taken to increase participation in online surveys. Various forms of incentives for participation can be provided to potential © 2006 The Authors. Journal compilation © 2006 British Educational Communications and Technology Agency.


581

respondents and the appeal for participation should be made as interesting and as nonthreatening as possible. The theoretical and practical importance of the research and its relevance should be made clear to potential respondents. Finally, it is important to design the research tool so that it is appealing to participants. The length of online questionnaires should be kept to a minimum and they should be easy to complete. Conclusions Online data collection carries the potential of accessing a large and geographically distributed population, along with being time and cost efficient for the researcher. Despite their limitations, web-based surveys provide researchers with unique opportunities for collecting data through the Internet. They can be particularly useful for collecting preliminary data and for pretesting research design and question comprehension. In addition to being advantageous for outside researchers, online data collection can also be useful for in-school purposes such as collecting information from students, teachers, staff and parents for school improvement. Our research suggests that differences may exist between population groups in reference to response rates. Careful consideration must be given to factors such as respondent age, gender, interest, maturity and duty to the profession. Because online surveys may no longer be as appealing as once believed, it is important that researchers give attention to the methods of encouraging participation in online data collection. Further research is needed to explore the reliability of the Internet as a data-collection tool and to better identify respondent characteristics and more effective recruitment methods. References Bachman, D., Elfrink, J. & Vazzana, G. (1996). Tracking the progress of email versus snail-mail. Marketing Research, 8, 31–35. Carbonaro, M. & Bainbridge, J. (2000). Design and development of a process for web-based survey research. The Alberta Journal of Educational Research, 46, 4, 392–394. Comley, P. (2000). Pop-up surveys: what works, what doesn’t work and what will work in the future. Retrieved November 10, 2004, from http://www.virtualsurveys.com/news/papers/ Ilieva, J., Baron, S. & Healey, H. M. (2002). Online surveys in marketing research: pros and cons. International Journal of Market Research, 44, 3, 362–380. Mathiasen, H. (2002). Personlige bærbare computere i undervisningen. København, Denmark: Danmarks Pædagogiske Universitet. Matz, C. M. (1999). Administration of Web versus paper surveys: mode effects and response rates (Master’s thesis, University of North Carolina). Retrieved November 11, 2004, from ERIC database. Mehta, R. & Sivadas, E. (1995). Comparing response rates and response content in mail versus electronic mail surveys. Journal of the Market Research Society, 37, 4, 429–439. Mertler, C. (2002). Demonstrating the potential for web-based survey methodology with a case study. American Secondary Education, 30, 2, 49–62. Moss, J. & Hendry, G. (2002). Use of electronic surveys in course evaluation. British Journal of Educational Technology, 33, 5, 583–592. Reid, S. (2002). Research in Ontario secondary schools. The integration of ICT into the classroom. Ontario: OISE/UT Trent Valley Centre. Saphore, R. B. (1999). A psychometric comparison of an electronic and classical survey instrument (Doctoral dissertation, University of Alabama, Dissertation Abstracts International, 60). © 2006 The Authors. Journal compilation © 2006 British Educational Communications and Technology Agency.

582


Vol 38 No 4 2007

Saunders, M., Lewis, P. & Thornhill, A. (1997). Research methods for business students. London: Pitman Publishing. Schaefer, R. & Dillman, D. A. (1998). Development of standard email methodology: results of an experiment. Public Opinion Quarterly, 62, 3, 378–397. Stanton, J. M. (1998). An empirical assessment of data collection using Internet. Personnel Psychology, 51, 709–725. Stefánsdóttir, L. (2001). Skyrsla um fartölvuverkefni Menntaskólans á Akureyri. Akureyri, Iceland: Menntaskólinn á Akureyri. Virtual Surveys Ltd (2001). Virtual surveys: web site research experts. Retrieved November 10, 2004, from http://www.virtualsurveys.com/services/vsurveys.htm

© 2006 The Authors. Journal compilation © 2006 British Educational Communications and Technology Agency.