J Acad Ethics (2009) 7:125–139 DOI 10.1007/s10805-009-9072-5
Ethical Issues in the Use of Multiple Survey Reminders Jacki Schirmer
Published online: 22 May 2009 # Springer Science + Business Media B.V. 2009
Abstract This paper identifies and discusses ethical issues associated with the increasingly common use of multiple reminders in postal survey research. When undertaking postal surveys, it is common to improve response rates by contacting survey recipients multiple times encouraging them to complete the questionnaire. The ethical implications of the use of multiple reminders are rarely discussed in the literature advocating their use. The paper briefly reviews the arguments encouraging use of multiple reminders, and identifies and discusses the potential ethical issues raised by their use in postal surveys. A set of principles is then proposed for using multiple reminders in an ethically responsible way. It is argued that by following these principles, the use of multiple reminders should present no increased potential for harassment or coercion of survey recipients compared to alternative survey techniques. The use of these principles in a recent postal survey is then presented and discussed; a 60% response rate was achieved utilising multiple survey reminders in a way consistent with this set of principles. Keywords Coercion . Ethics . Harassment . Postal surveys . Reminders . Response rates
Introduction This paper identifies and discusses potential ethical issues associated with the increasingly common use of multiple reminders in postal survey research. Postal surveys are commonly utilised by researchers in a wide range of disciplines across the social and physical sciences. A key issue when undertaking a postal survey is achieving an adequate response rate. Response rates of above 50 per cent are necessary in many circumstances to achieve a
J. Schirmer Fenner School of Environment and Society, Australian National University, Canberra, ACT 0200, Australia J. Schirmer (*) Cooperative Research Centre for Forestry, Hobart, Australia e-mail:
[email protected]
126
J. Schirmer
sample suitable for statistical analysis, but response rates as low as 10 to 15 per cent are common unless a range of techniques are used to increase response rates (Porter 2004; Dillman 2007). Multiple reminders — letters, brief reminder cards and repeat copies of questionnaires sent to survey recipients to remind them about the survey and request they complete it — are increasingly used as a way of improving postal survey response rates (Dillman 2007). In this paper, the term ‘multiple reminders’ is used inclusively to refer to all types of contact made with survey recipients pre- and post-survey. The ethical implications of the use of multiple reminders have rarely been discussed in the literature on their use. However, anecdotal evidence indicates that human research ethics committees (HRECs) are raising concerns about the use of multiple reminders in survey research, and sometimes seek to reduce the number of reminders used (Howell et al. 2003; Robertson et al. 2005; the author has also experienced this concern in interactions with her local HREC). The key issue for HRECs appears to be that multiple reminders are believed to involve harassment or coercion of survey recipients, threatening the ethical requirement that survey participation be voluntary (Howell et al. 2003: 43; Robertson et al. 2005): To be useful, surveys need to be returned in sufficient numbers, so maximising response rates is an important aspect of study design. While three to four reminders are often recommended, increasingly ethics committees are viewing this number of contacts as possible harassment of research subjects (Robertson et al. 2005, p. 571). The paper begins with a review of the methodological imperatives for using multiple reminders, with other options for improving survey response compared from the point of view of the researcher aiming to maximise the validity and reliability of their survey responses. The potential ethical issues of using multiple reminders versus alternatives are then identified and discussed. The ethical issues considered are drawn largely from Australia’s National Statement on Ethical Conduct in Research Involving Humans (NHMRC 1999), as well as from international literature on ethical considerations in survey research. A set of principles are proposed for using multiple reminders in an ethically responsible way. It is argued that by following these principles, the use of multiple reminders should present no increased potential for harassment or coercion of survey recipients compared to alternative survey techniques. The use of these principles in a recent postal survey is then presented and discussed. A 60% response rate was achieved utilising multiple survey reminders following the principles proposed in this paper.
Is the Use of Multiple Reminders Necessary? Howell et al. (2003) argue that ethics committees should allow use of multiple reminders because they are essential to achieve scientifically valid survey research. This argument is reviewed below with reference to guidelines for ethical research involving humans in Australia; these principles are similar to those used internationally. The Australian NHMRC (1999) guidelines on ethics in human research indicate that ethics committees may approve research procedures if (a) the procedure is scientifically justified, (b) there is no acceptable alternative and (c) reasonable steps are taken to protect individual rights. Multiple reminders should therefore only be used in survey research if they are scientifically justified, there are no alternatives, and individual rights can be adequately protected. This
Ethical Issues in the Use of Multiple Survey Reminders
127
section addresses the first two of these considerations, reviewing the evidence for the necessity of using multiple reminders and the alternative options available to researchers. This necessitates a brief review of the goals and needs of scientifically valid survey research.
The Role of Response Rate in Achieving Scientifically Valid Survey Research The goal of all quantitative survey research — whether the survey is delivered by phone, face to face, or by mail — is to gather data that provides useful information about a particular population of humans. To achieve this, it is essential the survey obtains responses from a representative sample of that population. Perhaps the greatest challenge in obtaining a representative sample is that of achieving a suitably high response rate to a survey. Low response rates generally increase survey response bias, as some groups will be more likely to respond than others, skewing the survey results (Porter 2004). While face to face surveys typically have high response rates, they are commonly prohibitively expensive, and many researchers cannot obtain adequate funding to use this survey instrument (Sitzia and Wood 1998). Telephone surveys are often used, but have increasingly been associated with a range of problems. In particular, it is often difficult to achieve a representative sample using phone surveys due to refusals to participate, increasing use of privately listed phone numbers, and the problem of timing phone calls to reach people who are often only home a small number of hours each day (Bourque and Fielder 2003). As a result, many people utilise postal surveys, which have a range of advantages compared to other survey instruments, including (Kanuk and Berenson 1975):
& & & &
the ability to ask sensitive questions that people are more willing to answer on paper than face to face or by phone; allowing respondents to answer questions in their own time and to refer to records if necessary; better ability to target a representative sample of the population than other methods; and lower costs than most other survey methods.
However, unless the survey is distributed using methods designed to increase response rates, it is common for response rates to a postal survey to be as low as 10 to 15 per cent. Response rates this low considerably reduce ability to identify representative characteristics of a surveyed population. There is evidence that willingness to respond to surveys has fallen over time in many cultures, although this falling response rate has mostly been documented in surveys of the general population, rather than surveys of specific subgroups e.g. farmers or particular interest groups (Porter 2004). While figures vary, most researchers agree that, ideally, a 50 to 75 per cent response rate is needed to produce reliable evidence from a well designed representative sample survey delivered by any method, including post (Howell et al. 2003; Robertson et al. 2005). Some argue that up to an 85 per cent response rate is needed for certain types of studies, such as epidemiological studies (Ward 2003).
Techniques for Improving Response Rates: What Works? A large body of research has examined methods for improving postal survey response rates, to a level sufficient to enable valid generalisations to be made about the population sampled.
128
J. Schirmer
Much research on improving survey response rates has been atheoretical and based on experimenting with particular techniques (Porter 2004). However in recent years some theories of survey response behaviour have developed to explain why response rates increase in certain situations. The most influential theory is based on social exchange theory and posits that response rates increase when survey researchers ensure they:
& & &
increase survey recipients’ perceptions of the individual or social gains, or rewards, of participating in the survey; decrease the costs a respondent faces to complete a survey (including time, money); and increase trust in the survey and the researchers undertaking it, to ensure recipients believe the rewards of participating are greater than the costs (Dillman 2007).
This theory is closely related to evidence that has consistently shown that the overall ‘pleasantness’ of a survey experience influences a person’s willingness to complete surveys (Nederhof 1987). It is generally agreed that the following methods can all improve survey response rates, consistent with social exchange theory (Kanuk and Berenson 1975; Warriner et al. 1996; Kypri and Gallagher 2003; Porter 2004; Robertson et al. 2005; Dillman 2007):
&
& &
&
& &
Careful design of survey. This includes design of the layout, questions and overall length of the questionnaire, with comprehensive pre-testing of survey questions and layout recommended during the survey design process. This reduces the costs or burden imposed on the survey respondent by making it easier to complete the survey; Increasing the relevance, or salience, of the questionnaire. Respondents are more likely to respond to questions they feel are relevant to them; Sending a pre-survey letter. The letter should explain the nature of the survey, particularly its goals and why it is worthwhile to complete, and how ethical issues are being addressed. This increases recipient’s trust in and understanding of the potential rewards of participating in the survey; Having several points of contact (reminders) with survey recipients. Multiple reminders take many forms, but researchers using Dillman’s Tailored Design Method typically have up a six ‘contact points’ with survey respondents, beginning with an introductory letter prior to sending a survey, followed by posting the survey itself, sending a weekly postcard reminder, and a second copy of the survey (Dillman 2007); Surveys sent by government or academic organisations typically achieve higher response rates than surveys sent by commercial businesses; and Providing a financial or other incentive. This is most successful if an incentive is sent to all recipients with the initial survey, although some argue it is successful if it is only paid to those who respond to the survey.
Other incentives, such as offering a prize draw won by only a small number of respondents (as opposed to a financial incentive given to every respondent), have not been consistently found to increase response rates, with varying levels of effectiveness found in different studies (Dillman 2007, p. 169–170). Postpaid incentives and prizes do not consistently increase response rates (Porter 2004, p. 13; Dillman 2007). The use of scratch lottery tickets has similarly had inconsistent results and cannot be said to increase response rates in all situations (Finsen and Storeheier 2006). There is considerable debate over how the mode of survey delivery affects response rates. Some argue that web based delivery achieves higher response rates than a paper
Ethical Issues in the Use of Multiple Survey Reminders
129
questionnaire; others disagree; and the reality is likely that the effectiveness of particular modes cannot be considered to be universal but must be considered in the context of the specific topic of the survey, and the population/s being surveyed (Porter 2004). Of the methods for increasing survey response listed above, the most effective is the use of multiple contact points, referred to in this paper as multiple reminders. The use of multiple reminders, combined with ensuring the questionnaire is well designed and easy to complete, typically increases response rates to 60 to 70 per cent. The effectiveness of multiple reminders compared to other technique has been confirmed in several meta-analyses of survey response rates since the 1970 s1 (see Kanuk and Berenson 1975; Asch et al. 1997; Dillman 2007), and is well accepted by researchers examining how to improve survey response rates: Without follow-up contacts, response rates will usually be 20–40 percentage points lower than those normally attained, regardless of how interesting the questionnaire or impressive the mailout package. This fact makes a carefully designed follow-up sequence imperative. (Dillman, 2007 p. 177) While the magnitude of the increase has varied across different studies, it is typically of the order of two to three times the response rate achieved without reminders, particularly when multiple reminders are combined with other survey design methods known to increase response rate (Porter 2004; Dillman 2007). Other methods of increasing response rate, meanwhile, generally show a much smaller increase in response rate. Ensuring questionnaires are of moderate length has a significant but small impact on responses, of around three to four percentage points (Porter 2004). Prepaid incentives of $US1 to $US5 per respondent can increase response rates by anywhere from 12 to 24 percentage points, making them the most effective approach after the use of multiple reminders (Porter 2004). What does this mean for an Ethics committee considering whether to permit the use of multiple reminders? There is strong evidence that (a) multiple reminders are a scientifically justified method for ensuring validity of survey research, and (b) alternative approaches do not raise response rates as effectively as using multiple reminders. This provides a strong argument for their use based on some of the key criteria considered by most HRECs. The question that then remains for ethics committees is whether individual rights can be adequately protected when using multiple reminders — in other words, the ethical considerations involved in utilising multiple reminders in survey research.
Ethical Considerations When Using Multiple Reminders All research involving humans needs to fulfil a range of ethical obligations. Survey research is no different, with researchers needing to address a range of issues to ensure participants are not at risk of harm (McKeown and Weed 2004). Five primary ethical considerations are discussed in this paper:
&
1
Voluntary participation: Ensuring respondents are not coerced inappropriately into completing the survey and are able to withdraw from the survey without prejudice before or after completing the questionnaire;
There is debate over how differently timed reminders affect overall response rate, particularly the importance of pre-survey versus post-survey contacts (Chiu and Brennan 1990). This does not change the overall finding that multiple reminders are effective in increasing response rates, but rather influences the extent to which different reminder timings increase response rates.
130
& & & &
J. Schirmer
Confidentiality and privacy: Ensuring a respondents’ identity remains confidential and their participation or non-participation has no negative repercussions on their lives; Representativeness: Ensuring the survey is inclusive of all relevant groups and does not discriminate against minorities or disadvantaged groups; Risk to study participants: Ensuring the study does not create or increase risks faced by study participants (this is related to issues of confidentiality and privacy); and Respondent burden: Ensuring that completing the survey presents as little burden as possible on the lives of respondents.
While these are not the only ethical issues to be considered when undertaking survey research, they are some of the principle areas commonly considered by ethics committees and survey researchers. Each of these five areas is discussed below, focusing on the primary ethical questions that need to be addressed when undertaking postal survey research, and the potential challenges raised by use of multiple reminders compared to other survey techniques. Voluntary Participation Guidelines for ethical conduct in university-based survey research generally include a requirement that survey participation must be voluntary. Researchers have an obligation to ensure that people asked to participate in survey research do not feel unduly pressured to participate, and are fully aware of the voluntary nature of the survey. It is common for ethics committees to require that researchers ensure study participants can withdraw from a survey even after initially completing and returning a questionnaire (see for example NHMRC 1999). In principle, the ethical issues relating to voluntary participation are no different for surveys that utilise multiple reminders and those that do not. In both cases, it is essential that participants are clearly informed of their rights both to choose not to participate in, and to withdraw existing participation in, a study. There are, however, practical differences. When multiple reminders are used, survey recipients receive regular requests for several weeks asking them to participate in the study. Many concerns raised by HRECs about use of multiple reminders relate to the impact of this practice on the voluntary nature of participation. The concern is that multiple reminders may lead to survey recipients feeling unduly pressured to participate in a study, as they feel their only option to stop the reminders arriving in their mail is to complete the questionnaire sent to them (Howell et al. 2003: 43; Robertson et al. 2005). There are many who would argue this is not the case. For example, some research has found that reminders (particularly those sent in the first one to 2 weeks after sending a questionnaire) achieve responses from people who had forgotten to complete the questionnaire despite being willing to complete it (Dillman 2007). In this situation the use of reminders does not coerce unwilling participants into responding. It seems likely, however, that at least some respondents may feel unduly coerced into responding as a result of receiving multiple reminders. Despite an intensive search, no literature was found that has specifically investigated whether this is, in fact, the case. This is a key gap in survey research literature which needs to be addressed. To prevent the potential for coercion, advocates of multiple reminders generally recommend that participants be provided with a clearly described and easily undertaken means for withdrawing from the study that enables them to stop further reminders being sent.
Ethical Issues in the Use of Multiple Survey Reminders
131
For example, Dillman (2007) suggests a system in which participants can return an uncompleted questionnaire in the pre-paid envelope provided to indicate withdrawal from the study. When the uncompleted questionnaire is received, no further reminders are sent. This involves no cost and very little time for the respondent. An alternative is to provide a free call telephone number participants can call to withdraw from the study, again imposing no cost and requiring very little time on the part of the survey recipient. Providing simple, accessible means of withdrawing from the study, clearly communicating these, and ensuring those who withdraw receive no further reminders should address ethical concerns over potential for respondents to feel coerced to participate. To ensure participants avoid feeling harassed, the first survey contact should clearly inform survey recipients that they will receive multiple survey reminders, and explain how they can withdraw from the study. This ensures participants are able to avoid being sent reminders altogether by withdrawing from the study after receiving an initial contact letter. Green (1996) compared response rates from respondents who had been explicitly informed in the initial survey contact that reminders would be sent to those who had not. Green found that those who had been informed (a) had higher response rates prior to the first reminder, and (b) were more likely to actively withdraw from the study (rather than passively choose not to participate without informing researchers of the decision not to take part). This may indicate that some survey participants will use this information to choose to withdraw from the study and prevent being ‘harassed’ by reminders, but also suggests that providing clear options for withdrawing will not necessarily have a negative impact on response rates, as many of the people who actively withdraw would likely have fallen into the group of passive non-responders if the survey had not utilised multiple reminders. Informed Consent The process of obtaining consent has two aspects according to Australia’s 1999 National Statement on Ethical Conduct in Research Involving Humans: ‘the provision of information and the capacity to make a voluntary choice’ (NHMRC 1999, p. 12). The capacity to make a voluntary choice was discussed in the previous section on voluntary participation. The remaining consent issue — that of providing adequate information — is discussed in this section. Ethical issues relating to providing adequate information for informed consent are identical whether or not multiple reminders are used. When reviewing the literature, no differences could be found regarding consent procedures when survey reminders were used versus when they were not. The only potential difference is the need to clearly inform questionnaire recipients at the point of first contact that multiple survey reminders will be sent to them if they do not respond, and that they have the right to withdraw from the study. This was addressed in the previous section. Whether or not multiple reminders are used, a key issue is the language used when providing information. In some cases HRECs specify the language to be used. This has resulted in concern from researchers that requirements for overly formal language reduce response rates, as this language may create undue confusion and fear about the potential consequences of participating in the survey. Grayson and Myles (2004) compared response rates to a questionnaire delivered in two different areas, which was identical in all respects except that the introductory information differed. In one region, the introductory information was phrased in dense legal language required by the university ethics committee which approved the research. In the other region the content was the same, but the language and phrasing used were designed to be more accessible to survey
132
J. Schirmer
respondents. The questionnaires sent with less ‘legalese’ language in the introductory information achieved a higher response rate than those in which information was provided in ‘legalese’. As a result, Grayson and Myles (2004) argue that HRECs should focus on specifying the content, rather than form, of information required to be communicated to questionnaire recipients. In some cases, HRECs are requiring that explicit signed consent be received before a questionnaire can be delivered. The process of obtaining prior consent before delivering a questionnaire appears to consistently and significantly decrease response rates (Angus et al. 2003). This decrease in response rate is so large it can prevent the survey producing valid or reliable results, and so is not considered appropriate to simultaneously achieving the goals of (a) scientifically credible research and (b) protecting individual rights. It is also unclear whether consent is more informed if potential questionnaire recipients are unable to view the survey instrument when making their decision on whether to respond to a survey. As the use of multiple reminders does not lead to new or different issues in the process of obtaining informed consent, beyond needing to clearly inform recipients in the first contact that multiple reminders will be used, this issue is not discussed further in this paper. Confidentiality and Privacy Ethics guidelines typically require that the identity of participants in a study be kept confidential wherever possible (NHMRC 1999). There are no major differences in how confidentiality and privacy issues are addressed in surveys which use multiple reminders versus those who do not. The only potential difference is that when multiple reminders are used, it is common to record an individualised number on each questionnaire. This enables researchers to remove the recipient from the list of people to be sent survey reminders once they have completed the questionnaire. This type of numbering system is also commonly used in surveys involving no reminders, as it can be used to analyse some types of non-response bias. To maintain privacy, it is common to store questionnaire recipient’s contact details separately to survey results, ensuring survey responses are not linked in a single database/spreadsheet to respondent’s contact details. As no differences could be identified in survey practice that related to the use/non-use of reminders, it is argued here that use of multiple reminders presents no new or different confidentiality or privacy considerations to those presented in all survey research. Representativeness McKeown and Weed (2004) argue that survey researchers have a particular ethical obligation to ensure their research includes all relevant groups, and does not exclude any groups who are typically underrepresented in research. To meet this ethical obligation, survey researchers have to both ensure that their survey sample adequately represents all relevant groups — an issue that is no different whether multiple reminders are used or not — and use methods which encourage response from all survey recipients, including those who may otherwise be underrepresented. The use of multiple reminders is relevant to achieving the latter. Improving response rates from particular groups may require a range of measures, such as including questionnaire information in multiple languages, providing support for those who have low literacy skills to complete the survey, and ensuring questions are phrased in ways understood by and relevant to different groups.
Ethical Issues in the Use of Multiple Survey Reminders
133
The use of multiple reminders is aimed at improving response rates across an entire survey, rather than from particular groups. However, increasing total response may improve the representation of groups who otherwise might be marginalised, through increasing their typically low response rate to one high enough to enable statistically valid analysis of that group. It is, however, possible that multiple reminders will increase response rates from some groups and not others, and may therefore have limited usefulness in this respect. Some studies have found that ‘late responders’ — those who respond only after several reminders have been sent — have different characteristics to ‘early responders’ — those who return a questionnaire without reminder, or with fewer reminders (Etter and Perneger 1997; Barclay et al. 2002). This indicates that use of multiple reminders is likely to result in response from groups that may otherwise be underrepresented in survey results. Hence it seems likely that the use of multiple reminders may assist in achieving the ethical obligation to ensure surveys are inclusive of the views of particular groups who may be easily ‘missed’ using other survey techniques, although further work is needed to identify which commonly ‘missed’ groups are reached via multiple reminders, and which are not. Risk to Study Participants Ethics guidelines commonly define ‘risk’ to survey participants as involving any way in which the questions asked, or requests to participate in a survey, could lead to self harm or stress amongst study participants (NHMRC 1999). Perhaps the most common risk discussed is the risk of participants, or information about their lives, being identified in research results in ways that invade their privacy and cause them harm. This was discussed in the section on privacy and confidentiality issues. It is also possible that survey questions on sensitive topics may create distress for participants. This risk must be addressed through careful questionnaire design and consideration of the risk potential. The risk does not differ if reminders are or are not used as part of the survey methodology. Multiple reminders have potential to increase risks to study participants if the reminders act to reinforce distress created by the sending of the questionnaire. This can be avoided by ensuring participants are provided opportunities to withdraw from the study from the point of first contact, as discussed earlier. Respondent Burden A key focus of survey researchers in recent years has been on ‘reducing response burden, both actual and perceived, by minimising costs, time, effort and the number of people involved’ in completing surveys (Snijkers and Luppes 2002, p. 206). Response burden generally refers to ‘the time and effort involved in participating in a survey’ or in multiple surveys over time (Porter et al. 2004, p. 64). Burden differs from individual to individual: Some people happily participate in incredibly long surveys, sometimes repeatedly, and find it little or no burden, whereas others are either bored or rude within one minute. (Ampt 2000, p. 3) It is generally believed that high respondent burden is correlated with lower survey response, although studies have not consistently demonstrated this link (McCarthy and Beckler 1999). As discussed earlier, response burden is minimised largely by ensuring questionnaires are well-designed. In particular, designers should ensure questions are relevant, that the layout enables easy completion of the questionnaire, and that length and
134
J. Schirmer
difficulty are minimised. Typically questionnaires are tested on a small sample and revised, sometimes several times, to achieve good question and layout design (Snijkers and Luppes 2002). Does the use of multiple reminders change the level of respondent burden imposed on questionnaire recipients? This question has not been addressed in much of the literature discussing respondent burden, although in some discussions the definition of respondent burden has been broadened to include ‘perceived difficulty, dissonance or intrusion that individuals associate with a survey’ (Ampt 2000, p. 1). The use of multiple reminders may have a bearing on the level of intrusiveness survey recipients associate with a survey. Some analyses of respondent burden have focused on the number of times respondents are asked to complete a questionnaire, focusing in particular on panel surveys where repeated questionnaires must be completed over time. Studies of response rates in panel surveys indicate that response rates tend to fall over time as individuals being re-surveyed feel the burden imposed is too high (McCarthy and Beckler 1999; Porter et al. 2004). Similarly, respondent burden appears to increase if a person is asked to complete multiple separate surveys in a short space of time, with response rates falling as more surveys are added (Porter et al. 2004). There are some contradictory findings, however — McCarthy and Beckler (1999) examined the percentage of people who refused to take part in agricultural surveys over time, and did not find that numbers of refusals were correlated with the number of times a potential survey recipient was contacted. None of the literature reviewed examined whether repeated contacts with a survey recipient about a single questionnaire leads to increased respondent burden.2 The findings reported above on multiple surveys are not necessarily relevant to the use of multiple reminders, as they involve repeated questionnaires being delivered to the same person, rather than several reminders about a single questionnaire. Given that response rates increase with multiple reminders, it is likely that there is a different reaction to multiple reminders compared to multiple surveys over time. The findings reported above are relevant in another way to discussion of the impact of multiple reminders, as they suggest that people experience higher respondent burden the more surveys they are asked to participate in. The use of multiple reminders ensures a higher response rate than would be achieved without reminders. This means that researchers can send questionnaires to a smaller number of recipients with confidence that a high response rate will be achieved. This is an important effect, as it means widespread use of multiple reminders may result in researchers sending each questionnaire to a smaller sample, to individuals receiving fewer surveys over time, and hence to lower overall response burden. The timing of a survey is a key factor in determining respondent burden. Telephone and face-to-face surveys, which often require a respondent to answer questions immediately, are generally considered most burdensome. Postal, email and internet surveys, which allow respondents to choose the most appropriate time to complete the survey, are often considered least burdensome (Ampt 2000). Use of posted multiple reminders, while
2
The measure of respondent burden generally used when identifying if multiple contacts raise burden is response rate, with higher non-response considered indicative of higher respondent burden. Using the response rate measure, it could be argued that the use of multiple reminders does not increase respondent burden as it consistently results in an increased, rather than decreased, response rate. However, other measures of response burden are possible. Given that multiple reminders are known to increase response rates significantly, any study examining respondent burden and multiple reminders should use measure burden using a measure other than survey response rate.
Ethical Issues in the Use of Multiple Survey Reminders
135
reminding the recipient about the survey, still enable a recipient to choose a time that imposes the least burden on them to complete the survey. As multiple reminders are also the main technique that can successfully increase postal survey response rates (Brennan 1992), they are a key part of enabling successful use of postal surveys and hence taking advantage of their usefulness in reducing this aspect of respondent burden. Ampt (2000) argues that respondent burden can be decreased by improving a survey recipient’s understanding or perception of the relevance of a questionnaire. She recommends reiterating the relevance of a survey over time — something which multiple reminders are generally designed to do. This suggests that including statements in the text of reminders that assist recipients in assessing the relevance of a survey to them may assist in reducing respondent burden.
Proposed Principles for Ethical Use of Multiple Reminders Based on the review and discussion above, five principles are proposed below for the ethical use of multiple reminders in survey research. These are designed to ensure key ethical considerations are addressed, while enabling the utilisation of this important technique for improving survey response rates. 1. Inform respondents there will be multiple reminders. At the point of initial contact (usually a pre-survey letter or the sending of the questionnaire itself), participants should be informed that reminders will be sent about the survey. This will ensure participants are aware of the use of multiple reminders. In conjunction with Principle #2 below, it allows participants to withdraw from the study and avoid being sent multiple reminders at all. 2. Ensure participants can withdraw easily. Ensure participants can withdraw from the study at any time without completing the questionnaire. The ability to withdraw should be clearly communicated in each contact with the survey recipient. After a questionnaire recipient withdraws from the study, the researcher must guarantee that no further reminders are posted to that person. Preferably multiple methods for withdrawing should be provided, to enable individuals to choose a method that is most comfortable for them (for example, providing both a free call phone number and the option of returning an uncompleted questionnaire in a pre-paid envelope). This measure ensures that respondents who feel multiple reminders are burdensome can prevent further reminders being sent to them. 3. Remind only non-respondents. Ensure that reminders are only sent to non-respondents. This reduces any potential aggravation for those who have already completed the questionnaire. 4. Survey fewer people. Ensure that the higher response rate from use of multiple reminders is taken into account when calculating sample size. The higher response rate achieved as a result of use of multiple reminders will mean a smaller sample can be used compared to the use of no reminders, as fewer questionnaires need to be distributed to achieve a suitable number of responses. This can reduce respondent burden by reducing the total number of people sent questionnaires. 5. Ensure reminder text provides information on the relevance of the survey. This measure will assist survey recipients in judging the worth of completing the questionnaire. If a survey is considered relevant and worthwhile, the survey respondent is likely to experience less respondent burden.
136
J. Schirmer
6. Stop sending reminders once a high response rate is achieved. To reduce potential for harassment, once an appropriate response rate has been achieved that enables statistically valid analysis to be undertaken, no further reminders should be sent, even if only a small number of reminders have been sent in total. This may be a somewhat controversial proposal for researchers whose goal is to obtain the highest possible response rate, but should be considered for its potential to reduce survey burden.
Case study: Use of the Proposed Principles in a Multiple Reminder Postal Survey The principles proposed above were implemented in the delivery of a small postal survey to a sample of rural Australian landholders in 2007. The use of multiple reminders as part of the survey was initially rejected by the relevant HREC, who raised concerns about harassment of survey participants. In response, the principles outlined in this paper were proposed and accepted by the HREC. This provided an opportunity to utilise the principles and identify whether it was still possible to achieve a high survey response rate. The survey aimed to better understand why rural landholders in parts of the Australian states of Victoria and South Australia had chosen to become involved in commercial tree plantation. Landholders sent the questionnaire had either established a commercial tree crop on their own property, leased land to a plantation company, or sold land to a plantation company. Results of the study are reported in Schirmer (2008). The survey was designed to achieve high response rates by:
& & & & &
having a thorough question design and testing process in which both content and layout of questions were thoroughly tested with a small sample of rural landholders; mailing the questionnaire together with a stamped self-addressed envelope for survey return; sending a personalised letter with the questionnaire, encouraging survey return; having a toll-free phone number survey recipients could call to receive advice and assistance with completing the questionnaire; and use of multiple reminders, in this case involving a pre-survey phone call, posting the questionnaire, and then posting reminders and a second copy of the questionnaire.
Ethical issues were addressed by clearly informing survey recipients from the first point of contact onwards:
& & &
that they would receive survey reminders; that they could withdraw from the survey by either posting an uncompleted questionnaire back in the prepaid envelope, or by calling the toll-free number provided; and why the survey was relevant for them to complete
The questionnaire was sent to a total of 262 landholders. This was the sample needed for a adequate response rate, assuming a 50 per cent response rate. Without the use of multiple survey reminders, a lower response rate would have been assumed and the questionnaire sent to a larger sample. Therefore the use of multiple reminders enabled a reduction in response burden in general as fewer landholders were surveyed than would otherwise have been the case. Initially, multiple reminders were planned; however high response rates were received after the first two contact points (the initial phone call and sending of the initial
Ethical Issues in the Use of Multiple Survey Reminders
137
questionnaire); and after an initial reminder — a second copy of the questionnaire - was sent, response rates reached 60.3 per cent. As the response rate was high, it was decided not to send further reminders, to reduce potential respondent burden. This experience indicates that the principles proposed in this paper can be used while still achieving a high response rate. It is, however, a single case study of a survey which involved a topic with high salience for recipients, sent to a specific groups of people. The extent to which it indicates likely outcomes of use of the principles in other situations is limited.
Discussion and Conclusions Ethics committees are understandably concerned about the potential impact of the practice of sending multiple reminders on survey recipients. This paper has identified several areas in which there are clear areas of ethical concern raised by the use of multiple reminders, as well as areas where multiple reminders can assist in achieving ethical obligations in survey research. Important ethical issues arise when multiple reminders are used in postal survey research. It is surprising that little research on survey methods has focused on whether use of multiple reminders increases response burden, particularly in relation to survey recipients feeling coerced or harassed into completing a questionnaire. Given that ethics committees are raising concerns about this issue, there is a clear need for research into this area. This research should be carefully designed, however, to examine all aspects of the use of multiple reminders, including whether the reduced sample size needed as a result of higher response rates compensates for any additional burden experienced by survey recipients when sent multiple reminders. Reducing the total numbers of surveys sent has considerable potential to reduce survey burden, particularly given the concerns raised about overall falling survey response rates as people become less willing to respond to the increasing number of questionnaires they are asked to complete on a regular basis. The brief case study presented in this paper indicates that a set of relatively simple ethical principles can be utilised when sending multiple survey reminders, while still achieving a high response rate. These principles are designed to address the most common ethical concerns, by clearly informing survey recipients that multiple reminders will be used, enabling them to easily withdraw from the survey, and only using as many reminders as needed to achieve an adequate response rate. Further research is needed which tests these ethical guidelines in different situations, to identify whether there are implications for response rates when following the proposed ethical guidelines in a wider range of circumstances. The case study presented here involved a relatively small survey of a specific group of people, and cannot be said to be generalisable to other survey situations. Rather than refusing the use of multiple reminders, ethics committees should ensure that appropriate practices are put in place to ensure multiple reminders do not compromise ethical guidelines. These practices are simple to implement, and mostly relate to ensuring survey participants can easily withdraw from a study at any time without cost, and that reminders are utilised in a way that helps survey recipients understand the survey and why they have been sent it. These practices can be utilised in at least some circumstances without significantly decreasing response rates, enabling multiple reminders to be utilised in an ethically appropriate manner without compromising the quality of the research data.
138
J. Schirmer
Acknowledgments The comments and suggestions of an anonymous reviewer are gratefully acknowledged.
References Ampt, E. (2000). Understanding the people we survey. Resource paper prepared for Workshop on Respondent Burden. Transportation Research Circular, Transportation Research Board, Washington USA. Retrieved June 18th, 2007, from http://pubsindex.trb.org/document/view/default.asp?lbid=686587 Angus, V. C., Entwhistle, V. A., Emslie, M. J., Walker, K. A., & Andrew, J. E. (2003). The requirement for prior consent to participate on survey response rates: a population-based survey in Grampian. BMC Health Services Research, 3, 21–30. doi:10.1186/1472-6963-3-21. Asch, D. A., Jedrziewski, M. K., & Christakis, N. A. (1997). Response rates to mail surveys published in medical journals. Journal of Clinical Epidemiology, 50(10), 1129–1136. doi:10.1016/S0895-4356(97)00126-1. Barclay, S., Todd, C., Finlay, I., Grande, G., & Wyatt, P. (2002). Not another questionnaire! Maximizing the response rate, predicting non-response and assessing non-response bias in postal questionnaire studies of GPs. Family Practice, 19(1), 105–111. doi:10.1093/fampra/19.1.105. Bourque, L. B., & Fielder, E. P. (2003). How to conduct telephone surveys (2nd ed.). London: Sage. Brennan, M. (1992). Techniques for improving mail survey response rates. Marketing Bulletin, 3, 24–37. Chiu, I., & Brennan, M. (1990). The effectiveness of some techniques for improving mail survey response rates: a meta-analysis. Marketing Bulletin, 1, 13–18. Dillman, D. A. (2007). Mail and internet surveys: The Tailored Design Method (2nd ed.). Hoboken: Wiley. Etter, J.-F., & Perneger, T. V. (1997). Analysis of non-response bias in a mailed health survey. Journal of Clinical Epidemiology, 50(10), 1123–1128. doi:10.1016/S0895-4356(97)00166-2. Finsen, V., & Storeheier, A. H. (2006). Scratch lottery tickets are a poor incentive to respond to mailed questionnaires. BMC Medical Research Methodology, 6, 19–23. doi:10.1186/1471-2288-6-19. Grayson, J. P., & Myles, R. (2004). How research ethics boards are undermining survey research on Canadian university students. Journal of Academic Ethics, 2, 293–314. doi:10.1007/s10805-0059005-x. Green, J. M. (1996). Warning that reminders will be sent increased response rate. Quality & Quantity, 30, 449–450. doi:10.1007/BF00170147. Howell, S.C., Quine, S., & Talley, N.J. (2003). Ethics review and use of reminder letters in postal surveys: are current practices compromising an evidence-based approach? The Medical Journal of Australia, 178, 43. Letter Kanuk, L., & Berenson, C. (1975). Mail surveys and response rates: a literature review. JMR, Journal of Marketing Research, 12(4), 440–453. doi:10.2307/3151093. Kypri, K., & Gallagher, S. J. (2003). Incentives to increase participation in an internet survey of alcohol use: a controlled experiment. Alcohol and Alcoholism (Oxford, Oxfordshire), 38(5), 437–441. doi:10.1093/ alcalc/agg107. McCarthy, J., & Beckler, D.G. (1999, October). An analysis of the relationship between survey burden and non-response: If we both them more, are they less cooperative? (Paper presented at the International Conference on Survey Non-Response, Portland Oregon). Document retrieved June 15th, 2007, from http://www.jpsm.umd.edu/icsn/papers/McCarthyBeckler.htm McKeown, R. E., & Weed, D. L. (2004). Ethical choices in survey research. Social and preventive medicine, 49, 67–68. Nederhof, A. J. (1987). When neutrality is negative: pleasantness of most recent survey experience and nonresponse. Quality & Quantity, 21, 425–432. doi:10.1007/BF00172567. NHMRC (National Health and Medical Research Council).(1999). National Statement on Ethical Conduct in Research Involving Humans. Issued by the National Health and Medical Research Council (NHMRC) in accordance with the NHMRC Act, 1992 (Cth). (Canberra, Commonwealth of Australia) Porter, S. R. (2004). Raising response rates: what works? New Directions for Institutional Research, 121, 5– 21. doi:10.1002/ir.97. Porter, S. R., Whitcomb, M. E., & Weitzer, W. H. (2004). Multiple surveys of students and survey fatigue. New directions for Institutional Research, 121, 63–73. Robertson, J., Walkom, E. J., & McGettigan, P. (2005). Response rates and representativeness: a lottery incentive improves physician survey return rates. Pharmacoepidemiology and Drug Safety, 14, 571–577. doi:10.1002/pds.1126. Schirmer, J. (2008). Impacts of land use change to farm forestry and plantation forestry: a survey of landholders: report prepared for the Socio-economic impacts of land use change in the Green Triangle and Central Victoria project. CRC for Forestry Technical R. Hobart: CRC for Forestry.
Ethical Issues in the Use of Multiple Survey Reminders
139
Sitzia, J., & Wood, N. (1998). Response rate in patient satisfaction research: an anlysis of 210 published studies. International Journal for Quality in Health Care, 10, 311–317. Snijkers, G., & Luppes, M. (2002). The role of a cognitive laboratory within Total/Tailored Survey Design and Total Quality Management. Case Study 4: Testing the annual establishment production survey. (In Snijkers, G. (Ed), Cognitive Laboratory Experiences: On Pre-testing Computerised Questionnaires and Data Quality. Doctoral thesis. (pp. 203–221). Utrecht: Utrecht University) Ward, J.E. (2003). Comment. Letters page. The Medical Journal of Australia, 178, 43 Warriner, K., Goyder, J., Gjertsen, H., Hohner, P., & McSpurren, K. (1996). Charities, No; Lotteries; No; Cash, Yes: Main effects and interactions in a Canadian incentives experiment. Public Opinion Quarterly, 60(4), 542–562. doi:10.1086/297772.
