Implementation of Responsive Collection Design for CATI Surveys at Statistics Canada François Laflamme Statistics Canada, Business Survey Methods Division,
[email protected]. Milana Karaganis Statistics Canada, Collection Planning and Management Division, milana.karaganis @statcan.gc.ca Abstract With the recent emphasis on the analysis of collection process data, paradata research has been focused on a better understanding of data collection processes leading to the identification of strategic improvement opportunities. For Computer-Assisted Telephone Interview (CATI) surveys, research findings have indicated that the same data collection approach does not work effectively throughout an entire data collection cycle, stressing the need to develop a more flexible and efficient data collection strategy. Over the last two years, the relationship between the quality, cost, productivity and responding potential of outstanding cases over the course of collection has been investigated. Additional tools have been developed to better assess and monitor progress, quality and performance during collection and to allow the development and implementation of Responsive Collection Design (RCD) strategy for CATI surveys at Statistics Canada. An RCD approach requires ongoing monitoring and analyses of collection progress against a predetermined set of quality, productivity, cost and propensity indicators. The RCD objectives are to identify critical data collection milestones and adjust collection strategies correspondingly to allow the most efficient use of available resources remaining. This paper provides an overview of the RCD strategy for CATI surveys at Statistics Canada and describes the implementation of the approach for the Households and the Environment Survey (HES), including new tools and indicators that were used to monitor collection and identify data collection milestones. Keywords: Paradata, collection strategy, active management, productivity, R-indicator, propensity model 1.
Introduction
Paradata research conducted over the past few years at Statistics Canada has indicated that the same collection strategy does not work effectively throughout an entire data collection cycle. As Mohl and Laflamme (2007) showed, a data collection strategy typically remains fairly static, i.e., a collection plan is developed prior to the collection start date specifying how collection effort (interviewer hours) will be applied. Once collection begins, collection plans are usually modified in response to cumulative use of collection resources (proportion of budget spent) and progress. Therefore, operational paradata research has stressed the need to develop a more flexible and efficient data collection strategy for CATI surveys, not only to maintain or reduce data collection costs but also to make better use of each call allowed under the cap on calls policy 1 . This approach implies an adaptive data collection or Responsive Collection Design (RCD) strategy. RCD was first discussed by Groves and Heeringa (2006) for Computer-Assisted Personal Interview (CAPI) surveys. Mohl and Laflamme (2007) expanded the application of RCD to CATI surveys, developed an RCD conceptual framework and proposed several RCD strategies in the Statistics Canada context. 1
The cap on calls policy limits the number of calls that can be made for each case. 1
At Statistics Canada, data collection for CATI social surveys is conducted from six call centres managed by Regional Offices (ROs). CATI survey data are collected with the help of the Blaise application. The management strategy for each survey can vary by site depending on the mix of surveys in collection, workload and availability of interviewers. Survey management uses the standard Management Information System (MIS) and customized active management reports that are based on Blaise Transaction History (BTH) files, Survey Operations Payroll System (SOPS) files for interviewers and sample design information available prior to data collection. BTH and payroll paradata are available in a timely manner, i.e., the day after data is collected or recorded. The framework proposed by Mohl and Laflamme (2007) includes two main components: active management (Hunter and Carbonneau (2005) and Laflamme et al. (2008a)) and adaptive collection. The main idea is to constantly assess the data collection process using the most recent paradata information available (active management), and adapt data collection strategies in order to make the most efficient use of available resources remaining (adaptive collection). In other words, RCD strategy aims to use information available prior and during collection (accumulated paradata) to identify when changes to collection approach are required in response to how well the collection progresses. The next section of the paper presents a more detailed overview of the RCD strategy proposed for CATI surveys at Statistics Canada. The third section begins with a brief description of the Households and the Environment Survey (HES) that was used as a pilot survey for the RCD strategy2 . The RCD strategy implementation for the HES is then presented, including an overview of how the strategy was monitored and assessed during collection and which practical issues were encountered. The focus of this paper is on describing the RCD approach and its key elements (communication strategy, initial planning stage, active management tools, etc.) as well as its pilot implementation. Note that, the results of the RCD pilot and its evaluation will be presented in another paper. 2.
Overview of the RCD strategy
The proposed RCD strategy breaks down the survey data collection process into four phases: planning, initial collection, RCD phase-in 1 and RCD phase-in 2 (as shown in Figure 1). The first phase (planning) occurs before data collection starts. During the planning phase, data collection activities and strategies are planned out, developed and tested for the other three data collection phases. The main activities include the analysis of previous data collection cycle(s) to identify improvement opportunities, frame and sample assessment and validation, the development of active management tools and MIS reports and the establishment of staffing plans. The second phase (initial collection) includes the first portion of the data collection process, from the collection start date up until it is determined that RCD phase-in 1 needs to be initiated. During the second phase, new features can be introduced into the collection process (for example, an intermediate cap on calls to avoid cases capping out before the last data collection phases, or new time slice 3 settings aiming at improving contact rates). During this initial collection phase, many key indicators of the quality, productivity, cost and responding potential of in-progress cases are closely monitored to identify when the next phase should be initiated. The third phase (RCD phase-in 1) categorizes and prioritizes inprogress cases using information available prior to the beginning of collection and paradata information accumulated during collection with the objective of improving overall response rates. In particular, a propensity model (logistic regression)4 is used to evaluate each unit’s likelihood of being interviewed and to categorize and prioritize each in-progress case. During this phase, key indicators continue to be monitored. In particular, the sample representativity indicator (R-indicator) 5 provides information on the variability of response rates between domains of interest to determine when the last 2
The Survey of Labour and Income Dynamics (SLID) will use about the same RCD strategy in January 2010. Time slices ensure that a specific number of calls is attempted at different times of the day, and on different days of the week. 4 Fore more information about response propensity model, see Tarabuchi et al. (2010) 5 The R-indicator concept was first discussed by Schouten, Cobben and Bethlehem (2009), 3
2
phase should begin. The last phase (RCD phase-in 2) aims to reduce the variance of response rates between the domains of interest (improving sample representativity) by targeting cases that belong to the domains with lower response rates. 3.
RCD strategy
The HES was selected to pilot the RCD strategy at Statistics Canada. The main objectives were to determine the feasibility of RCD for a CATI survey, to learn how practical considerations (technical limitations and communication channels) might affect the RCD strategy, to assess the impact and benefits of RCD and adjust the RCD strategy for implementation in other surveys. The proposed RCD strategy is relatively simple, cost effective and low-risk in order to avoid any negative effect on the quality of the data and resulting estimates. This section provides an overview of the HES and describes how the RCD strategy was implemented, how its impact was monitored and measured and how operational constraints were taken into consideration. 3.1
The HES
The HES is a dwelling-based cross-sectional survey that measures the environmental practices and behaviours of Canadian households. A sample of 20,000 units was selected from the 2009 Canadian Community Health Survey (CCHS) 6 respondents who were interviewed between January and June 2009. The sample design strategy used was essentially the same as in the last HES conducted in 2007. HES data collection took place in October and November 2009, and the target response rate was set at 75%. The HES was expected to produce estimates at the national, provincial and urban/rural levels. 3.2 RCD strategy for the HES Figure 1 presents a summary of the RCD strategy for the HES. Each RCD phase is described in greater detail below. Initial Collection Phase Collection Start
Responsive Collection Design Phase-in 1
Control Group
Responsive Collection Design Phase-in 2
Control Group
~ 10,000
Planning Phase
Miscellaneous Planning Including Sample validation
HES Sample
Miscellaneous
Responsive Design
Noncontact
Province/ Rural-Urban Priority
~ 10,000
High Probability
Special
Intermediate Cap on Calls (20)
Figure 1: RCD strategy for the HES
6
The Canadian Community Health Survey (CCHS) is a large cross-sectional household survey that uses both the CATI and CAPI collection modes. A selected household is rostered and one household member aged 12+ is selected and interviewed. 3
3.3
Phase 1 – Planning
The planning phase occurred before the start of data collection. First, the previous data collection cycle (HES 2007) was analyzed to identify opportunities for improvement. This analysis was also used to improve existing active management tools and develop new indicators and approaches to better monitor collection progress (the objective was to support survey managers in the timely adaptation of their collection strategies). The following list describes the main findings of this analysis and their application to the RCD strategy for the HES 2009. o In 2007, HES sample was also selected from CCHS respondents. The CCHS cases that were completed after relatively many attempts (more than an average number of attempts) were also more difficult to complete in the HES 2007 (they had lower response rates). To avoid collection bias, this information was not used in the initial collection phase. Instead, it was utilized in the propensity model to evaluate the probability of completion for all in-progress cases (RCD phase-in 1). o The HES 2007 sample was derived from CCHS units that responded at either the dwelling or person level. About 5% of the units included in the HES 2007 sample responded to the CCHS at the dwelling level only. These cases had a lower response rate than the remaining sample (about 12% lower). For the HES 2009 sample, this proportion rose to 10%, which could have had an impact on the HES 2009 response rate. This information was also used in the propensity model. o No strong correlation was observed between the time slices in which the same case was completed in the CCHS 2007 and HES 2007. Detailed analysis showed that only about half of the HES 2007 interviews took place in the same time slice as the CCHS 2007 interview (similar results were observed for other CATI surveys). Thus, there was no obvious reason to force the first call for the HES 2009 to be made in the same time slice when a CCHS 2009 interview was completed. However, this assumption could be verified through a controlled experiment in future RCD surveys. o Using HES 2007 data collection paradata, a new set of indicators was developed to better monitor survey progress, productivity and cost, as well as the response propensity and responding potential of outstanding cases, during collection. o For the RCD phase-in 1 collection phase, a response propensity model was developed to assign a probability of response (score between 0 and 1) to each outstanding case in the sample. o The HES 2009 sample was divided into two equal groups, the control group and responsive design group, to assess the impact of the RCD strategy. Two groups were randomly created according to sample design information (regional office, province, urban/rural) and CCHS 2009 sampling frame information (number of calls to complete interview and CCHS 2009 dwelling respondent only). Data collection for the control group was conducted using a typical CATI collection approach, while the new RCD strategy was applied to the RCD group. Note that the outstanding cases in the control group and responsive design group were merged for last responsive design phase. More findings were derived from this analysis and will be discussed throughout the paper in relevant sections. It is also important to note that RCD planning allowed enough time to develop, program and test the data collection application, pre-planned strategies and new features before the start of collection. 3.4
Phase 2 – Initial collection
Phase 2 initial collection proceeded as an usual CATI collection with the exception of two new features applied to the RCD group: an intermediate cap on calls and new time slice settings. Based on the analysis of the call patterns and their potential impact on the response rate, a cap on calls was introduced in 2006 for CATI surveys to limit the maximum number of calls that can be made on one case, to reduce respondent burden and to make better use of available resources. For HES, the cap on 4
calls was set at 25 attempts. In order to avoid cases capping out before the last two collection phases, an intermediate cap of 20 was introduced for HES. Once a case in the RCD group reached 20 attempts, the case went into an intermediate cap group for further analysis. Detailed information about these cases was provided to each RO to make a decision on the best use of the last 5 attempts for each case. A case assigned to the intermediate cap group remained in that group except if the case reached the cap on calls or was finalized. Time slices were implemented into the Blaise application a few years ago to ensure a better distribution of calls at different periods of the day throughout the week in order to increase the chances of contacting selected respondents. The approach used for the HES 2007 distributed the allowed 25 attempts according to the following standard time slice approach7 : morning (9:00–12:00) – 5 calls, afternoon (12:00–16:00) – 5 calls, early evening (16:00–19:00) – 5 calls, late evening (19:00–21:00) – 5 calls and weekend (Saturday and Sunday) – 5 calls. This approach was used for the HES 2009 control group. The RCD group followed new time slice settings that divided the allowed 25 attempts into 5 sequential groups of 5 attempts: day (9:00–16:00) – 2 calls, evening (16:00–21:00) – 2 calls, and weekend (Saturday–Sunday) – 1 call. Once 5 attempts following this pattern were completed, the cycle would start from the beginning. The objective of the new time slice strategy was to improve the distribution of attempts and collect more informative paradata. During the initial collection phase, key quality, productivity and cost indicators were closely monitored. In addition, the responding potential of in-progress cases was assessed throughout the collection period. These indicators were assessed separately for each of the six ROs to identify when to initiate RCD phase-in 1 since collection progressed at different pace in each RO. As shown in Figure 2, in the Sturgeon Falls office, the response rate (quality indicator) continued to increase at the same rate as costs (% of budget spent expressed as % of budgeted System Time and % of budgeted payroll hours) at the beginning of the survey. However, once productivity 8 started decreasing more sharply (around the 26th collection day), the gap between these indicators started growing (the same effort produced less interviews than at the beginning). In addition, the proportion of in-progress regular cases (in-progress cases that are not refusals, tracing or special outcomes cases) started to decrease while the average number of attempts made 9 continued to increase, also suggesting that effort was being spent on a smaller number of cases with less productivity. According to Figure 2, a critical data collection milestone was identified between the 26th and 32nd data collection days: it was time to initiate RCD phase-in 1 for that particular RO. Some Indicators to Identify Start of Responsive Collection Design Phase 1, Sturgeon Falls, HES 2009 Response Rate
Average Productivity over the last 5 days
Proportion of System Time budget spent
Proportion of SOPS (payroll hours) budget spent
Proportion of 'regular' cases
Average number of attempts made on 'regular' cases / cap on calls
Daily Productivity 100%
80%
60%
40%
20%
0% 1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
33
35
37
39
41
43
45
47
49
Data Collection Days
Figure 2: Key indicators 7
Only cases with a previous no-contact attempt are subject to the time slice rules at any point in time during collection. Productivity is defined as the ratio of Interview System Time (time devoted solely to the interview itself) to Total System Time which includes all successful and unsuccessful calls (Laflamme (2008b), (2009)). 9 Figure 2 shows the ratio of the average number of attempts made divided by the cap on calls (25). 8
5
It should be noted that “easier cases” (ones that required less calls and effort to complete) are more likely to be contacted and interviewed during the initial phase. Previous research has shown that 40% to 50% of the total number of CATI respondents is reached on the first contact attempt. An additional 4% to 7% of respondents are reached on the same day as the first contact attempt was made (Laflamme (2008b)). Reaching the remaining half of respondents requires a significant additional effort and different collection strategy to achieve higher collection efficiency. 3.5
Phase 3 – RCD phase-in 1
During RCD phase-in 1, CCHS 2009 paradata and HES paradata accumulated during collection were used to assign a response probability to each outstanding case in the sample (by applying the response propensity model). Cases with higher scores had higher chances of being completed during the remaining data collection period. The response propensity model used some CCHS paradata variables (type of CCHS respondents – person or dwelling level respondents, number of attempts needed to complete CCHS interview, household composition, etc.) as well as HES paradata variables accumulated since the beginning of the collection period (number of attempts/contacts/appointments by time slice, number of attempts with specific outcome codes – refusal or tracing, number of attempts after the first refusal or tracing outcome, etc.). The model had generalized R-squared of about 0.5 (an adjusted R-squared of about 0.65). The model was developed and validated during the planning stage to ensure that the model assigned higher completion probabilities to cases that ended up as complete in HES 2007 as opposed to cases that ended up as non-responses. At the beginning of each day during the RCD Phase-in 1, all in-progress cases were categorized and prioritized based on the response probability and the analysis of the sequence of calls. Cases were grouped in the following way: o The intermediate cap on calls group (created at the initial phase) continued to be used until the end of this phase. A case assigned to the intermediate cap group remained in that group unless it reached the cap on calls or was finalized. o The no-contact group consisted of all cases for which no contact with a respondent had ever been made since the beginning of the collection period (excluding cases in the intermediate cap on calls group). These cases were considered as either requiring an extra effort to make contact (hard-to-reach respondents) or as not having been given enough effort due to how the Blaise scheduler would bring up the cases. In either situation, separating these cases helped focus collection efforts. o The high probability group consisted of 20% of the in-progress cases with the highest probability of completion as assigned by the propensity model (excluding cases in the intermediate cap on calls and no-contact groups). These cases were considered to be the most likely to be completed. o The miscellaneous group consisted of all other outstanding cases that were not assigned to one of the first three groups. o A special group was only created at the end of RCD phase-in 1 after a significant amount of effort was spent on tracing cases, converting refusals or trying to resolve other special outstanding cases (those assigned during collection to special Blaise queues). This group consisted of about 10% of cases from these special queues that had the highest completion probabilities. The idea behind this group was to review these cases by RO and determine if any extra effort (and what kind of effort) could be applied to these units. A practical challenge faced by ROs required balancing a number of interviewers assigned to work on each of these groups. On the one hand, if too many interviewers were assigned to a group, there was the possibility that they would run out of cases. On the other hand, if not enough staff was assigned to a particular group, some cases would not have been tried. A special report was produced to advise ROs of the expected number of cases in each of the groups, which in turn could help determine the number of interviewers to assign to each group. These reports had to be reviewed daily to adjust staffing assignments for the day.
6
During RCD phase-in 1, which aimed at improving response rates, quality, productivity and cost indicators, along with the response propensity of the remaining cases were monitored to determine when a given RO should initiate the last RCD phase. In particular, the R-indicator defined as (1standard deviation of response rates), tracked the variance between response rates in domains of interests (see Figure 3). The decision to initiate the last phase was based on many factors, including the R-indicator (quality), productivity and average daily increase in response rates over the last 5 collection days (productivity), proportion of the budgeted system time and payroll hours claimed (cost) and other indicators of the responding potential of outstanding cases. R-Indicator, National and Some Reginal Offices, HES 2009 1.000 0.990 0.980 0.970 0.960 0.950
National Sherbrooke Sturgeon Falls Edmonton
0.940 0.930 0.920 0.910 0.900 1
3
5
7
9
11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53
Data Collection Day
Figure 3: R-indicator 3.6
Phase 4 – RCD Phase-in 2
The RCD Phase-in 2 aimed at improving sample representativity (thereby reducing the variance of response rates in the domains of interest). Sample representativity was monitored on a daily basis so that collection efforts could be switched between different domains depending on collection progress and response rates obtained. The R-indicator, defined as (1-standard deviation of response rates), provided a summary representativity indicator within each RO as well as at the national level. This approach could potentially result in “conflicting” objectives. For example, the R-indicator could be high (close to 1) in one particular RO while its overall response rate is lower than the national response rate. In other words, while no group priority would be required at the RO level, response rates would have to be increased in all domains to improve the national R-indicator. In practice, the expected and achieved response rates varied by RO for various reasons. This had to be taken into account to set realistic regional and national expectations. Finally, it should be noted that cases were still subject to the cap on calls policy during this last collection phase and this also had to be considered during implementation. 3.7
Active management
Decisions about collection strategies needed to be based on cost, productivity, effort, and data quality indicators in addition to response rates. Response rate is not the only measure that should be used to monitor and assess data collection. Instead, response rate should be used in conjunction with other indicators such as survey productivity, cost indicators and R-indicator to make the best use of data collection resources while taking into account the trade-off between quality and cost. The purpose of active management in the Responsive Collection Design context is to provide timely and relevant data on survey performance based on many indicators and customized information, so that problems with collection are identified and corrected in a timely fashion. In addition, active management tools can be used to determine data collection milestones. The challenge, however, is to produce relevant, customized and manageable reports that can be easily analyzed and used by survey managers at different points in time during collection. 7
For HES collection, active management reports that are used by all CATI surveys were enhanced to provide all necessary information. Extra reports were produced on a daily basis to show the Rindicator, costs, response rate and characteristics of the remaining sample. Analysing this information requires new analytical skills for survey managers that will need to be developed and maintained. At the same time, a daily/weekly collection monitoring routine had to be modified to allow sufficient time for reviewing and analysing reports. This experience demonstrated the need to factor in extra resource requirements as well as extra training needs to support and use the elaborate active management tools that are an integral part of RCD. 3.8
Communication strategy
The HES experience clearly highlighted the importance of an efficient communication plan. The communication strategy involved emails, conference calls and regular as well as ad-hoc meetings based on observed results. All parties involved in collection (Regional Offices, subject matter, collection manager) had to maintain on-going communication to identify collection issues and agree on any changes to collection strategies in a timely manner. In other words, a collaborative approach was used. Some technical issues also impacted HES collection and required ongoing communication to resolve them and assess their impact. Finally, ongoing communication was essential to monitor how well new collection strategies performed and whether any further adjustments were required. The amount of additional communication instigated by RCD was fairly significant. As expected, the tools developed for RCD allowed constant progress monitoring and identification of issues, which in turn required regular discussions to better understand what the numbers said. This experience demonstrated the need to factor in an elaborate communication strategy and the resource impact it might have. 4.
Conclusion
The RCD pilot for the HES was a very important first step in the development and implementation of a more flexible and efficient data collection strategy for CATI surveys at Statistics Canada. This experiment was also very useful for learning the impact of practical considerations on the RCD, improving active management tools, developing staff analytical skills and learning how to adjust the RCD strategy for other CATI surveys. In the long run, the expected benefits of the RCD approach are the following: o o o o o o
improved collection monitoring and management data collection effort targeting key areas throughout collection improved communication flow and collaborative approach between various partners, potential to maintain and/or increase response rates, potential to increase data quality by focusing on key domains, and potential to decrease collection costs by reducing collection effort to contact and interview respondents as well as to identify non-responding and out-of-scope cases.
The purpose of this paper was to describe the RCD strategy and its pilot implementation, including the following key elements: communication strategy, initial planning stage and active management tools. A second RCD pilot will be conducted in January 2010 with the Survey of Labour and Income Dynamics (SLID). The results of these RCD experiments and their evaluation will be presented in another paper.
8
5.
References
Groves, R.M. and Heeringa, S.G. (2006), “Responsive design for household surveys: Tools for actively controlling survey errors and costs”. Journal of the Royal Statistical Society Series A. Volume 169, Part 3. Hunter, L. and Carbonneau, J.-F. (2005), “An Active Management Approach to Survey Collection”. Proceedings from the 2005 Statistics Canada International Symposium on Methodological Issues. Laflamme, F., Maydan, M. and Miller, A. (2008a), “Using Paradata to Actively Manage Data Collection”. 2008 American Statistical Association, Proceedings of the Section on Survey Research Methods Laflamme, F., (2008b), “Data Collection Research using Paradata at Statistics Canada”. Proceedings from the 2008 Statistics Canada International Symposium on Methodological Issues. Laflamme, F., (2009), “Experiences in Assessing, Monitoring and Controlling Survey Productivity and Costs at Statistics Canada”. Proceedings from the 57th International Statistical Institute Conference. Mohl , C. and Laflamme, F., “Research and Responsive Design Options for Survey Data Collection at Statistics Canada”. 2007 American Statistical Association, Proceedings of the Section on Survey Research Methods. Schouten, B., Cobben, F. and Bethlehem, J. (2009), “Indicators for the representativeness of survey response”, Survey Methodology, 35, pp. 101-114. Tabuchi, T, Laflamme, F., Phillips, O., Karaganis, M. and Villeneuve, A. (2010), “Responsive Design for the Survey of Labour and Income Dynamics”. Proceedings from the 2010 Statistics Canada International Symposium on Methodological Issues.
9
