In 2001 the instruction on the form was 'List all members of your household who ... unexpected increases in the workload of Census district managers and their teams. .... drawn from all local authorities in England and Wales. The survey ..... groups are MYEs and health registers, the discussion above of the issues with these ...
Po p u l a t i o n Tre n d s 1 1 3
Autumn 2003
Key Issues in the Quality Assurance of the One Number Census
Ian Diamond Economic and Social Research Council Owen Abbott and Neil Jackson Designing Sources Division Office for National Statistics
INTRODUCTION
As part of the 2001 Census, the One Number Census project estimated and adjusted the Census database for underenumeration. As a result of the highly innovative One Number Census and the Quality Assurance process it encompassed, it was also ensured that robust results could be obtained for each local authority area. This article examines some of the issues and analyses that were undertaken as part of that assessment of the 2001 Census population counts for England and Wales.
The 2001 Census was the most innovative census of recent times in England and Wales. This article aims to present some of the key issues that arose in the assessment of the 2001 Census population counts. Integral to this assessment is the 2001 Census fieldwork, the publicity surrounding the operation, the One Number Census (ONC) methodology and the Quality Assurance of the results. The key questions that this article addresses are: • • • •
how did the census operation go; were users needs recognised and were they consulted; how rigorous was the Quality Assurance process; and; are the One Number Census results plausible?
The article firstly outlines the methodologies employed in the 2001 Census, and highlights the key issues that led to the successful implementation. As part of this, a brief overview of the One Number Census project is given, together with a description of the Quality Assurance strategy that underpins the population estimates arising from the 2001 Census. The article then explores the 2001 Census results through a series of demographic analyses to illustrate the sorts of issues investigated during the One Number Census Quality Assurance process itself. These analyses look at the patterns contained within the results, and comparisons with key alternative sources of population counts. Finally, a summary is provided.
The article firstly highlights the key issues surrounding the implementation of the 2001 Census fieldwork. The article then explores the 2001 Census results through a series of demographic analyses to illustrate the sorts of issues investigated during the One Number Census Quality Assurance process itself. These analyses look at the patterns contained within the results, and comparisons with key alternative sources of population counts. Overall, these in-depth analyses and investigations provide further credence to the plausibility of the One Number Census results.
THE 2001 CENSUS Conducting a census is a huge and complex task and the 2001 Census had to count a population in a society that was undergoing significant
11
National Statistics
Po pul ati on Tre n ds 1 1 3
Autumn 2003
change in societal and technological terms. In addition, the challenges which the 2001 Census had to deal with, amid a foot and mouth outbreak and the prospect of a General Election, made the field operation one of the most difficult of recent times.1 Following evaluation of the 1991 Census a number of new initiatives were introduced to maximise coverage. These included: •
•
• • • •
the use of just one population base in order to avoid the coverage issues that were experienced from the 1991 Census population definitions; encouraging people to post back their census forms to enable census fieldstaff to focus on those most likely to have difficulty filling in their forms; smaller workloads for staff in the more difficult areas; redesigned and carefully tested forms and questions; a community liaison programme including translation of census material into 26 languages; and; a focused programme of awareness raising and publicity.
This section outlines some of the key issues that underline the performance of the 2001 Census.
Population Bases It is a legal requirement for residents of England and Wales to be included on a Census return. In 1991 people who were away from home on Census night, visiting another address in the UK, would have fulfilled their legal requirement if they were included on a form where they were on Census night. This was so even though they would only have been recorded as a visitor at that address. There was no transfer base to put residents away from home on Census night back to their usual address. Thus it was necessary to implement absent household imputation in respect of residents away from home, otherwise they would only have been included in the 1991 resident count if they also completed a Census form on their return to their home address. In order to avoid a recurrence of this problem in 2001 it was decided to count people just once, at their usual address, people away from home on Census night were not included on the form at that address. Instead they were required to complete the form on their return to their resident address. ONS was able to make adjustments for people who avoided the Census through the ONC estimation process with the proviso that the methodology assumed each individual in the population had a non-zero probability of being included either in the Census or, where appropriate, in the CCS. In 2001 the instruction on the form was ‘List all members of your household who usually live at this address, including yourself’ which might have created difficulties for people who were genuinely unsure if they were required to complete a form. For instance, people such as foreign nationals who were here for a short stay of between 6 months and three years. Many such issues would have been likely to have been picked up by the CCS. However, there may well be individuals temporaily overseas or temporaily with no ‘usual residence’ in this country who would not have been counted.
Postback A key change in 2001 was the introduction of postback following tests that established the willingness and ability of the public to return forms by post.2 Postback as a collection methodology was a success. In all about 22,000,000 forms (88 per cent) were mailed back by the public. This was far in excess of the expected 70 per cent.1 The effect of this high return meant, in theory, far less effort was needed to collect non-returns. However, the large volumes of mail caused backlogs in
National Statistics
12
determining which households required following up and imposed unexpected increases in the workload of Census district managers and their teams. The fieldwork period was extended in some areas because of this. This backlog led to the public contacting the Census helplines in far greater numbers than was expected. This demonstrates the level of buy-in to the Census from the general public. Such a high postal response rate and volume of calls to the helpline indicated that the majority of the population wanted to, and did, complete their Census forms. However, the pressure on fieldwork teams caused by postal problems may have had some impact on subsequent call-backs and hence led in the end to a higher non-response rate.
Field operations The Census was carried out by 70,000 field staff compared with 105,000 in 1991. The fieldforce was structured to allow greater flexibility of field staff with less emphasis on a traditional Census management hierarchy, as recruitment issues were greater than in 1991. Senior field managers were actively encouraged to use experience and skills from their previous careers. This was particularly so in setting up networks or making contacts, dealing with the media and resolving a wide range of localised issues. Team working proved a key factor in overcoming many local enumeration difficulties. For example when postal problems emerged, some enumerators were used to help clear the backlog. In addition the 2001 Census was in the field during the foot and mouth crisis, which presented a unique and unexpected challenge. However, the teams in those areas seriously affected by the outbreak worked hard to overcome the difficulties imposed by the restrictions, for instance by arranging for forms to be posted to farmers or to arrange delivery in a safe area. In order to ensure that the outbreak did not affect the results, additional data were fed into the One Number Census Quality Assurance process to ensure that any impact of this factor could be examined. There is no evidence from the QA process to suggest that response rates in such areas were seriously affected due to the crisis. Moreover, there were few reports of complaints from members of the public and the farming community even in the badly hit areas. Qualitative information from the field operation was gathered in order to evaluate its success. This has been achieved by debriefing field staff, reviewing written reports from field staff, evaluating each project and the contracted services, and assessing the extensive range of internal data and information databases which have been built up over the many months of operations. Additional data have been collected from the series of nation-wide debriefings with senior Census managers, numerous reports and notes from all levels of field staff; plus information arising from the interaction between field staff and the regional management team at the Census Office. This information, which included data on many of the problems and issues discussed above, was fed into the ONC Quality Assurance process.
Publicity and Community Liaison The publicity campaign worked extremely well, as demonstrated by the unprecedented demand on the telephone helpline. Media coverage of the Census was, in the main, positive in the national press but particularly so locally, where Census Managers played a vital role in widely promoting the benefits and value of the Census. Even in Wales, where the coverage tended to be more acerbic, the Census was given extensive exposure. There could have been few parts of the country or sectors of the community unaware that a Census was taking place.3 On the helpline some two and half million calls were received from across the UK, with as many as a quarter of a million on one day alone – more than had been received throughout the 1991 Census. The majority of the calls concerned requests for Census forms due to the raised publicity prior to the delivery taking place. Levels of complaint about the content of the form itself or about the Census in general were
Po p u l a t i o n Tre n d s 1 1 3
proportionately few. This provides further evidence that the public engaged positively with the Census – due in no small part to the publicity campaign and the non-sensitive nature of the questions. The Community Liaison Programme must be regarded as one of the major successes of the 2001 Census in helping to target the traditionally hard-to-enumerate sections of the population. It provided a particularly useful function by facilitating the provision of the Census questions and instruction leaflet in 26 foreign languages, two of which were introduced at a late stage in the programme in response to requirements identified in the field. This helped enormously with the enumeration, as community leaders were encouraged to publicise and help the exercise.4
THE ONE NUMBER CENSUS Despite the innovative fieldwork procedures, it was recognised at an early stage that 100 per cent response would not be achieved. A pattern of falling Census response rates is evident in many other Census-taking countries. In addition the 1991 survey conducted to estimate those that were missed was not of a large enough scale to identify fully the extent and distribution of the under-enumeration. As a result, it was necessary to base the national population estimates for 1991 on demographic estimates. This process was only valid at a high level of aggregation and the Census counts had to be adjusted down to local authority level to make them add up to the national total based on the demographic estimate.5 In consultation, users were clear that for 2001 they wanted a fully adjusted set of counts covering 100 per cent of the population. The One Number Census project was set up to meet this need. By conducting a redesigned and much larger post-enumeration survey (the Census Coverage Survey (CCS)) and combining the results of both the Census and CCS in what is known as a dual system approach, the aim of the project was to estimate and adjust the Census database for under-enumeration so that all statistics add up to ‘One Number’.6 The estimation strategy is described in ONS (2001).7 It also aimed to ensure that robust results could be obtained for each local authority area. Central to the consultation was the acceptance that the Census counts would have a confidence interval associated with them. The ONC methodologies were researched and developed over a number of years by a joint team of statisticians from the ONS and the University of Southampton. The work was overseen by a Steering Committee that included experts from central government, statistical agencies overseas, other academic institutions and local government. In addition, census users were directly consulted at several stages in the methodological development process through Census user group meetings and special workshops. It is important to emphasise that this methodology has had the most peer reviews and user consultations of all National Statistics – an example are the sixty publicly available research articles representing over five years of work on the National Statistics website. A description of the risk management process for the project is found in Holt et al (2001).6 The One Number Census process involved a number of stages: • • •
•
a Census Coverage Survey was designed and conducted independently of the Census during May/June 2001; records from the CCS were matched to those from the 2001 Census; populations of the sample areas were estimated from the results of the matching using dual system estimation techniques which enabled an estimate of those persons missed by both the Census and the CCS to be made; populations for each local authority by age and sex were then estimated using a combination of standard regression and small area
•
•
Autumn 2003
estimation techniques; households and persons estimated to have been missed by the Census were then imputed to produce a fully adjusted Census database; and finally all population estimates were carefully quality assured using demographic analysis and comparison with aggregate level administrative data.
The key element of the One Number Census was the CCS. This was specifically designed to enable Census population counts to be adjusted for under-enumeration at the national, local and small area level. It consisted of a completely independent and intensive face to face survey of a sample of over 16,000 postcodes containing 320,000 households drawn from all local authorities in England and Wales. The survey was designed as 101 independent samples that were each surveyed using identical training and management procedures. This is an important feature, as the results are effectively 101 independent applications of the One Number Census methodology. The sample design also took into account the uneven distribution of under-enumeration across the country by stratifying by a ‘Hard to Count’ index based upon characteristics likely to be associated with under-enumeration, such as the number of multi-occupied addresses.8 Furthermore, the CCS was operationally independent from the Census enumeration exercise. The sample postcodes were kept confidential, CCS interviewers did not have any sight of the address lists produced in carrying out the Census, nor the Census forms returned in the area in which they were interviewing. The interviewers focused on making as many calls as necessary to achieve an interview, and the timing of these calls was varied to maximise the probability of making contact. The CCS in England and Wales achieved a response from 91 per cent of the households identified by interviewers. This is a very high response rate for such a large-scale voluntary survey when compared to other national surveys, and highlights that the principles and methodology underlying the survey were successful. The survey succeeded in meeting its objective of identifying households and persons that had been missed by the 2001 Census, as the One Number Census results show.9
QUALITY ASSURANCE All the ONC population estimates have been subject to rigorous quality assurance. The quality assurance process followed an agreed strategy that had been the subject of wide consultation with Census users and was agreed in December 2000 with representatives from the local authorities (see ONS (2001)7). The population of each local authority by age and sex was considered in a consistent and detailed manner – involving comparison against diagnostic ranges derived from rolled-forward population estimates and aggregated administrative sources (such as birth registration and pensions data). These diagnostics provided the best indicators of population that were available prior to the Census. It was never the intention for the Census to be adjusted to these diagnostics. Where there was a difference between the ONC estimates and the diagnostic ranges, extensive checks of the ONC results and diagnostic ranges were undertaken with respect to, for example, sample sizes and outliers and contingency action was taken if any issues were identified. The quality assurance process also included analysis for each local authority of a number of specific population subgroups known from 1991 to be prone to under-enumeration. These were full-time students, home armed forces, foreign armed forces and their dependants and prisoners. The estimates for these subgroups were compared with data from other official sources to determine whether the results were plausible. All ONC estimates were discussed and signed-off by an expert panel at a series of about sixty meetings over the course of several months. These
13
National Statistics
Po pul ati on Tre n ds 1 1 3
Autumn 2003
meetings adopted a rigorous and consistent approach throughout, with the results by age and sex for each of the 376 local authorities in England and Wales being considered in detail. The quality assurance process was also subject to external observation. The panel consisted of the head of Census statistical development, the ONC project manager, a member of ONS methodology group, the head of Population Estimates Unit, Ian Diamond, then of the University of Southampton and members of the ONC processing team. The Quality Assurance panel were also provided with detailed summaries of qualitative information from the field and processing operations, such as maps, fieldstaff debriefings, coding error rates and management information. These were examined in conjunction with the quantitative data in order to obtain a complete picture of the Census within an area. The panel frequently asked for additional information or analysis to be carried out on any or all of these data sources, and in some cases held back areas for consideration when other similar areas had been processed. Population Estimates Unit intelligence was also compiled for each area in order to provide information on population history, including migration flows. More specific information and data from the Quality Assurance process is available on the National Statistics website at www.statistics.gov.uk/census2001/onc_quality_info.asp.10
The overall response rate for England and Wales was 94 per cent, meaning that 6 per cent of the population was imputed from the CCS results rather than being counted by the Census. As can be seen from Figure 1, the undercount varied by age and sex. The group most poorly enumerated were males aged 20 to 24, with 15 per cent missed by the Census, while males aged 70 to 79 and females aged 60 to 79 were the best-enumerated groups, with only 2 per cent missed. The two peaks, for young children and those in their twenties and thirties, suggest that parents and children in the same households are being missed. Overall, the pattern is broadly as would be expected, with young children and those in their twenties and thirties most likely to be missed, the elderly least likely and males generally more likely to be missed than females.
Figure 2
Census underenumeration by local authority district Distribution of LAD level underenumeration
80 70 60
RESULTS Frequency
50
This section describes some of the analyses arising from the quality assurance process. The quality assurance was able to identify some of the surprises in the data, including a slower than expected rate of population growth and a change in the sex ratio for young adults. This section explains supporting evidence for these results, and provides an overview of the analyses and the issues they raise. Further details can be found at www.statistics.gov.uk/census2001.
40 30 20 10
For the Census results to be plausible, the pattern of underenumeration measured by the ONC must be realistic. We will therefore begin by examining the response rates at national and local level.
98% 100 %
96%
94%
92%
90%
88%
86%
84%
82%
80%
78%
76%
74%
72%
70%
Response rate
Figure 3 Figure 1
68%
64%
0 66%
Response rates
Census underenumeration by age-sex group
Census underenumeration by local authority for three age-sex groups Distribution of LAD level underenumeration
Underenumeration of Census by age-group 16.0% 14.0%
100
Males
90
Females
80 70
12.0%
50 8.0% 40 6.0%
30
4.0%
20 10
2.0%
Age group
National Statistics
14
60% 62% 64% 66% 68% 70% 72% 74% 76% 78% 80% 82% 84% 86% 88% 90% 92% 94% 96% 98% 100 %
0 0.0% 0 1–4 5–9 10– 14 15– 19 20– 24 25– 29 30– 34 35– 39 40– 44 45– 49 50– 54 55– 59 60– 64 65– 69 70– 74 75– 79 80– 8 85 a 4 n ove d r
ONC/Census
60 10.0%
Males 20–24
Males 35–39
Males 60–64
Po p u l a t i o n Tre n d s 1 1 3
The frequency distribution of overall response rates at local authority level is shown in Figure 2. It can be seen that the majority of local authorities have a response rate in the high nineties. This is again as would be expected – enumeration was good in most areas, less good in others with a few relatively poor. Those with a response rate below 90 per cent are almost exclusively London boroughs, and other low rates are generally found in urban areas. Figure 3 shows the same information for three specific male age groups, and illustrates how response rates vary across age groups and local authorities. Again the patterns are plausible. As might be expected from the national picture, the 60 to 64 age group generally has the highest response rates and the 20 to 24 age group the lowest. Even in the 20 to 24 age group, though, over half of local authorities have a response rate of over 90 per cent, and those below 80 per cent are mainly in London.
Figure 5
Differences between Census and pension numbers by local authority All persons 65 and over
100 90 80 70 Frequency
This lends credence at national level to the adjustments made to the raw Census counts.
Autumn 2003
60 50 40 30
Comparison with administrative sources
20
The most accurate and nationally consistent administrative sources are generally considered to be child benefit and pensions data, and for this reason they were used as comparator data in the ONC quality assurance process. The child benefit and pensions data used were cleaned by an expert at the University of Oxford to overcome the standard measurement error issues with raw counts of such data. Figures 4 and 5 show the distributions of differences between benefits data and Census counts for the relevant age groups. Green bars show a Census figure higher than the benefit count for the area, while those coloured red represent the reverse. As might be expected, the Census often counts more people than are registered for benefit, either because they do not claim the benefit or are not entitled to it, for instance children of US armed forces. The three areas with a particularly large excess of Census over child benefit figures are Forest Heath (which has a very high proportion of US armed forces), Kensington and Chelsea and the City of London. Occasionally there are more people registered for benefits than counted in the Census, the largest differences for child benefit being in Halton in Cheshire, Middlesbrough and Manchester. This is less likely to happen with pensioners than children, and indeed there is a smaller proportion of local authorities where pensioner numbers exceed the Census count. Figure 4
Differences between Census and child benefit numbers by local authority
All children 60
0 -5%
0%
5%
10%
Distribution of differences between Pensions and ONC across LADs
The largest excess of Census over pensions figures is in Kensington and Chelsea, while Wandsworth and Waltham Forest are the areas with the largest excess of pensioner numbers. This distribution in Figures 4 and 5 is broadly in agreement with that which would be expected for an accurate Census count. Benefits data such as those described in the previous section only relate to specific sections of the population, whereas the mid-year estimates (MYEs) are published each year for every age-sex group in every local authority. The MYEs were also used as comparator data in the quality assurance process. These estimates are produced by ONS’s population estimates unit (PEU) and are normally based on the most recent Census, adjusted for births, deaths and migration. Unusually in 1991, as mentioned above, the Census counts were differentially adjusted for undercoverage in a way that ensured the estimates summed to the national total based on the demographic estimate of the population in 1991 rather than the Census. This adjustment is explained in Heady et al (1994).5 Almost 100 per cent of births and deaths in the UK are registered and so estimates of natural change are of a high quality. However, since there is no compulsory address register the migration figures are necessarily based on survey and proxy data and so can only ever be estimates. International migration is estimated from the International Passenger Survey (IPS) and Home Office data in respect of people who are unlikely to be captured by the IPS on entry to the UK. Subnational migration is derived from patient register using data on re-registrations with a GP following a change of resident address (although prior to the 1999 estimates, changes in the numbers registered on electoral registers were used to help make internal migration estimates at local area level). The data sources and their limitations are explored in more detail in the article ‘Implications of 2001 Census for local authority district mid-year population estimates’.11
50
40 Frequency
10
30
20
10
0 -10%
-5%
0%
5%
10%
15%
20%
25%
Distribution of difference between ONC and Child Benefit
Even after the adjustment for undercount made by the ONC, the Census figures did not always coincide with the MYEs based on the 1981 Census. It is therefore essential to assess whether the evidence supports the contention that the Census results are closer to the ‘truth’ than the
15
National Statistics
Po pul ati on Tre n ds 1 1 3
Figure 6a
Autumn 2003
Underenumeration adjustment necessary for ONC figures to match mid-year estimates for males
Figure 6b
Underenumeration adjustment necessary for ONC figures to match mid-year estimates for females Underenumeration of Census – Females
Underenumeration of Census – Males 20
30
25
ONC/Census
ONC/Census
MYE/Census
MYE/Census 15
20
Percent Per cent
Per cent
10 15
10
5
5 0
-5
0 1–4
5–9 10– 14 15– 19 20– 24 25– 29 30– 34 35– 39 40– 44 45– 49 50– 54 55– 59 60– 64 65– 69 70– 74 75– 79 80– 8 85 a 4 n ove d r
0 1–4
-5
5–9 10– 14 15– 19 20– 24 25– 29 30– 34 35– 39 40– 44 45– 49 50– 54 55– 59 60– 64 65– 69 70– 74 75– 79 80– 84 85 a n ove d r
0
Age Age Range range
Age range
National Statistics
16
Figure 7a
Difference between adjusted patient registers and mid-year estimates for males aged 20–24
100 90 80 70 60 50 40 30 20 10
-10 to < 0 0 to < 10 10 t o< 20 20 t o< 30 30 t o< 40 40 t o< 50 50 t o< 60 60 t o< 70 70 t o< 80 -10
to < -20
-30
to < -30
-40
to < -40
-50
to
