An evaluation of census quality - IOS Press

121

Statistical Journal of the IAOS 28 (2012) 121–135 DOI 10.3233/SJI-2012-0752 IOS Press

An evaluation of census quality Bernard Baffoura,∗ and Paolo Valenteb,1 a

b

Southampton Statistical Sciences Research Institute, University of Southampton, Southampton, UK Statistical Division, United Nations Economic Commission for Europe, Palais des Nations, Geneva, Switzerland

Abstract. The primary purpose of a census is to provide accurate estimates of a country’s population. These underpin a myriad of key planning decisions in between consecutive censuses at both the local and national level. In addition, reliable census data is needed at an international level. For example, the UN requires member states to take at least a census every ten years and EU member states have a statutory requirement to provide comprehensive population and housing data at regular defined periods. As such censuses have a pivotal role to play within official statistics. There are, however, different modes in which a country can conduct a census. Furthermore, despite all efforts, it is inevitable that the census cannot be perfect. Therefore, there is the need to measure the quality of the data produced in a census. The measurement of quality within a census is not an easy undertaking, mainly because there is currently no standard method of quality assessment that applies to all census methodologies. Thus this paper aims to bridge this gap, by examining quality assessment with regards to population censuses. It defines census quality, and then discusses the assessment of quality for different types of censuses. Keywords: Census, population, quality

1. Introduction The aim of a census is to provide data to users, but it follows that this data is most useable if it is of reasonable quality. Hence, the assessment of quality in a census is important, primarily for fostering confidence in the information produced by the census. Further, the management of quality in a census is vital not only because of the expansive range of users of the census data, but also because the census, very often, serves as the benchmark for the national statistical system. Moreover, the census is often the best, if not the only source of information on small population groups and small areas. However, there is currently no streamlined definition, and assessment, of census quality that can be used by national statistical offices. Therefore, this paper looks to present a census quality assessment framework, and this assessment can be changed depending on the methodology used to produce census data. ∗ Corresponding author: Bernard Baffour, Southampton Statistical Sciences Research Institute (S3RI), University of Southampton, Southampton, SO17 1SJ, UK. E-mail: bb1304@soton. ac.uk. 1 The views expressed here do not represent the views of the United Nations.

Firstly, it is important to define what is meant by a census. There are different definitions, but in this paper, the United Nations definition will be used (see page 6–7 of [19]). Here, a census is defined as an operation which produces an official count of a country’s population, right down to the smallest level of geographical detail, at regular intervals. This definition is preferred since it is able to distinguish the census from other methods of data collection, by means of stating essential characteristics that differentiate a census from other of data collection strategies. These are individual enumeration, simultaneity, universality and defined periodicity. Hence a census obtains information on everyone, at the same time, from the whole country and the outcome is that this information is updated on a regular basis. The outline of the paper is as follows. In Section 2, a definition of quality and census quality is given, and the individual dimensions of quality are introduced. Section 3, sets out how census quality can be measured focusing on the different dimensions of quality. Since the objectives of census are mostly tailored to suit the specific country’s needs, the census has a fairly unique role hugely dependent on the demand for statistics. This

1874-7655/12/$27.50  2012 – IOS Press and the authors. All rights reserved

122

B. Baffour and P. Valente / An evaluation of census quality

has to be balanced with the cost, which can be very high. In the majority of countries, the census can be the most complex peacetime exercise that the country undertakes (page 5 of [21]). Section 4 therefore looks to describe how quality assessment can be undertaken for the most common censuses methodologies, taking note of specific issues that may arise for these methodologies. The final section consists of some concluding remarks. 2. What is quality? What is census quality? The overarching purpose of a census is to have information or data. In our framework we are using data and information interchangeably. However, data and information are technically not one and the same thing. Data refers to the facts, raw numbers and statistics, while information refers to the physical representation of the data [7]. The main task of a national statistical office is to transform the data collected into information about the economy, population and society so as to inform policy. It follows that information (data) is the end product and the main purpose for which a census is undertaken. Therefore, census quality assessment translates into an assessment of the information that is produced by the census. The definition of quality, on the other hand, is not too straightforward. Since quality refers to usefulness of the information produced, it could be assessed in terms of value for money, where value for money is a measure of whether or not maximal benefit has been obtained from the information, given the cost of production. In fact in the service delivery industry, value for money and quality are often equated (for instance [4, 6,13]). Clearly, value for money is an important facet of a national statistical system [12,14], nonetheless a definition of quality that is solely focused on value for money may be too narrow. The most widely used definition of quality focuses on the ‘fitness for use’ of the information, and this will be used as basis for explaining what is meant by census quality. According to page 7 of [7] the quality of data is related to its use. In other words data has no quality; it only has potential value when it is used for a purpose. Therefore, within a census context, quality is a multi-dimensional representation of the information, and characterized in terms of six elements, namely: relevance, accuracy, timeliness, accessibility, interpretability, and coherence. This definition of quality is the same one used by the United Nations (see pp. 17–

20 of [19]). These dimensions are similar to the ones used to define quality within the wider context of data management (see for example [15,18,24]). (a) The relevance of information reflects the degree to which the census data meets the needs of the population, users and stakeholders. As it is impossible to measure every phenomenon, this dimension looks at whether the country’s pertinent informational needs are satisfied by the census. The relevance dimension, hence, looks at achieving a balance between meeting the (sometimes conflicting) user requirements and satisfying the most important needs within the confines of constrained resources, i.e. there is ‘value for money’ attained. (b) The accuracy of census results is the degree to which the data describes the phenomenon of interest. It concerns the reliability and precision of the population estimates, and the accuracy is usually characterized in terms of errors. It is long held view that the precise measurement of any phenomena, although ideal, is unattainable due to the fact that no measuring instrument is perfect, in other words there is the distinct likelihood of measurement error. In regards to the census, the accuracy of information will depend on the explicit methods put in place to identify and control for any errors that occur throughout the census process, from inception to fruition. Simply put, to achieve the most accurate results in a census there needs to be an extensive effort put into the design, collection and processing procedures. This dimension of quality is linked to the degree of coherence and timeliness of information, and oftentimes, key decisions taken to improve accuracy will impact on the delivery timescales, and how coherent the outputs are to users. (c) The timeliness dimension refers to the time frame of the census process, and in particular to the interval between the census reference date and the time when the results become available. Evidently most censuses take a long time - even as the results are being disseminated, there is planning underway for the subsequent census. Clearly, the longer it takes for the census data to be released, the less of a ‘snapshot’ it is, and questions start to arise as regards to the data validity and relevance. Even so, it must be reiterated that time must be taken in the post-censal processes in order to ensure that the quality of


outputs are of the best standard, i.e. there is a trade-off with the accuracy dimension. (d) The accessibility of census data refers to the availability and ease with which the data are disseminated to the stakeholders, and public at large. Census information is normally disseminated through a mix of free and bespoke products and services. The strategy adopted including the costs of the services and products will influence the degree to which the data is accessible. Additionally included in this dimension is the disclosure control strategy in place to safeguard confidential information, without compromising the quality of the data. The census is conducted to meet the needs of various categories of users, such as central and local government, researchers, businesses, non- governmental organisations, and the citizens. Consequently, care is needed to ensure that albeit widely accessible, census data is also coherent, and easily interpretable so as to facilitate its effective and appropriate use. A key aspect of accessibility is on ownership of the data. (e) Interpretability refers to the degree to which the information is easy to understand, and any salient census results are easily found by the user; in essence this dimension of data quality focuses on how the information ‘makes sense’ to users. The interpretability dimension, therefore measures how the data input and data output relate, particularly whether the responses given answer the (census) questions being asked. A detailed pre-testing programme should ameliorate this aspect of interpretability. The other aspect concerns how available supplementary information is that would make the census output easier to understand (i.e. this is often referred to as metadata). This metadata should explain the underlying concepts, terms, definitions, and classifications. Also included are links to papers and publications that detail the methodology and assumptions behind the data collection and adjustment processes. (f) The coherence of census information reflects the degree to which the census data can be brought together with other existing statistical information. It also concerns the conceptual integrity of information, which can be assessed by comparing to existing information either through older censuses, or surveys or administrative data. A detailed coherence strategy will consequently

123

have a programme of certification and validation of the data so that it investigates and explains any deviances from the expected trends. These six elements, although suitably distinct, may not be independent, and actions taken to address one element of quality may have an effect on the others. Accordingly, any actions to improve a particular dimension need to be weighed against any possible detrimental effects this might have on the other dimensions. Realistically, an optimum balance may be achieved by careful appraisal of the effect of each of the six attributes.

3. Measuring census quality The evaluation of the quality of the census is a very important exercise for a number of reasons. From an organisational point of view, the evaluation should allow verifying whether the very large effort and investment of resources required by the census was worthwhile. Moreover, the evaluation exercise can help in identifying any aspects of the census organisation that could be improved. This could be particularly useful for planning future censuses, and improving their efficiency and effectiveness. Regarding to the use of the census results, the evaluation allows the provision of users with some (tangible) measures of the quality of the data, which will help them to better interpret the results. The census is a pivotal part of the official statistics produced by a country because it, typically, provides a benchmark for the population count at national and local levels, as done in the UK, for example [3]. The results of the census are also used for a wide range of statistical activities, and for several years (usually until another census has been conducted and new data is available). The census is commonly the main (if not the only available) source of information when it comes to the measurement of some specific social phenomenon, particularly with reference to small population groups, or data for small geographical areas. In addition, census data is important for facilitating international comparisons, for example in the monitoring and evaluation of the Millennium Development Goals. The evaluation of census quality is, therefore, important in terms of fostering confidence in the information produced and as such needs to be an integral part of the census. The best quality assessment undertakes a comprehensive evaluation of the various phases of

124


census operations. In the following paragraphs, some general issues will be discussed on the measurement of the different dimensions of quality, described in the previous section. 3.1. Assessment of relevance The programmes and outputs of a national statistical office must reflect the country’s most important and pertinent informational needs. The relevance requirements have to be managed to ensure that not only is the most pertinent information being collected, but the manner in which the information is being collected is also considered, in order to minimize the burden on the public when it comes to collecting the census data. This can be achieved through methods to assess the relevance of previous census content and to identify any new and potential informational gaps that may appropriately be filled through the current census. This assessment should look also at other alternative sources, especially with regards to surveys and administrative sources. The census loses its effectiveness if the public fail to engage with it and so due care should be taken to make sure that the most relevant phenomena are measured via the census so as to reduce public response burden. Seemingly the best way of assessing the relevance of information is through feedback from users [2,8, 18]. They can provide comments on the adequacy and completeness of the data used in their analyses. These consultative exercises, in addition to aiding in the planning, allow the census authorities to provide a forum at which to discuss, and be responsive to, the needs of the users and public. It can also serve to encourage a greater understanding of the census plans and activities. As such the assessment of relevance should include: (i) client and stakeholder feedback mechanisms to facilitate their active participation in the census process; (ii) user and public consultations during the planning process; (iii) consultation with professional bodies to determine the most relevant mode of measurement of different phenomena; (iv) consultation with other agencies collecting data to review and co-ordinate data collection; (v) monitoring the use of the data; (vi) market research on the efficiency of different response mechanisms.

3.2. Assessment of accuracy Generally, errors can be classified into three major categories: coverage errors, content errors and operational errors. The assessment of accuracy is the multifaceted evaluation that seeks to identify and quantify these errors. Coverage errors are those that affect the completeness of the census. It is obvious that the primary aim of a census is to count everyone, count them once, and count them at their correct address. A failure of the census to perform these leads to coverage error. In practical terms, the assessment of coverage errors seeks to quantify the proportion of omissions (undercount) and erroneous enumerations (overcount). The most common way of doing this is to perform a post-enumeration survey,which consists of an intensive re-enumeration of a sample of areas within the country. The results of the survey are then compared to the census results. This permits estimates to be made of completeness (in other words, coverage) of the census population, specifically identifying the proportion of people who have been missed or erroneously included in the census. Another way of assessing the completeness of the population count is through demographic analysis, whereby the population estimates are found by using current vital statistics on births and deaths and adjusting for in-migration and out-migration (i.e. net- migration). A comparison of the population estimates derived under demographic analysis to the census can provide a basis for judging the accuracy of the census information. However, the validity of this approach clearly depends on the quality of the demographic estimates, especially the data (and/or assumptions) used to estimate migration (since data on births and deaths are sufficiently reliable in most countries). It is normal that there will be some differences between both sets of population estimates; the key is to look for considerable differences. For example, in the UK there were divergences in the sex ratios for specific age-cohorts, and this highlighted missingness amongst men in their 20s [3]. Alternatively, the coverage of the census can be assessed by comparing the census to data from other ancillary sources, such administrative registers or existing longitudinal surveys. Depending on the availability of such data, the comparison can be done either at an individual or aggregate level. The second type of errors is content error, which includes the incorrect reporting or recording of the data, and also errors caused by the non-reporting or nonrecording of data. The errors may be caused by a variety


of factors – non-response, enumerator or interviewer effects, recall effects, mode of measurement effects etc. Many of these errors can be avoided by adequate care in the design and implementation of the methodology used to collect the census data. Furthermore, national statistical offices usually develop an edit and imputation strategy with the aim of resolving any inconsistencies and providing estimates of the missing data (at least for basic variables), within the data processing phase. The editing process irons out any inconsistencies while the imputation process fills in some or all the missing information. During the imputation suitable substitutes (referred to as “donors”) are found from respondents with similar attributes. Doing this has the advantage of producing a set of consistent, complete, results. The third type of error is operational error, which results from the day-to-day processes of the census. Operational error can be difficult to quantify, and more often will be less of an issue when compared to the coverage and content errors. Some types of operational error are data capture error, coding error, tabulation error and classification error. Evidently, the operational error is minimised if there is a rigorous review of all the processes that make up the census, right from the initial pre-census activities of consultation, planning, development and testing of the census methodology, to the actual census processes of enumeration and data processing. All of these errors – coverage, content and operational – are interlinked. Clearly, a poorly implemented census will be susceptible to operational error, and will fail to adequately count everyone correctly, leading to coverage error. In addition, the impact of high coverage error is to lead to a lack of suitable respondents (i.e. donors) with complete data that can be used for the edit and imputation algorithm and this will inadvertently introduce more content error. 3.3. Assessment of timeliness The timeliness of the release of census data is an issue of concern to many users and stakeholders. There is often a trade-off between accuracy, coherence and interpretability, but more so in terms of the relevance. It goes without saying that the timelier the release of the census data, the greater correspondence the information is to the phenomena it has been commissioned to measure. Hence the timeliness of the data can be assessed by (i) the realistic setting of targets relating to the release of census data, through consultation and feedback;

125

(ii) announcing in advance the release dates so as to manage user expectation; (iii) setting realistic targets for customized products and ad hoc services provided to users who require non-standard outputs or additional analyses; (iv) the dissemination schedule should be made clear to users in advance, and they should be made aware of the differences between the planned outputs; (v) a detailed time schedule for the planned release of specific census outputs at different periods (for instance, preliminary and final results, or regular updates of certain variables), with possible associated costs. 3.4. Assessment of accessibility The benefits of a census are increased exponentially if the results are made widely available. This will also lead to an increase in awareness of the usefulness of the census, and therefore contribute to improved participation in future censuses. There are a number of issues to be considered with regard to the evaluation of accessibility, and these are that: (i) there should be a clear dissemination strategy with a wide scope of free and priced products and services available; (ii) the products and services should be in different formats and designs to cater for all the crosssections of the population, e.g. the blind and partially sighted; (iii) for chargeable products and services, the prices should be kept at a minimum and affordable to the general public; (iv) consultancy services and specially commissioned products could be offered to users who want detailed analysis, against payment of the corresponding costs; (v) sets of microdata should be available to selected categories of users, with the appropriate measures taken to ensure data confidentiality; (vi) a fundamental aspect of data accessibility is the availability of a dedicated website; (vii) the provision of a dedicated customer service is also important to assist users to in finding the data they need and answer any queries, particularly in the period shortly after the release date; (viii) client feedback on the different products and services should be monitored so as to make improvements and inform future census dissemination.

126


3.5. Assessment of Interpretability The assessment of interpretability is closely entwined with coherence, and it can be observed that a meticulous coherence strategy will assist the interpretability of the census information. However, the primary concern of the assessment of the interpretability is the level of provision of additional information (metadata). This has a twofold benefit: firstly, it facilitates user understanding of the statistical data being disseminated, and secondly, it can demonstrate transparency of the census. The metadata required for the census is, therefore dependent on both the information and the context of its use. Concerning metadata, there is an initiative by international organisations to facilitate the standardised exchange of statistical information, known as the Statistical Data and Metadata Exchange (SDMX) [17]. In accordance with SDMX there are two main categories of metadata, namely reference and structural metadata. Reference metadata refers to the additional information that provides further description of the information, for instance supporting documentation with definitions, concepts, methodologies and any other useful background information. This includes information on the general administration, execution and dissemination of the census, for example operations, procedures and other systemic processes. The information on the quality assessment activities fall under reference metadata. Structural metadata, meanwhile, refers to the additional information that aids the look-up of relevant information, and so concerns things such as indices, categorisations and keys that users can consult when looking for specific census data (bearing in mind that there is often an incredible amount of output disseminated as part of the census). The evaluation of interpretability can be based on (i) comprehensive provision of concise and relevant metadata, including information on the methodology used in the data collection and processing, particularly if sampling has been employed; (ii) clear details of any new elements or changes that may have taken place since the previous census, and that may affect comparability between current and past census results; (iii) clear and complete explanation of new concepts, especially if new questions are included in the census; (iv) clear statement of any limitations to the released data;

(v) summaries of key findings with directions to where to locate relevant data, in consideration of the fact that the census contains a wealth of information, which may be daunting to users; (vi) availability of a glossary describing underlying concepts, methodology, definitions, variables and classifications; (vii) compliance with standard or international categorisations, with additional explanation where alternative categorisations have been used; (viii) details of the quality assessment should be made available to users. 3.6. Assessment of coherence There are two aspects to the assessment of the coherence dimension of the data quality namely internal coherence and external coherence. The data is internally coherent if the whole census results are consistent within themselves. In order to have internally coherent data, a number of verification and validation tests should be carried out prior to dissemination. Special care should be paid to the possible impact on internal coherence of the data imputation and editing processes, which have a direct impact on changing the data. For these processes, verification tests should be complete in the sense that they should cover all operations. For other processes that do not impact directly on the census data, such as disclosure control, sample verification should be sufficient, particularly if a continuous quality assessment programme has been implemented. Validation tests are evaluative exercises that analyze planned tabulations of the census data to ensure that there is consistency. This is important when there are perturbations to the final tabular counts due to disclosure control. Validation tests should be compare the totals, frequencies and distributions produced. Care should also be taken in the aggregation of small areas to make sure that relations between variables or sets of variables are kept. External checks using current available data will ensure that the data is externally coherent. Moreover, the census data (after editing and imputation) will have to be checked against prior censuses so as to identify any incongruities. Added checks of external coherence can be carried out based on comparison of the census results against other statistical information either from surveys or administrative sources, administered by the national statistical office or other external bodies. The assessment of the coherence could include


(i) validation and verification tests should be undertaken before the dissemination of the data; (ii) clear indication of definitions, concepts, frameworks and classifications to be used for the census in a consistent manner; (iii) concepts, definitions and classifications for which international standards exist, should be adhered to in order to foster international comparability; (iv) an adherence to standard classification of variables and tabular derivations, with explanatory text for any reclassification and retabulation; (v) the use of common question formats so as to make it easier to compare to other surveys and censuses – ideally the questions should keep the historical formulation to facilitate longitudinal comparison; (vi) any unusual trends or inconsistencies in the data should be detailed. 4. Census quality evaluation for different census methodologies The preceding section has covered the different dimensions of data quality and presented a general overview of how they can be assessed within a census application. However, the definition of what constitutes a census is broad, coming in a number of various forms and guises, and so the quality assessment described above can seem vague and abstract. Thus, the various census methodologies will be subdivided into distinct groups, and then for each group some issues pertaining to the assessment of the different dimensions of quality will be discussed. It must be mentioned that despite all efforts, the list of census methodologies considered here may not be exhaustive. There is the growing tendency for countries to employ more than one census methodology owing to the complexities surrounding the population subgroups being measured. In a recent survey of 40 European countries, the number of countries using more than one census method for population estimation has doubled in the period between the current and the previous census [23]. 4.1. Traditional census – quality assessment issues The traditional census is a field enumeration of all individuals in the population with everyone counted either at their usual address (de jure) or where they are found on the reference date specified for the census

127

to take place (de facto). In general, the census information is recorded on a questionnaire. The administering of this questionnaire is either through an enumerator where a census official goes to every household and completes the questionnaire, or through selfenumeration where the census questionnaire is delivered to the household and they are meant to complete it and send it back to the census office. The questionnaire can also either be a short form which collects basic information, or a long form where there are detailed questions on a wide range of topics. The short form is normally universally administered while the long form usually collects data on a sample of the population. The traditional census is hugely dependent on the availability of an adequate address frame, or comprehensive maps that, at least, delimit the geographical boundaries so that the enumerators can accomplish their tasks efficiently. The different dimensions of quality and how they come into play in a traditional census will now be considered. 4.1.1. Relevance The overarching rationale behind the assessment of relevance is to ensure that the questionnaire responses are an accurate reflection of the phenomena under investigation. As such in a traditional census a. it is important to make sure that user requirement is satisfied – this is the challenge of the census in keeping the balance between conflicting needs of different stakeholders and managing the available resources; b. a detailed question development exercise should be done pre-census in consultation with the users and stakeholders should identify the data that is required; c. a post-census consultation with users should confirm whether the data met their needs; d. supplementary data sources that may provide more relevant data could be investigated so as to reduce public response burden and increase efficiency. 4.1.2. Accuracy In determining the level of accuracy for traditional census, the following issues need to be considered: a. the assessment of coverage is usually conducted through a post-enumeration survey, and it is essential that this survey is undertaken by a different group of field staff than those who participate in the initial census field enumeration;

128


b. differential undercount should be assessed through a breakdown of the coverage achieved by geography and key demographic variables; c. the question non-response rates and imputation rates should be reported; d. for countries that use sampling to administer the long form questionnaire, the sampling error associated should be included in the operational error; e. the sampling design and frame should also be assessed since census incompleteness could be due to frame deficiencies; f. the assessment should also cover the management of outsourced activities, including adequate contingency provision against failure; g. the assessment should cover errors inherently introduced through the design, collection or processing. 4.1.3. Timeliness The timely release of census output is frequently difficult to achieve in a traditional census due to the size of the operation and so some issues that may arise in the quality assessment are a. the expedient release of the data should not be the over-riding principle, as this could lead to compromised accuracy; b. for data that apply to particular pressing emerging issues (e.g. migration at a specific period) the results could be released earlier, after sufficient data quality tests; c. other data sources that can accurately plug the gap between the release of the census results should be investigated so as not to create a vacuum between censuses; d. user needs which have to be managed since there is always a certain amount of time required to perform the post enumeration processes for the data disseminated to be of the utmost quality. 4.1.4. Accessibility The traditional census produces a wealth of data and so the accessibility can be facilitated by 1. devising a wide range of dissemination products with the help of users to ensure the general public can easily access the census results; 2. the availability of a dedicated website can be very beneficial;

3. there being a dedicated contact point for the census, particularly in the period shortly after the release date; 4. the provision of consultancy services and specially commissioned products by the national statistical office so that the revenue collected could offset the costs of the census. 4.1.5. Interpretability The interpretation of the information goes hand in hand with the data dissemination process. A good dissemination protocol can therefore improve the interpretability of information to census users. This can be achieved by ensuring that the census outputs can be in a format, such as in tables and graphs, which allow users to obtain the bulk of the information with the least effort. In addition, some metadata can be provided such as a. the methodology used in the data collection and processing, particularly if there has been sampling employed; b. any new concepts that have been used in the census, notably when a new question has been included; c. a glossary that gives brief descriptions of the underlying concepts, methodology, definitions and classifications; d. details of the quality assessment. 4.1.6. Coherence Information gathered under the traditional census is easy to bring together with other statistical information within a broad analytical framework. Nonetheless, a number of issues to ensure optimal coherence within a traditional census are a. for internal coherence the imputation and editing processes should use respondents gathered from the post-enumeration exercises; b. for external coherence international standard concepts, definitions and classifications should be adhered to; c. any deviations and changes should be clearly detailed. 4.2. Register-based census – quality assessment issues In a register-based census the population is measured using a number of administrative registers, which are combined together (usually at an individual level). The


development of a register-based population system is typically a long, protracted, process and takes a huge amount of effort and planning. There is often not a direct jump from a traditional census to a register-based one, but a gradual movement. Most countries who currently conduct a traditional census also use administrative source information, especially in the review and assurance of census results. Moreover, the address file, which contains the list of all the dwellings and buildings, that is the backbone of any field or postal enumeration exercise is an administrative register. This shows that the view that the traditional census is a polar opposite to the register-based census is not entirely true. The dwelling register serves as the basis for most registerbased censuses. Additionally, it is required that there is a common unique identification number for every individual to facilitate the linkage of data from different administrative sources. With these two vital building blocks, other information from demographic and civil registration records can be accessed for each individual via the common identifier. The reduction in costs, once the system for a registerbased census has been established, is without a doubt the biggest advantage of moving from a traditional census to a register-based one. In the same vein, there is a reduction in the public response burden, and the information found on registers is often the most significant as the data is collected under circumstances of public necessity. In addition, a register-based census fosters cooperation between different government agencies through data sharing. Another advantage is that information can be regularly updated. With the introduction of a register-based system some census statistics can be compiled on a much more regular basis and so users can have access to current up-to-date data. The major disadvantage is that register-based censuses rely on information currently collected for administrative purposes. Therefore, not all census variables may be available, and there could be some comparability issues, for instance when definitions used for the registers differ from those to be adopted for the census. Moreover, the ability of a traditional census to ask questions that are of emerging importance is lost, unless a new administrative source is assembled to measure the phenomenon of interest. Further, the restriction of the housing register to conventional dwellings fails to enumerate people in less conventional dwellings, such as the homeless, people of no fixed abode, migrants and the highly transient. Finally the role of national statistical office can be difficult to define, especially when it performs both statistical and administrative activities

129

as well as sharing information with other government agencies. The independence and impartiality of the national statistical office can be difficult to guarantee, unless there is some specific legislation, for instance in Finland [20]. It is also extremely important that the general public appreciates and understands the benefits of the registerbased census; without public support a register-based census, more so than a traditional census, will fail. This is achieved through open discussion and debate explaining the rationale and benefits of administrative registers. Administrative registers function best with active participation of the public in information the statistical office of any changes in their personal attributes, and in an expedient manner. The public need to be supportive of the statistical office using administrative data for statistical purposes. It often helps if there is some national register legislation which sets outs the role of the statistical office, so as to make their registration activities transparent. There are number of quality assessment issues that need to be borne in mind as they explicitly affect register-based censuses which will now be discussed. 4.2.1. Relevance In a register-based census, the relevance dimension is particularly important since only information on topics covered in current registers can be collected. Since most registers are rarely created from scratch there is frequently little discussion with users, although user consultation can help ensure that the proposed registers are relevant. User consultation is important in identifying other existing sources that could be useful in providing better information on the phenomena being measured. It also helps to undertake a question development strategy of the sort carried out under a traditional census will allow that the registers being used are appropriate in answering the census questions. Without the active involvement of users a register-based statistical system can lead to statistics that are not very useable for research. 4.2.2. Accuracy A properly maintained register can provide very accurate data, but registers in their very nature are susceptible to errors in coverage. Since the coverage of the register-based census is incomplete, there needs to be some coverage assessment, similar to that carried out in traditional censuses. In addition, if a single register is assembled from a series of different registers there could be varying degrees of accuracy due to differences

130


in administration, database management and so on. The coverage and quality of the registers used clearly plays a fundamental role in determining the accuracy of the census results. It is also important to assess the way in which data from disparate registers are integrated. In this context, the analysis of the matching rates (direct matching and statistical matching) can provide useful information on accuracy.

4.3. Census based on registers combined with sample data – quality assessment issues

4.2.5. Interpretability The nature of metadata is significantly different for register-based censuses, and often there is a huge amount of supplementary information needed to suitably interpret the data. Additional metadata information could relate to the register(s) used, the creation of units and variables, the different sources and how they are linked, and any changes to the administrative system.

With the wide availability of good quality and welldesigned annual surveys that are nationally representative, it becomes possible to obtain census information from a combination of different administrative registers and the results from sample surveys that have been weighted to the population totals. One advantage of this is that surveys are more regular than the census and therefore capture current changes in the characteristics of the population much better. Also, the requirement for detailed data on social, economic and housing statistics which may not be appropriate for collection in a full scale census has motivated the need for a continued programme of annual (or regular) household surveys. The Netherlands developed this approach for their 2001 census, known as the Virtual Census [16]. Here register data was integrated with results from existing household surveys. Key to this was the unique personal identifier which was used to combine data from different sources (surveys and registers) at an individual level so as to produce a single set of reliable and consistent results. Clearly, a country can only choose to implement this type of census under three conditions. Firstly, for this to work census information should be available from the different administrative and survey sources currently held by the country. Secondly, it should be possible to link the different information collected from the disparate sources at an individual record level. Finally, the feasibility of conducting such a census is largely contingent on the country having the correct legal mechanism in place (the Dutch Virtual Census worked so well because there was a change in the law granting the national statistical office access to administrative data). The issues surrounding the quality assurance of a census of this type are a merger of the issues from administrative registers and surveys; however, there are some specific points to note.

4.2.6. Coherence Internal coherence of register-based census data should be assured in the phase of data integration. Any inconsistencies or ambiguities that are found at this stage should be resolved by making sure that there is consistency at the micro-level. At an early stage, there should be a harmonization process aimed at bringing together the different registers under one conceptual and definitional framework. This will be important in ensuring that there are not assorted registers measuring the same thing. This harmonization can extend to other existing non-administrative sources to facilitate external coherence.

4.3.1. Relevance Compared to purely register-based censuses, this approach may provide more relevant data given the possibility to rely on data from sample surveys, in addition to data from registers. However, data from sample surveys (because of their very nature) will provide limited geographic and suitably detailed information, especially for small, difficult to measure populations. There should, therefore, be a programme review undertaken to assess the relevance of registers and the sample surveys. Sample surveys are normally only representative at the national (and perhaps regional level), but clearly not for small areas. This has a bearing on the

4.2.3. Timeliness In principle, it could be expected that register-based censuses produce results in a timelier manner compared to traditional censuses since the labour intensive and time consuming operations like field data collection and data entry are not required. In practice, however, there are a number of complex procedures needed to be implemented to link the various registers and produce the census data. Furthermore, the time needed to produce census variables may vary from one subject to the other due to different routines for updating the different registers. This could result in some census results being released timelier than others. 4.2.4. Accessibility Since administrative sources are designed predominantly for the use of government agencies, there are issues surrounding how a population register can be accessed by the general public. There are also issues with regards to confidentiality and disclosure control.


relevance which will have to be considered. Furthermore, the sampling design could have an impact on the results, and possible implications on relevance should be considered. 4.3.2. Accuracy and coherence Combining data from administrative sources and surveys has the advantage of promoting coherence between the different statistical data collection instruments of the national statistical office. In terms of accuracy, non-response measurement and adjustment is important. Sometimes, different non-response strategies may be used for different surveys. Also, the selection of respondents to take part in different surveys will have to be coordinated carefully so that it balances lowering the response burden with collecting detailed information, particularly for small populations. 4.3.3. Timeliness and accessibility The timely dissemination of the census data is hugely dependent on how well synchronized the data collection processes of the different surveys are. There is a lot of thought needed at the start to coordinate the surveys, and sometimes a redesign of some (if not all) the surveys might be necessary. This will ensure that the data collected are nationally representative and detailed enough to obtain accurate low level data. Once a mechanism is in place that produces tabular output by counting the available register information or weighting up the sample information the resulting census tables can be made accessible to the public, and maybe subject to disclosure control procedures. 4.3.4. Interpretability The process of integrating information from registers and sample surveys could be viewed with suspicion if the methodology is not very transparent. This may be countervailed by including clear and complete metadata on the characteristics of the different registers and surveys. By including the operational processes used to combine and integrate the data, users should be able to reproduce some of the results. Metadata should also specify which census data are produced from what so that users may be aware of any possible implications to their analyses. For example data derived from sample surveys may not meet the level of statistical significance or geographical detail required by some users. 4.4. Census based on registers combined with full enumeration – quality assessment issues The traditional census has been shown in most countries to start with a definitive administrative list, in

131

the form of an address register or buildings inventory, which is used as a basis to enumerate the population. With an increasing amount of information being collected as part of these address registers, it is becoming possible to make more use of this registry information. As such the census that combines register-based information with a complete enumeration aims to make use of the available population registers to obtain census information. However, it then complements this by undertaking an exhaustive enumeration of the population in the classical sense. There are two main objectives of this type of census. Firstly, combining register data with a traditional enumeration can lead to improved accuracy of the population counts. Secondly, it is possible to obtain census variables that are not available from the administrative sources, or cannot be easily collected through a register. This type of census is very similar to the census that combines administrative registers with sample surveys, but it is most feasible in countries where there are no nationally representative surveys. The obvious advantage of an intensive field enumeration of the population (instead of a survey) is that there is less need to worry about estimation, modelling, weighting and sampling errors. It is also possible to check the coverage of the population register for both overcoverage (where people appear more than once) and undercoverage (where people do not appear). Many of the considerations made for the previous methodology (registers combined with sample surveys) are valid here, but there are some additional issues that will be discussed. 4.4.1. Accuracy By expectation, the population counts in this approach should be more precise than in a traditional census or a fully register-based census. Unlike in the sample survey-based census where there are questions surrounding small geographical areas, this type of census permits maximum geographical detail. Moreover, the register data serves as a benchmark for the data collected through field enumeration, whilst the fieldwork can be used to cross-check registration records. 4.4.2. Timeliness The timeliness of the release of the census data can be affected. This is because a full enumeration of the population, albeit with a shortened questionnaire, will be a fairly huge and time-consuming task. It must also be remembered that the normal traditional census operations (testing, enumeration, processing) will have to be carried out. In addition, these data will have to be integrated and made consistent with data from the registers.

132


4.4.3. Interpretability Metadata should provide information about the different administrative sources used and on the field enumeration procedures, including the question testing. Also, a glossary of terms, definitions, and concepts should be prepared. Documentation should cover the data integration of the sources so as to aid users understand how the final data were produced. 4.4.4. Coherence Internal coherence is probably facilitated much better in this type of census since the field enumeration is used to cross-check the register information, and vice versa. Even though any edit and imputation strategy will be using donors that are expected to be closely linked to the reality since the field enumeration will hopefully yield more donors,this harmonization of field enumeration to registers is not a simple undertaking. 4.5. Rolling census – quality assessment issues To answer the increasing demand for data that is upto-date and regular, in addition to spreading to costs and burden for conducting a census over a longer period of time, a rolling (or rotating) census can be considered. This methodology was proposed by the American statistician Leslie Kish (see for example [9,10]). A rolling census represents an alternative approach to the traditional census model. Here a continuous survey covering the whole country is undertaken, over a period of time. The idea behind a rolling census is that instead of enumerating the entire population at a single point in time, the population is subdivided and a full enumeration of one section of the population takes place, and then the next, and so forth. The census is therefore said to be ‘rolling’ through the population. There are a number of ways the rolling census can be implemented. The basic tenet of the census still holds, so information should be available for every individual at a specified time. This usually achieved through developing a nationally representative survey designed in such a way that suitably detailed (and accurate) information at the lowest geography can be determined. The obvious advantage is that the census resources can be more efficiently targeted. Further, the census data can be frequently updated since in comparison with the traditional census which provides decennial (or quinquennial) benchmarks, the benchmarking can be done on an annual basis through the rolling census. Currently, only France uses the rolling census for population estimation [5]. In France, this approach

has been implemented since 2005 on a 5 yearly cycle. Data collection is different for large municipalities (with populations greater than 10,000 inhabitants) where sample surveys are conducted every year, and small municipalities (with populations less than 10,000 inhabitants) where an exhaustive data collection is conducted every five years on a rotation basis. Some issues that could arise in the assessment of quality, based on the French model, will now be discussed. 4.5.1. Relevance One of the advantages of the rolling census is that since data is collected continuously, with annual waves it is possible to modify the questionnaire at relatively short time frames in order to provide information on emerging topics. This does contribute to improve the relevance of the census, though a price might have to be paid in terms of comparability of the census results over time. Also, as the rolling census allows the producing of annual updates this approach might be better suited for the measurement of phenomena that evolve relatively quickly in the society. Such phenomena could be difficult to monitor through a traditional census. 4.5.2. Accuracy There are a number of issues surrounding the accuracy of the rolling census. Firstly, in large municipalities the census relies on sample data. Thus, the coverage assessment should cover the sampling errors associated with the design of the census. Secondly, although care is taken to ensure that the dwellings register used as a sampling frame is kept as up-to-date as possible, some properties will unavoidably be missed, so there needs to be a regular assessment of frame coverage. In small municipalities, where an exhaustive data collection is conducted, the same issues discussed with regard to the traditional census should apply. 4.5.3. Timeliness One of the main advantages of this approach is that data collection and processing is distributed over time. This, in combination with the adoption of sampling for large municipalities, should allow for producing results in a timelier manner compared to other approaches. 4.5.4. Interpretability The concept of the rolling census is complicated, even to statisticians. Therefore, it is particularly important that clear and complete documentation is provided at differing levels of detail, for experienced users and the general public. The documentation should explain how the population figures are derived for the different municipalities and the country.


4.5.5. Coherence Since currently there is only one country implementing the rolling census, external coherence with other countries can be difficult. The production of population figures that count everyone, and at the same time, is not easy under a rolling census because of the way different municipalities are sampled. In addition, there are issues surrounding how the rolling census satisfies the simultaneity characteristic of the census. Further, changing definitions over time might be problematic. 4.6. Census based on full enumeration with annual updates – quality assessment issues The census which combines an exhaustive field enumeration of the population with yearly updates is a variant of the traditional census design. Here regular (i.e. annual) household surveys are used to obtain detailed characteristics of the population on frequent basis. However, for the dual purpose of benchmarking and coverage assessment, basic demographic characteristics are collected less frequently (e.g. every decade). Currently the USA is the only country implementing this type of census. Previously, in the 2000 census, the US Census Bureau collected basic demographic information on the whole population and more detailed information on a selected sample of the population (roughly one-sixth). However, in their 2010 census every household was sent a 10-question questionnaire, and a sample of households was chosen to participate in a monthly survey known as the American Community Survey. The idea is that eventually over the census (10 year) cycle one-sixth of the population is sampled. The American Community Survey shares a lot of similarities with French rolling census; in fact it was when Leslie Kish was at the US Census Bureau that rotating sampling was further developed with the aim of it being used to replace the decennial census so as to provide more frequent data [11]. The benefits of such a census are manifold if implemented properly. Primarily, the motivation for this approach is the timely, frequent availability of data that is pivotal in decision-making at all levels of government. Secondly, there is improved budgeting which can lead to huge cost savings. Although there is the need for a comprehensive planning, development and testing in the beginning, there are savings realised in later years. Thirdly, the comparatively smaller nature of the continuous survey facilitates better operational planning and it can be easier to target resources leading to greater efficiency. Fourthly, it spreads the response burden

133

over a longer period. This is particularly important in the face of declining census participation – with some sub-groups experiencing greater declines. As such, to improve universal coverage means that the census has to be able to modify its data collection for different populations. Finally, the census is more equipped to recruit and train its staff over a longer term, which has the added benefit of improving professional competency (see Chapter 2 of [22]). The quality assessment issues that affect a census of this kind are fairly similar to those of the rolling census. However, since once in a decade there is some benchmarked information collected on every individual the issues of individual enumeration and simultaneity that affect the rolling census may be avoided. 4.6.1. Relevance As for other methods that are based on sampling, the recourse to a sample survey (albeit a very large one) to provide information on the detailed characteristics of the population can affect the relevance of the results. Further, small area estimation may not always be possible. On the other hand, the annual surveys allow the possibility to include new topics in the census, as well as follow the evolution of phenomena through the continuous survey. 4.6.2. Accuracy An issue that could potentially affect the accuracy is the survey response rate. Owing to the size of the survey (250,000 households are surveyed monthly in the US [1]), there is the risk that the long term response rate could fall. There needs to be some consideration given to what would happen to the accuracy of the population estimates if response rates fall below the projected levels. Similarly, the risk that households will be sampled more frequently may lead in future to apathy and reduced questionnaire response. Furthermore, the estimates of the population using the survey can be liable to significantly larger margins of error, and subsequently lead to lower precision of small area estimates. 4.6.3. Timeliness The distribution over time of data collection and processing, in addition to the adoption of sampling, allows this method to produce timelier results in comparison to most other census methods. Moreover, for detailed characteristics the sample survey results should represent an improvement over the traditional census in terms of timeliness and frequency of data.

134


4.6.4. Interpretability Besides the metadata associated with a traditional census, there needs to be information on the weighting scheme used to produce population estimates, including the sampling mechanism and statistical modelling assumptions. 4.6.5. Accessibility The main advantage of this methodology is that more continuous census information provided, however user access needs to properly managed, specifically in terms of disclosure control. 4.6.6. Coherence External coherence should be assessed with results from previous censuses. This allows the assessment of the impact the new methodology has on the census results. 5. Conclusion The census is the pre-eminent approach of collecting information about the size, distribution and characteristics of a country’s population. Such information is important for planning, resource allocation and decision making. The use of information derived by the census are plentiful, and in order to ensure that the census data are at their most useable an assessment of quality of the census should be included in the general census process. However, it is clear that the methodology adopted to conduct the census will have a definite influence on the assessment of the different dimensions of quality. It has been realised that no census is perfect, and regardless of the methodology adopted, the results will be susceptible to error due to wide ranging factors. These errors may affect in different ways the quality of the census data. For this reason, comprehensive evaluation programme should be planned as an integral part of the census, aimed at assessing the quality of the census results. In this way, users will be provided with some measures of quality of the census data, which will aid the interpretation of the census information. The findings of the evaluation could be used to adjust the census results, as is at present done in most countries, in order to provide the best estimates of the size and characteristics of the population. It must be mentioned that the methodological approaches considered have been broadly defined for the purposes of the paper so as to cover the majority of censuses being undertaken by countries. Nonetheless,

there are some countries that may conduct their population census according to variants or combinations of these methodologies, or may develop entirely new methodologies that have not been discussed here. But the general assessment of census quality, which looks at the six dimensions of relevance, accuracy, timeliness, accessibility, interpretability and coherence remain an important requirement regardless of the census methodology adopted. The assessment and measurement of quality within censuses is an important undertaking so as to maintain the integrity and utility of the information produced by the census data collection. Even though there are different types of censuses in existence, this paper has sought to provide some common quality assessment framework so as to permit comparability of data obtained from different censuses. Finally, the paper has focused on formulating a definition of quality and how it can be assessed within the context of a census. Nonetheless, assessment of quality involves a further step of evaluation and measurement, which can be developed to give users a tangible measure of the census data. Clearly, proper evaluation will involve incorporating measures of the six dimensions of quality defined within this paper. Acknowledgements Tom King made valuable and insightful comments. We would also like to thank the referees and editor for their helpful comments and suggestions. An earlier version of this paper was presented at the 11th Joint UNECE/Eurostat Meeting on Population and Housing Censuses. The work was carried out whilst Bernard Baffour was on internship at the United Nations office in Geneva with the financial support of the UK Economic and Social Research Council. References [1] [2] [3]

C.H. Alexander, Still rolling: Leslie Kish’s “rolling samples” and the American Community Survey, Quality Control and Applied Statistics 48(6) (2003), 621–624. D.P. Ballou, R.Y. Wang, H. Pazer and G.K. Tayi, Modelling information manufacturing systems to determine information product quality, Management Science 44 (1998), 462–484. J.J. Brown, I.D. Diamond, R.L. Chambers, L.J. Buckner and A.D. Teague, A methodological strategy for a one-number census in the UK, Journal of the Royal Statistical Society: Series A (Statistics in Society) 162(2) (1999), 247–267.

B. Baffour and P. Valente / An evaluation of census quality [4]

M. Corder, M. Horsburgh and M. Melrose, Qualilty monitoring, innovation and transformative learning, Journal of Further and Higher Education 23(1) (1999), 101–108. [5] J.M. Durr, The French new rolling census, Statistical Journal of the United Nations Economic Commission for Europe 22(1) (2005), 3–12. [6] J.M. Grimshaw and A. Hutchinson, Clinical practice guidelines – do they enhance value for money in health care? British Medical Bulletin 51(4) (1995), 927–940. [7] T.N. Herzog, F. Scheuren and W.E. Winkler, Data Quality and Record Linkage Techniques, Springer Verlag, 2007. [8] B.K. Kahn, D.M. Strong and R.Y. Wang, Information quality benchmarks: product and service performance, Communications of the ACM 45(4) (2002), 182–192. [9] L. Kish, Samples and censuses, International Statistical Review/Revue Internationale de Statistique 47(2) (1979), 99– 109. [10] L. Kish, Using cumulated rolling samples to integrate census and survey operations of the US Census Bureau. US Government Printing Office: Washington DC, 1981. [11] L. Kish, Periodic and rolling samples and censuses. Statistics and public policy, Chapter 7, pp. 122–139. Edited by Bruce D. Spencer. Oxford University Press, USA, 1997. [12] R. Laux, R. Alldritt and K. Dunnell, Evolution of the United Kingdom statistical system, Statistical Journal of the IAOS: Journal of the International Association for Official Statistics 24(1) (2007), 47–59. [13] S.D. Pearson and M.D. Rawlins, Quality, Innovation and Value for Money: NICE and the British Health Service, Journal of the American Medical Association 294(50) (2005), 157–172. [14] B. Pink, Official statistics – A recognisable and enduring national resource, Statistical Journal of the United Nations Economic Commission for Europe 21(3) (2004), 231–239.

[15] [16]

[17] [18] [19]

[20]

[21]

[22]

[23] [24]

135

L.L. Pipino, Y.W. Lee and R.Y. Wang, Data quality assessment, Communications of the ACM 45(4) (2002), 211–218. E. Schulte Nordholt, The Dutch virtual Census 2001: A new approach by combining different sources, Statistical Journal of the United Nations Economic Commission for Europe 22(1) (2005), 25–37. Statistical Data and Metadata Exchange (SMDX), Statistical Data and Metadata Exchange – Content-Oriented Guidelines (release date: February 2008), http://www.sdmx.org, 2008. D.M. Strong, Y.W. Lee and R.Y. Wang, Data quality in context, Communications of the ACM 40(5) (1997), 103–110. United Nations Economic Commission for Europe (UNECE) (2006). Conference of European Statisticians Recommendations for the 2010 Censuses of Population and Housing. United Nations Publications: New York and Geneva. United Nations Economic Commission for Europe (UNECE) (2007). Register-based Statistics in the Nordic Countries: Review of Best Practices with Focus on Population and Social Statistics. United Nations Publications: New York and Geneva. United Nations Statistics Division (UNSD) (2008). Principles and Recommendations for Population and Housing Censuses: Revision 2. Department of Economic and Social Affairs. United Nations Publications: New York. US Census Bureau (2009). Design and Methodology: American Community Survey. US Census Bureau, US Department of Commerce – Economics and Statistics Administration. US Government Printing Office: Washington DC. P. Valente, Census taking in Europe: how are populations counted in 2010? Population and Societies 467 (2010), 1–4. Y. Wand and R.Y. Wang, Anchoring data quality dimensions in ontological foundations, Communications of the ACM 39(11) (1996), 86–95.