Conceptualising and Measuring Democracy. Evaluating Alternative Indices. G.L.
Munck & J. Verkuilen. Foivos Skalidis. 29/04/2013. Comparative Politics.
Conceptualisation
Measurement
Aggregation
Conceptualising and Measuring Democracy Evaluating Alternative Indices G.L. Munck & J. Verkuilen Foivos Skalidis
29/04/2013
Comparative Politics
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Motivation
Quantitative researchers have overemphasised the problems of causal inference... ...while neglecting the quality of the data on democracy that they analyse.
Comparative Politics
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Motivation
Quantitative researchers have overemphasised the problems of causal inference... ...while neglecting the quality of the data on democracy that they analyse.
Comparative Politics
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Datasets on Democracy
Comparative Politics
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Datasets on Democracy
Comparative Politics
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Aim
Constructing a framework for the analysis of data, discussing three challenges: I
Conceptualisation
I
Measurement
I
Aggregation
Comparative Politics
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Conceptualisation 1. Identification of attributes I
I
Avoid maximalist definitions Decreases the usefulness of a concept Little analytical use Avoid minimalist definitions All cases automatically become instances Difficulty in discriminating among cases
2. Vertical organisation of attributes by level of abstraction I
Comparative Politics
Avoid redundancy and conflation of attributes Directly affects measurement and aggregation
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Conceptualisation 1. Identification of attributes I
I
Avoid maximalist definitions Decreases the usefulness of a concept Little analytical use Avoid minimalist definitions All cases automatically become instances Difficulty in discriminating among cases
2. Vertical organisation of attributes by level of abstraction I
Comparative Politics
Avoid redundancy and conflation of attributes Directly affects measurement and aggregation
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Conceptualisation 1. Identification of attributes I
I
Avoid maximalist definitions Decreases the usefulness of a concept Little analytical use Avoid minimalist definitions All cases automatically become instances Difficulty in discriminating among cases
2. Vertical organisation of attributes by level of abstraction I
Comparative Politics
Avoid redundancy and conflation of attributes Directly affects measurement and aggregation
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Conceptualisation
Figure: The logical structure of concepts
Comparative Politics
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Measurement 1. Selection of indicators I
I
I
Use of multiple indicators Single indicators entail the risk of bias Multiple indicators pose the burden of establishing cross-system equivalence Use of indicators that minimise measurement error Do unbiased historical data exist? Reliability The same data collection process should produce the same data
2. Selection of measurement level I
I
Comparative Politics
Maximise homogeneity within measurement classes with the minimum number of necessary distinctions Too fine-grained, too coarse-grained distinctions Reliability Multiple coders should produce the same codings Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Measurement 1. Selection of indicators I
I
I
Use of multiple indicators Single indicators entail the risk of bias Multiple indicators pose the burden of establishing cross-system equivalence Use of indicators that minimise measurement error Do unbiased historical data exist? Reliability The same data collection process should produce the same data
2. Selection of measurement level I
I
Comparative Politics
Maximise homogeneity within measurement classes with the minimum number of necessary distinctions Too fine-grained, too coarse-grained distinctions Reliability Multiple coders should produce the same codings Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Measurement 1. Selection of indicators I
I
I
Use of multiple indicators Single indicators entail the risk of bias Multiple indicators pose the burden of establishing cross-system equivalence Use of indicators that minimise measurement error Do unbiased historical data exist? Reliability The same data collection process should produce the same data
2. Selection of measurement level I
I
Comparative Politics
Maximise homogeneity within measurement classes with the minimum number of necessary distinctions Too fine-grained, too coarse-grained distinctions Reliability Multiple coders should produce the same codings Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Measurement
3. Recording and publicising of coding rules, coding process and disaggregate data I
Replicability
Comparative Politics
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Aggregation 1. Selection of level of aggregation I
Balance between parsimony and the concern of underlying dimensionality and differentiation High level of aggregation results in loss of validity Low level of aggregation impedes theorising and testing
2. Selection of aggregation rule I
I
Comparative Politics
Ensure the correspondence between the theory of the relationship between attributes and the selected rule of aggregation Attributes could be sufficient, necessary Robustness of aggregate data Changes in the aggregation rule should result in proportionate changes in the aggregate data Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Aggregation 1. Selection of level of aggregation I
Balance between parsimony and the concern of underlying dimensionality and differentiation High level of aggregation results in loss of validity Low level of aggregation impedes theorising and testing
2. Selection of aggregation rule I
I
Comparative Politics
Ensure the correspondence between the theory of the relationship between attributes and the selected rule of aggregation Attributes could be sufficient, necessary Robustness of aggregate data Changes in the aggregation rule should result in proportionate changes in the aggregate data Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Aggregation 1. Selection of level of aggregation I
Balance between parsimony and the concern of underlying dimensionality and differentiation High level of aggregation results in loss of validity Low level of aggregation impedes theorising and testing
2. Selection of aggregation rule I
I
Comparative Politics
Ensure the correspondence between the theory of the relationship between attributes and the selected rule of aggregation Attributes could be sufficient, necessary Robustness of aggregate data Changes in the aggregation rule should result in proportionate changes in the aggregate data Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Aggregation 1. Selection of level of aggregation I
Balance between parsimony and the concern of underlying dimensionality and differentiation High level of aggregation results in loss of validity Low level of aggregation impedes theorising and testing
2. Selection of aggregation rule I
I
Comparative Politics
Ensure the correspondence between the theory of the relationship between attributes and the selected rule of aggregation Attributes could be sufficient, necessary Robustness of aggregate data Changes in the aggregation rule should result in proportionate changes in the aggregate data Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Aggregation
3. Recording and publicising of aggregation rules and aggregate data I
Replicability
Comparative Politics
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Conclusions
No single index offers a satisfactory response to all three challenges Comparative Politics
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Conclusions
Interestingly, datasets have a high level of correlation. Why? I
Same sources, same precoded data
I
No sense of validity, only of their reliability
I
Correlation tests have been performed with highly aggregated data
Comparative Politics
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Conclusions
Finally... I
...mathematical statistics often presume that the relationship between theory, data and observations has been well established....
I
...this comprehensive framework for the generation and/or analysis of data, draws attention to issues underpinning causal inference.
Comparative Politics
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Conclusions
Finally... I
...mathematical statistics often presume that the relationship between theory, data and observations has been well established....
I
...this comprehensive framework for the generation and/or analysis of data, draws attention to issues underpinning causal inference.
Comparative Politics
Universit` a degli Studi di Milano
Conceptualisation
Measurement
Aggregation
Conclusions
Finally... I
...mathematical statistics often presume that the relationship between theory, data and observations has been well established....
I
...this comprehensive framework for the generation and/or analysis of data, draws attention to issues underpinning causal inference.