Risk Prediction Models: A Framework for Assessment

Policy Statement Public Health Genomics 2012;15:98–105 DOI: 10.1159/000334436

Received: July 5, 2011 Accepted after revision: October 18, 2011 Published online: December 14, 2011

Risk Prediction Models: A Framework for Assessment T.H.S. Dent a C.F. Wright a B.C.M. Stephan b C. Brayne b A.C.J.W. Janssens c

a c

PHG Foundation, and b Institute of Public Health, University of Cambridge, Cambridge, UK; Department of Epidemiology, Erasmus University Medical Center, Rotterdam, The Netherlands

Key Words Assessment ⴢ Evaluation ⴢ Genetic ⴢ Risk prediction

Abstract Background: Medical risk prediction models estimate the likelihood of future health-related events. Many make use of information derived from analysis of the genome. Models predict health outcomes such as cardiovascular disease, stroke and cancer, and for some conditions several models exist. Although risk models can help decision-making in clinical medicine and public health, they can also be harmful, for example, by misdirecting clinical effort away from those who are most likely to benefit towards people with less need, thus exacerbating health inequalities. Discussion: Risk prediction models need careful assessment before implementation, but the current approach to their development, evaluation and implementation is inappropriate. As a result, some models are pressed into use before it is clear whether they are suitable, while in other cases there is confusion about which model to use. This paper proposes an approach to the appraisal of risk-scoring models, based on a conference of UK experts. Summary: By specifying what needs to be known before a model can be judged suitable for translation from research into practice, we can ensure that useful models are taken up promptly, that less well-proven ones undergo further evaluation and that resources are not wasted on ineffective ones. Copyright © 2011 S. Karger AG, Basel

© 2011 S. Karger AG, Basel 1662–4246/12/0152–0098$38.00/0 Fax +41 61 306 12 34 E-Mail [email protected] www.karger.com

Accessible online at: www.karger.com/phg

What Are Risk Prediction Models?

Nearly all the important threats to population health in industrialised countries, and increasingly elsewhere, have complex multifactorial aetiologies. The epidemiological knowledge that underpins understanding of the causes of these diseases also permits the estimation of individuals’ risks of developing them. Risk is often assessed using complex multifactorial models which incorporate and weight the relevant risk factors [1]. For example, risk scoring systems use information such as age, genomic information, personal and family history, lifestyle, physical examination findings, the results of psychometric testing, and molecular and genetic biomarkers to estimate the probability of conditions such as coronary heart disease, stroke, breast, and colorectal and prostate cancers occurring in an individual over a defined time period (table 1). Risk scoring models are inherently attractive. They can be used to target interventions which are costly, scarce or pose risks for the recipient and may mitigate

This article is derived in part from the conclusions of a meeting convened at the Wellcome Trust Genome Campus in Hinxton, UK, in March 2010. It was drafted jointly by T.D. and C.W., with input from the other authors, and T.D. is its guarantor. A full report on the meeting is available at www.phgfoundation.org.

Thomas H.S. Dent PHG Foundation 2 Worts Causeway Cambridge, CB1 8RN (UK) Tel. +44 122 374 0200, E-Mail tom.dent @ phgfoundation.org

Table 1. Examples of risk models

Disease specific Cardiovascular disease

Type 2 diabetes

Many competing risk models have been used clinically to predict an individual’s risk of having a cardiovascular event in the future. They include Framingham, QRISK1 and 2, ASSIGN, ETHRISK, SCORE and ProCam [2]. These are based on a combination of traditional risk factors such as age, sex, body mass index, cholesterol, blood pressure, smoking, and socioeconomic status. There are also many publications assessing the addition of novel risk factors to these models, such as C-reactive protein [3], APOE status, and the gene 9p21.3 [4, 5].

Several models exist for predicting an individual’s risk of developing type 2 diabetes, such as QDScore and PreDXTM [6]. These are based on conventional risk factors such as age, sex, smoking, body mass index, blood pressure, or on a combination of novel genetic or other biomarkers with or without the conventional risk factors [7]. To date, there has been limited clinical application of these models, in part because the link with clinical decision-making is poorly defined.

Breast cancer

Context specific Intensive care

Analyte specific Genomic risk models

Several risk prediction models have been developed to assess the likelihood of a woman developing familial breast cancer; examples include Gail, Claus, BRCAPro and BOADICEA [8, 9]. They are based on family history in combination with conventional (e.g. age, hormonal and reproductive factors) and/ or genetic risk factors. These models are commonly applied in specialist settings to women deemed to be at risk of inherited breast cancer and can be used to guide clinical decision-making and individual choice regarding prophylactic bilateral mastectomy. Other models predict recurrence risk [10]. Risk models have been developed for application in the data-rich environment of intensive care (e.g. APACHE I–III, Mortality Probability Model, and the ICNARC model) [11]. These are based on a set of clinical and environmental risk factors, such as age, diagnosis and reason for admission. They are primarily used for auditing clinical processes rather than individual risk prediction and patient care. A new type of risk prediction service has recently appeared based solely on an individual’s genetic sequence (usually using only common single nucleotide polymorphisms). Genomic risk profiling companies (such as 23andMe, deCODEme and Navigenics) offer risk prediction of multiple diseases [12]. The risk estimates in these models are based on combining data from separate genome-wide association studies, but to date there has been limited assessment of their clinical validity and value in improving health.

misinformed or inconsistent clinical decision-making. They can also be applied in population or opportunistic screening to predict future disease before its onset, in the early diagnosis of disease before the development of symptoms or in evaluating prognosis. Scoring systems are likely to grow in both importance and number over the coming years. New candidate risk factors, single nucleotide polymorphisms and biomarkers are being identified at an unprecedented rate, and more interventions are becoming available to reduce risk, via both primary and secondary prevention. There is increasing pressure for a shift in both medicine and public health from diagnosis and treatment to prediction and prevention of disease. There is also rising societal and professional interest in personalised or stratified medicine; this tailored approach is as relevant to prevention as it is to treatment and will increase interest in risk prediction models. Furthermore, the identification of more risk

factors, coupled with the concomitant development of new statistical techniques, such as longitudinal modelling and reclassification analysis, will mean that disease risk models will become more complex. The growing availability, variety, complexity, and potential value of risk prediction models have important implications for clinical medicine, public health and the wider community. Physicians, scientists, policy-makers, and consumers will need to assess the validity, utility and wider implications of approaches to risk prediction and to choose which models to use and in which setting. But at present they lack the means and skills to do so in a systematic manner that assesses their impact on the whole of patient care. There is also no coherent approach to their evaluation, to their translation into practice and to ensuring that they produce clinical or public health benefit. We also need to consider the cost-effectiveness of models’ use and their ethical, social and legal implications.

Assessing Risk Prediction Models

Public Health Genomics 2012;15:98–105

99

How Can Risk Prediction Models Be Evaluated?

Of the many uncertainties which need attention before a risk prediction model can be considered suitable for use, it is statistical issues which have received the most attention. Several statistical techniques exist for assessing the properties and behaviour of a risk model, including consideration of calibration (whether the predicted risks are correct), discrimination (whether high-risk subjects are distinguished from low-risk ones), reclassification (how a risk prediction model behaves close to an important threshold), and the proportion of variation explained [13]. These techniques have been inconsistently applied to existing risk scores; where model evaluation has taken place, results are often difficult for policy-makers to use in evidence-based decision-making: • Risk models may give different results on different metrics of performance listed above, but it is not clear how to respond to this discrepancy. Which metrics are more important in indicating suitability for general use, or in particular clinical and policy situations? • It is not clear how to weigh or interpret differences in performance metrics. For example, small improvements in a receiver operator characteristic curve (a means of displaying discrimination graphically) may be statistically significant, but does this imply a clinically important improvement in discrimination? • Risk models may give different results in different settings, for example, in clinical rather than populationbased samples. • We are aware of no agreed standards for evaluating clinical utility. For example, to be clinically useful, a risk prediction model needs to link an individual’s estimated absolute risk of disease to a threshold for action over a particular interval. How should epidemiological data be used to calculate and agree these risks and thresholds? • We can find few assessments of the cost-effectiveness of using models. Recently, reporting standards have been developed for genetic risk prediction studies, intended to address inconsistencies in the evaluation and reporting of risk models [14]. Our framework extends this rigour in evaluation and reporting through to the translation of all risk models into practice. However, these standards have a specific purpose and are not intended to help the wider, non-statistical audience approach and assess the utility of novel risk prediction models. Health technology assessment is the systematic analysis of the impact of a health technology in practice. It can 100


be applied to risk prediction models. A useful approach to the evaluation of test performance, originally developed for the assessment of genomic tests, is the ACCE framework; ACCE stands for the Analytical validity of the assay(s), the Clinical validity and Clinical utility of the test in a particular context, and any Ethical, legal and social issues that it raises [15]. By combining this approach with the phases of evaluation for a novel risk marker [16], we have adapted this framework to the evaluation of risk prediction models (table 2).

Can We Develop a Universal Set of Quality Standards for Risk Prediction?

This adapted ACCE framework assesses test performance, only one of the questions to be considered before a risk prediction framework is implemented. Users of risk prediction models need wider criteria by which to assess whether models emerging from research are ready for general implementation. We convened an expert workshop in March 2010 to discuss the evaluation of disease risk models. The participants concluded that the diversity of diseases and contexts for which risk models are being developed and applied makes it unlikely that a single quality standard would be of use in determining which to adopt. Nonetheless, non-statisticians, including physicians and policy-makers, need to assess the value of risk prediction in a particular situation. We therefore decided to develop a set of questions arranged in domains, covering what type of information needs to be known before a model is ready for implementation. Simply evaluating the performance of a model through statistical metrics is not enough; the context in which the model would be used and the wider issues around implementation must also be considered. Even if, hypothetically, a perfectly accurate risk prediction model existed, in practice its use could still be inappropriate, ineffective or unfeasible. We identified 3 domains that should guide the assessment of a risk prediction model: (1) the context in which it will be used • purpose, clinical or public health context, population and disease • availability of intervention(s) and thresholds for use • risks and costs associated with test and treatments (2) the performance of the model itself • quality and applicability of data upon which the model was built Dent /Wright /Stephan /Brayne /Janssens

Table 2. The ACCE framework adapted for risk prediction model evaluation

Component

Description

Evidence needed for a risk prediction model

Analytical validity

Ability of each individual assay to measure accurately and reliably the component of interest

Measurement of the accuracy and precision of each assay used to measure each of the individual components of the model using measures of sensitivity, specificity, positive and negative predictive value

Clinical validity

Ability of the model to adequately predict the future development of clinical disease

Robust epidemiological evidence of a biomarker-disease association for each biomarker in the model and clinical evaluation of the incremental or overall performance of the risk prediction model in the population of interest, using measures such as calibration, discrimination and effect size

Clinical utility

Likelihood that using the risk prediction model will lead to an improved health outcome

Balance of risks and benefits of modelling, which relates to the purpose of predicting risk, thresholds for clinical action, the availability and safety of interventions to reduce risk, and cost-effectiveness

ELSI

Ethical, legal and social implications of risk prediction

Consideration of safeguards and impediments, and the acceptability of risk prediction to society

• performance metrics • external validation (3) issues relating to implementation • service delivery, feasibility and acceptability • cost-effectiveness • unintended benefits and harms

How Would This Work in Practice?

In table 3, we propose a framework for evaluating risk prediction models. It is a series of questions covering the domains listed above. To illustrate the use of the framework, it is applied to 2 situations in which risk prediction modelling may be undertaken. In coronary heart disease, the context and implementation issues have mostly been addressed, with uncertainty now confined to selecting the most appropriate model for use. Conversely, despite numerous models developed to predict the risk of dementia, little attention has been paid to the clinical context in which a risk model would be used, the potential benefits and harms, and the wider issues around implementation. The framework enables its user to understand more fully the extent to which a risk prediction model is ready for mainstream application. It does not provide a readymade answer to that question, which depends on judgement and may vary from setting to setting.


What Important Issues Remain Unresolved?

Although developing a framework to assess risk prediction models will improve the handling of their transition from research to practice, important questions need further work: • What is meant by validity in statistically complex risk prediction models, and how should it be measured? Metrics of validity such as calibration, discrimination and reclassification are all of value, but it is unclear how to integrate their results. • How can the interaction between risk factors be adequately reflected in prediction models? Genetic, environmental and other risk factors interact in a complex way which needs to be incorporated in the model. • How should clinical validity and utility be assessed? Connections need to be made between the results of risk prediction models for individuals from populations in whom the model might be used, decisions that follow, interventions that may be offered, and clinical outcomes which result. The effectiveness of such a sequence of inter-dependent steps is hard to assess using a randomised controlled trial, but other forms of assessment may not yield reliable enough conclusions. The assessment of cost-effectiveness is even less straightforward. • Who will develop risk prediction models, and who will fund their development? The proliferation of inadequately assessed risk models in dementia illustrates a lack of clarity of purpose among researchers and the agencies which fund them. Public Health Genomics 2012;15:98–105

101

Table 3. Examples of assessment of risk prediction models Context

Coronary heart disease

Dementia

What is the purpose of the model?

To identify people at higher risk of coronary heart disease

To identify people at high risk of incident dementia

What disease(s) does it relate to?

Coronary heart disease

All-cause dementia or subtypes including Alzheimer’s dementia or vascular dementia

What are the potential benefits of the model’s use?

To target preventive treatment

To target prevention and treatment of dementia, when treatment and prevention strategies become available in the future

Are there defined, non-arbitrary thresholds for intervention?

Yes: The National Institute for Health and Clinical Excellence has recommended the use of statins for those above a 20% ten-year risk

Not at present: However, there are NICE recommendations for the use of medications to slow progression of disease, depending on dementia severity

Is an effective intervention available for those defined as high risk by the model?

Yes – statins

No

What are the overall risks and costs of the model?

Low: It seems unlikely that the application of the model and subsequent care would be hazardous or expensive

Unknown, but potentially high: The investigations of those at apparently high risk could be invasive and expensive

What are the risks and costs relating to any tests that may follow use of the model?

Low: likely only to be blood pressure measurement and blood tests

Potentially high: It is possible that accurate prediction will require costly serial in-depth clinical screening that may include cognitive, functional, medical, and lifestyle assessment in addition to information on genetic, blood, cerebral spinal fluid, and neuroimaging biomarkers

What are the risks and costs relating to any interventions?

Low: unless symptomatic coronary heart disease suspected

Low for interventions to reduce vascular risk factors associated with risk of dementia, but potentially high for interventions to prevent neurodegenerative pathologies

In what population and clinical or public health context is it to be used?

Middle-aged and older adults in general practice

Both clinical and population-based approaches are described in the specialist literature, age group of relevance depends on change potential at different ages

Model

Framingham

QRISK

Numerous published, none in clinical use

What was the quality of the data with which the model was built?

Adequate for its original purpose

High: a large general population database

Variable across models

What study design and sample size was used?

Successive samples of tens of thousands of residents of a town in Mass., USA

Cohort study of 23 million people in the UK


How representative was the sample (age, sex, clinical setting)?

Highly representative of Population predominantly the UK population at risk white and prosperous; relevance to modern UK limited, because of chronological, ethnic and social differences


How were participants in the development of the model selected?

Residence in Framingham, USA

Unselected UK primary care cohort

Inclusion criteria vary depending on study design, from highly selected specialist clinics to full population samples

Are the variables in the model easily assessable and available?

Yes

Yes

Variable across models, but usually no, e.g. cerebrospinal fluid

How accurately were the variables measured?

With acceptable accuracy

Variably

Variably, depending on study design

102


Dent /Wright /Stephan /Brayne /Janssens

Table 3 (continued) Model

Framingham

How were the risk categories defined?

They were not defined at the They were not defined at the cohort’s inception, but cohort’s inception, but emerged during follow-up emerged during follow-up

Using clinical criteria for mild cognitive impairment or arbitrarily

How was the health outcome defined?

Clearly; clinical manifestations of coronary heart disease

Clearly; clinical manifestations of coronary heart disease

Variably, depending on study: usually either all-cause dementia or Alzheimer’s disease

How complete were the data?

Good

Adequate

Variable

What was the follow-up time?

Decades

Up to 12 years

Variable: range from short (1 year) to long (20 years)

Was the model development scientifically rigorous?

Yes

Yes

Variable – none fulfilling all criteria

What metrics of the model’s performance are available?

Calibration, discrimination, proportion of variation explained, comparison with other risk scores

Calibration, discrimination, proportion of variation explained, reclassification vs. Framingham and ASSIGN

Variable metrics which could include one or more of the following: sensitivity, specificity, predictive values, area under the receiver operator curve

What do they show about the model’s performance?

Calibration and discrimination both good in initial studies

Calibration and discrimination both good, and better than for alternatives

Variable, though none can accurately distinguish between progressive and nonprogressive mild cognitive impairment

How does the model’s performance compare with that of other models or tests for the same disease?

It performs better than those It performs less well in it has been tested against contemporary European populations than alternatives

Statistical comparison has not yet been undertaken

Has the model been externally validated?

Yes

Yes

No model has been externally validated; criteria for mild cognitive impairment have been applied across different samples and settings; however, there are no agreed methods for operationalisation of mild cognitive impairment component criteria and mapping varies across studies, so cross-study comparison is difficult

What do validation studies show about the model’s performance?

They confirm its poor calibration in contemporary European populations

They confirm its good calibration and discrimination

For mild cognitive impairment criteria, prediction of dementia risk is better in clinical vs. population-based samples

How applicable is the validation to the population in question?

Highly applicable: there are several validation studies in British populations

Highly applicable: the validation population were from British primary care

Not applicable

Implementation

Primary care

Unknown (clinical or population screening)

How would the model be used in practice?

Applied by primary care clinicians

No model is currently recommended for screening in clinical or population based samples because of the lack of clear treatment and preventative options; it is claimed that knowledge of risk would help plan for the future (e.g. arrange finances and future care needs) or streamline individuals for further assessment, but the issue of misclassification is rarely taken into account

How would the model be integrated into clinical systems?

Readily, especially for practices with compatible IT systems

Some computerised testing sets are actively promoted to primary care

How would services need to be developed to ensure equity?

Primary care practitioners would need to ensure the model was used in all people of appropriate age

Not applicable

What is its feasibility and acceptability?

High: it is already widely used

Not known in the UK


QRISK

Numerous published, none in clinical use


103

Table 3 (continued) Implementation

Primary care

Unknown (clinical or population screening)

How much additional professional training will be needed?

Minimal

Considerable, given implications of a positive result

What would be the cost-effectiveness of using the model?

Probably highly cost-effective, because costs of using the model are low and the improvement in targeting of prevention is potentially very valuable

Perhaps highly cost-effective when preventative and treatment strategies are available, depending on prevention potential for dementia over the whole life span and intensity of testing required and misclassification levels (i.e. unnecessary intervention and possible high number needed to treat)

What are the unintended benefits and harms likely to be?

Benefits: prevention of other diseases by statin; harms: adverse effects of medications such as statins; medicalisation of asymptomatic people

Benefits: identification of reversible dementias where symptoms are the result of a treatable medical or psychiatric condition; harms: misclassification, overtreatment

What effect will implementation have on the management of other diseases?

It might reduce the risk of type 2 diabetes and stroke, but have a consequent effect on the incidence of late-onset diseases such as dementia

Treatment of other diseases that affect the diagnosis of dementia

Are there issues of particular sensitivity, such as end-of-life care?

No

Yes: issues of care when planning for a disease where symptoms worsen and result in dependence

Are there any specific ethical concerns, such as providing for informed consent?

No

Yes: insurance, consent to clinical trials, outcome following risk assessment as no treatment or prevention is currently available

For example, a recent systematic review found 25 risk prediction models for dementia, none of which had been validated externally, and few of which had undergone rigorous statistical testing, such as calibration or sensitivity analysis across factors such as age and education [17]. We believe that there should be more emphasis on the evaluation and improvement of existing models and less on the creation of new and inadequately evaluated ones. • Who will appraise risk prediction models, and who will manage their translation into practice? Risk prediction models are capable of substantial effects on health, both positive and adverse. Yet they are unregulated, patchily commercialised and the subject of insufficient authoritative guidance; it is therefore not surprising that their uptake is inappropriate. • What role can regulators play? At present, the access of risk prediction models to the market is not controlled, but they could be regulated in a similar way to pharmaceuticals, on the basis of their capacity to benefit or harm.

104


Conclusions

The complexity and diversity of risk prediction models and the clinical and public health issues that they address make a simple approach to their assessment impossible. Decisions about the translation of risk prediction models from research to practice should be made by considering the context in which the model will be used, how the model was built and tested, and issues relating to its implementation. Clinical utility can only be assured by demonstrating a connection to improved outcomes. The assessment framework which we propose ensures users of medical risk prediction models are aware of the questions that need to be considered as decisions about models’ fitness for use are made. It would avoid the premature introduction of inadequately evaluated risk prediction scores, such as those for dementia, and helps elucidate the differences between rival approaches when both are already in use, such as for coronary heart disease. Further work is needed to develop the framework and explore how best it can be used. Nonetheless, we believe the framework to be a constructive first step in understanding how we can use our growing understanding of the origins and interactions of disease risk to improve health, while avoiding harm and wasted resources. Dent /Wright /Stephan /Brayne /Janssens

Acknowledgements We acknowledge with thanks the contributions of the meeting’s participants: Ms Corinna Alberg, Project Manager, PHG Foundation, Cambridge; Professor Doug Altman, Director, Centre for Statistics in Medicine, University of Oxford; Professor Carol Brayne, Director, Institute of Public Health, University of Cambridge; Professor David Clayton, Professor of Biostatistics, Cambridge Institute for Medical Research, University of Cambridge; Dr Tom Dent, Programme Associate, PHG Foundation, Cambridge; Dr Emanuele Di Angelantonio, Senior Research Associate, Department of Public Health and Primary Care, University of Cambridge; Professor Doug Easton, Professor of Genetic Epidemiology, University of Cambridge; Dr Pei Gao, Epidemiologist, Department of Public Health and Primary Care, University of Cambridge; Dr David Harrison, Senior Statistician, Intensive Care National Audit and Research Centre, London; Professor Julia Hippisley-Cox, Professor of Clinical Epidemiology and General Practice, Nottingham University; Professor Steve Humphries, BHF Professor of Cardiovascular Genetics, University College London; Dr Cecile Janssens, Associate Professor, Department of Epidemiology, Erasmus University Medical Center, Rotterdam; Dr Stephen Kaptoge, Senior Statistician, Department of Public Health and Primary Care, University of Cambridge; Dr Mike

Knapton, Associate Medical Director, British Heart Foundation, London; Dr Mark Kroese, Consultant in Public Health Medicine, PHG Foundation, Cambridge; Professor Cathryn Lewis, Professor of Genetic Epidemiology and Statistics, King’s College London; Professor Jonathan Mant, Professor of Primary Care Research, University of Cambridge; Professor David Neal, Professor of Surgical Oncology, University of Cambridge; Dr Nora Pashayan, Cancer Reseach UK Training Fellow in cancer public health and epidemiology, Institute of Public Health, University of Cambridge; Dr Paul Pharoah, Cancer Research UK Senior Clinical Research Fellow, Strangeways Laboratory, Cambridge; Dr John Robson, Senior Lecturer in General Practice, Queen Mary University of London; Professor Kathy Rowan, Director, Intensive Care National Audit and Research Centre, London; Professor David Spiegelhalter, Winton Professor for the Public Understanding of Risk, University of Cambridge; Dr Blossom Stephan, FLARE Fellow, Institute of Public Health, University of Cambridge; Dr Richard Stevens, Senior Statistician, Department of Public Health and Primary Health Care, University of Oxford; Dr Angela Wood, Lecturer in Biostatistics, Department of Public Health and Primary Care, University of Cambridge; Dr Caroline Wright, then Head of Science, PHG Foundation, Cambridge; Dr Ron Zimmern, Chairman, PHG Foundation, Cambridge.

References 1 Royston P, Moons KG, Altman DG, Vergouwe Y: Prognosis and prognostic research: developing a prognostic model. BMJ 2009; 338:b604. 2 Dent TH: Predicting the risk of coronary heart disease I. The use of conventional risk markers. Atherosclerosis 2010; 213: 345– 351. 3 Dent TH: Predicting the risk of coronary heart disease. II: the role of novel molecular biomarkers and genetics in estimating risk, and the future of risk prediction. Atherosclerosis 2010;213:352–362. 4 Shah T, Casas JP, Cooper JA, Tzoulaki I, Sofat R, McCormack V, Smeeth L, Deanfield JE, Lowe GD, Rumley A, Fowkes FG, Humphries SE, Hingorani AD: Critical appraisal of CRP measurement for the prediction of coronary heart disease events: new data and systematic review of 31 prospective cohorts. Int J Epidemiol 2009;38:217–231. 5 Talmud PJ, Cooper JA, Palmen J, Lovering R, Drenos F, Hingorani AD, Humphries SE: Chromosome 9p21.3 coronary heart disease locus genotype and prospective risk of CHD in healthy middle-aged men. Clin Chem 2008;54:467–474. 6 Hippisley-Cox J, Coupland C, Robson J, Sheikh A, Brindle P: Predicting risk of type 2 diabetes in England and Wales: prospective derivation and validation of QDScore. BMJ 2009;338:b880.


7 Talmud PJ, Hingorani AD, Cooper JA, Marmot MG, Brunner EJ, Kumari M, Kivimäki M, Humphries SE: Utility of genetic and non-genetic risk factors in prediction of type 2 diabetes: Whitehall II prospective cohort study. BMJ 2010;340:b4838. 8 Gail MH, Brinton LA, Byar DP, Corle DK, Green SB, Schairer C, Mulvihill JJ: Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. J Natl Cancer Inst 1989;81:1879–1886. 9 Antoniou AC, Cunningham AP, Peto J, Evans DG, Lalloo F, Narod SA, Risch HA, Eyfjord JE, Hopper JL, Southey MC, Olsson H, Johannsson O, Borg A, Pasini B, Radice P, Manoukian S, Eccles DM, Tang N, Olah E, Anton-Culver H, Warner E, Lubinski J, Gronwald J, Gorski B, Tryggvadottir L, Syrjakoski K, Kallioniemi OP, Eerola H, Nevanlinna H, Pharoah PD, Easton DF: The BOADICEA model of genetic susceptibility to breast and ovarian cancers: updates and extensions. Br J Cancer 2008;98:1457–1466. 10 Mook S, Van’t Veer LJ, Rutgers EJ, PiccartGebhart MJ, Cardoso F: Individualization of therapy using Mammaprint: from development to the MINDACT Trial. Cancer Genomics Proteomics 2007;4:147–155. 11 Harrison DA, Rowan KM: Outcome prediction in critical care: the ICNARC model. Curr Opin Crit Care 2008;14:506–512.

12 Edelman E, Eng C: A practical guide to interpretation and clinical application of personal genomic screening. BMJ 2009;339:b4253. 13 Altman DG, Vergouwe Y, Royston P, Moons KG: Prognosis and prognostic research: validating a prognostic model. BMJ 2009; 338:b605. 14 Janssens AC, Ioannidis JPA, van Duijn CM, Little J, Khoury MJ; GRIPS Group: Strengthening the reporting of genetic risk prediction studies: the GRIPS statement. BMJ 2011; 342:d631. 15 Haddow JE, Palomaki GE: A model process for the evaluating data on emerging genetic tests; in Khoury MJ, Little J, Burke W (eds): Human Genome Epidemiology: Scope and Strategies. New York, Oxford University Press, 2004, pp 217–233. 16 Hlatky MA, Greenland P, Arnett DK, Ballantyne CM, Criqui MH, Elkind MS, Go AS, Harrell FE Jr, Hong Y, Howard BV, Howard VJ, Hsue PY, Kramer CM, McConnell JP, Normand SL, O’Donnell CJ, Smith SC Jr, Wilson PW; American Heart Association Expert Panel on Subclinical Atherosclerotic Diseases and Emerging Risk Factors and the Stroke Council: Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation 2009;119:2408–2416. 17 Stephan BC, Kurth T, Matthews FE, Brayne C, Dufouil C: Dementia risk prediction in the population: are screening models accurate? Nat Rev Neurol 2010;6:318–326.


105

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.