Implementing Clinical Prediction Models: Pushing the

PERSPECTIVES

COMMENTARY Implementing Clinical Prediction Models: Pushing the Needle Towards Precision Pharmacotherapy SR Dorajoo1,2 and A Chan1,3,4 Clinical prediction models promise a future of precision drug therapy. However, very few models are used in practice. Although models can eliminate unwanted clinical guesswork, several barriers hinder model implementation in practice. Here we discuss the less well-recognized barriers hindering model implementation for precision prescribing, highlighting areas that warrant attention, primarily relating to ethical and regulatory considerations that are requisite in paving the way towards a future of precision pharmacotherapy. The quest for precision dominates the current era of pharmacotherapy. An unprecedented abundance of clinically relevant data has hinted at the possibility of precision prescribing—the holy grail of pharmacotherapy. Modern statistical modeling methods offer sophisticated means of extracting insights from multifaceted data, rendering models that can add precision to the pharmacological armamentarium. The literature is saturated with studies proposing novel algorithms and nomograms for personalizing therapy in a bid to improve outcomes. Unfortunately, very few predictive tools enter routine clinical practice. Indeed, skepticism over the purported benefits of these tools have surfaced.1 As prediction models traverse the hype cycle of inflated expectations, a deliberation of their role in precision drug therapy is timely.

Response heterogeneity remains a reality of clinical practice. Even where highly efficacious drug therapy prevails, many patients do not respond. Prediction models may serve as a useful means of addressing response heterogeneity. The process of generating a prediction model typically involves integrating several pieces of information to generate a response prediction. The process begins with data acquisition, cleaning and processing before model derivation and validation may be performed. While these have been discussed in the literature at length, considerably less guidance is available on how such models may be brought to practice to actualize precision drug therapy.2 The critical aspect of model validation that remains unanswered is that of its adequacy in warranting model implementation. Specifically, to what extent does

retrospective validation provide evidence of model safety and efficacy? Could excellent retrospective validation performance eliminate the need to evaluate models in a randomized prospective trial? While necessary, retrospective validation alone is insufficient to guarantee improvements in clinical outcomes following implementation. Several other factors should be considered when determining the sufficiency of evidence warranting clinical implementation (Table 1). Our proposed list includes considerations relating to the specific clinical problem and factors related to the model itself. While randomized trials can reveal safety and efficacy, there may be genuine circumstances where bypassing a trial is ethically justifiable (Figure 1). Prospective trials are costly and slow to complete. Accelerated model implementation may be indicated when there is genuine urgency for alternative therapeutic approaches. Nonetheless, such decisions should be arrived at after a thorough evaluation of the evidence and clinical circumstances. Unfortunately, clear guidelines are lacking to facilitate ethical implementation of prediction models, considering the vast number of prediction models that have been constructed and validated but unimplemented in clinical practice. Productizing models in a suitable form for clinical use is a vital consideration in ensuring their accurate and sustained use. Traditionally, prediction models have manifested as pen-and-paper scoring tables and/or nomograms to facilitate pointof-care predictions.3 Arguably, these still remain preferred but entail disadvantages, including the need to categorize continuous predictors and to round off coefficients for quick and easy risk assessment. However, these procedures can blunt a model’s accuracy. With the availability of

1

Department of Pharmacy, National University of Singapore, Singapore; 2Department of Pharmacy, Khoo Teck Puat Hospital Singapore, Singapore; Department of Pharmacy, National Cancer Centre Singapore, Singapore, Singapore; 4Duke-NUS Graduate Medical School Singapore, Singapore. Correspondence: A Chan ([email protected]) 3

doi:10.1002/cpt.752 CLINICAL PHARMACOLOGY & THERAPEUTICS | VOLUME 00 NUMBER 00 | MONTH 2017

1

PERSPECTIVES Table 1 Factors determining the likelihood and the extent of evidence necessary before prediction models are implemented in clinical practice. Elaboration/considerations

Examples

Clinical factors Severity or urgency of unmet clinical need

Clinical implementation of models may be hastened if the degree of uncertainty in practice is widely accepted to be a cause of poor outcomes, necessitating newer and potentially better approaches to therapy.

Predicting survival to assess appropriateness of palliative chemotherapy in patients with metastatic cancers with dismal prognosis.

Presence of “safety net” in practice

Sufficient evidence to define acceptable safety limits within which model-based predictions may be used, to negate risks of prediction errors.

The ability to monitor serum vancomycin concentrations and adjust doses may facilitate clinical adoption of model developed to improve dosing precision as long as maximum doses fall within guideline recommended limits.

Diversity of clinical interventions tied to predictions

The availability of a variety of effective therapeutic agents with the potential to cater to the needs of various patient subtypes will justify the adoption of models for patient profiling and stratification.

Using appropriate anti-diabetic therapy in patients with type 2 diabetes of varied pathophysiology.

Resource intensiveness of intervention

When treatment or prevention become prohibitively expensive to administer to all patients, defining patient subtypes corresponding to their expected response will be necessary for channelling resources to those who stand to benefit most, sparing the cost and harm to those unlikely to benefit.

Initiating a resource intensive home-visit programme aimed at reducing morbidity and readmission risk.4

Delayed onset of clinical outcome

It may not be practical to carry out a randomized controlled trial to assess a model that predicts outcomes that occur after several years, considering that model relevance may tend to dissipate over time due to advancements in clinical practice.

Models predicting 5-year survival or 10-year cardiovascular event risk would require an extended trial period if prospectively validated via randomized controlled trials.

Intended clinical function

Is the model intended to direct or merely justify predetermined clinical decisions with objective data?

Predictions from pharmacokinetic models recommending specific dosage regimens may be followed very closely whereas models predicting the appropriateness of a given drug may more likely assume a smaller role in swaying clinical decisions.

Evidence of validity

Has the model been recently validated in the local population?

Vancomycin dose prediction models are believed to be poorly generalizable outside of populations used to develop the model. If a pre-existing model is to be used in a given clinical setting, evidence of successful local validity will be necessary.10

Predictive accuracy on external validation

Has the model demonstrated prediction accuracy on validation in independent patient cohort(s)?

Extensive external validation of the CHA2DS2-VASc risk score has led it to become routinely used in practice to quantify stroke risk in patients with atrial fibrillation, determining the need for prophylactic anti-coagulation.

End user trust

Do the identified predictors and the assigned weightages resonate with clinical experience and judgement? Do predictors satisfy Hill’s criteria of causation?

Uninterpretable ‘black box’ models developed with machine learning methods using clinically irrelevant variables are likely to suffer from end user scepticism and distrust, despite superior prediction accuracy.

Ease of use

Is the model in a suited form to provide predictions at the point of care? Is the predictor information routinely available?

Mobile applications and online calculators may be useful for models used occasionally (e.g. survival in metastatic lung cancer) while frequent usage should warrant integration with the electronic health records system to allow automatically generate predictions (e.g. readmission risk among discharging patients)

Actionability of predictions

Does the model provide insights that are clinically actionable for improving clinical outcomes?

Using a model to predict the risk of adverse reactions following antiepileptic drug use is meaningful only if provisions for frequent follow up and measurement of drug levels are available.

Model-based factors

2

VOLUME 00 NUMBER 00 | MONTH 2017 | www.cpt-journal.com

PERSPECTIVES

Figure 1 Advantages and disadvantages of the two possible routes to bring prediction models into clinical practice.

computational resources at the point-ofcare, these limitations may be avoided. Models can now be implemented as mobile/web applications or even be integrated with electronic health records for passively generating predictions.4 Computationally productized models may also facilitate risk interpretation through model-based visualizations, instantaneously generated and displayed at the point-ofcare. Such individualized risk profiles may facilitate risk communication, creating the opportunity for shared decision-making using objective evidence (e.g., http://bit.ly/ Stage4PrognosticScore). Given the myriad of possibilities, model developers should work closely with clinicians to develop tools that suit enduser needs and preferences. Following productization, rigorous, prospective trial-based evidence demonstrating safety and efficacy of prediction models would serve as a strong justification for clinical implementation. Nonetheless, several well-established models that are routinely used to assist in drug therapy decisions,

including statins for cardiovascular disease prevention (QRISK2), bisphosphonates for fracture prevention (FRAX), and anticoagulation for stroke prevention in atrial fibrillation (CHADSVASc), have not been prospectively validated via randomized trials. Still, these models are implemented in clinical practice to aid and not replace clinical decision-making. Before implementation, practitioners will need to be trained on the following: appropriate model utilization, interpreting predictions to guide decisions, precise point in the care delivery chain where the model will feature, model exclusions and the rationale behind the exclusions, and required documentation and monitoring as part of postimplementation surveillance. Technical support provision may be required if models are implemented as mobile/web-based services. Once implemented and used, it may become apparent that the model is differentially effective among patients receiving model-guided therapy. Some patients may

CLINICAL PHARMACOLOGY & THERAPEUTICS | VOLUME 00 NUMBER 00 | MONTH 2017

be incorrectly treated/dosed. It is important to reiterate that the model is unable, and not expected, to completely eliminate therapy imprecision. Rather, it is hoped to impart an appreciable improvement to clinical decision-making and only through it, outcomes. Preemptive efforts to manage enduser expectations will help allay disappointment and disillusionment, if expectations were inflated on initial implementation. Postimplementation surveillance findings of model-guided practice should be disseminated to key stakeholders as soon as a predetermined number of patients have been treated. A pre-/postimplementation assessment should serve to illustrate the effect of the model on the outcomes of interest in most instances, provided external time-dependent changes are unlikely to influence the impact assessment of the model. If indeed there has been a worsening of outcomes attributable to modelbased decisions, then model-based practices should be immediately terminated. 3

PERSPECTIVES Conversely, if the model demonstrates promise, it is prudent to continue surveillance efforts. With accumulating data, the exclusion criteria may be further refined so that model-based decisions are offered to patients who appear to benefit most. Continued passive monitoring of a handful of indicators over time is also recommended. A periodic review of these indicators will reveal the presence of drifts in efficacy and safety that may occur over time. This should warrant some form of model updating, from simple recalibration to more drastic measures such as revising input parameters and their coefficients to ensure that the model matches the requirements of current patients.5,6 Institutional leadership, procedural infrastructure, and regulatory oversight are necessary components in ensuring that prediction models are used in a safe and efficacious manner in the drive towards improving precision and patient outcomes. As to how pharmacy and therapeutics committees currently deliberate formulary listings and govern the use of agents in an institution, a similar organizational setup involving key stakeholders may be required to oversee and guide appropriate clinical use of prediction models. Such committees can carefully consider the circumstances warranting the restrictions and use of models deemed setting-appropriate. Perhaps the most challenging aspect of delivering sustained precision to practice through prediction models is to ensure that the model remains perpetually relevant. Changes to the practice landscape are constant and occur increasingly rapidly. In 2010, the doubling time of medical knowledge was estimated to be 3.5 years; by 2020, it is anticipated to drop precipitously to a mere 73 days.7 Many of these findings will be practice changing. Estimating the frequency of updates necessary for each model is challenging, but it is almost certain that all models will have a half-life of clinical relevance. Most prediction models are generated through batch modeling procedures, essentially constructing models using fixed quantities of historical data. Batch modeling procedures generate static models that only serve as a representation of past data and

4

require repeated manual updating over time. One solution may involve the use of stream mining (alternatively referred to as incremental learning) algorithms. Stream mining algorithms constantly adapt to changing patterns in the outcome variable of interest with constant exposure to data streams to generate dynamic models, capable of periodically updating model parameters following exposure to newer data. This is done without needing to access older data used to construct the original model. In anticipation of the many impending practice altering advancements, many outcomes would exhibit nonstationary distributions over time. Stream mining methods have already been applied in a variety of settings, including weather forecasting and robotics.8 Although attractive, the application of stream mining to clinical data has not been widely investigated and such methodologies have yet to penetrate mainstream statistical software. Experimentation with these self-learning technologies that border on the realm of artificial intelligence will confront us with many more challenges in the near future. Models are implemented to ease clinical decision-making but can paradoxically impart added complexity through ethical dilemmas. The use, misuse, or lack of use of prediction models can adversely influence malpractice risk and liability exposure.9 When model-based recommendations are followed and result in harm, what degree of responsibility befalls the clinician? Relatedly, while tight regulations are instituted to restrict the use of medicinal products at specified doses, prediction models may encourage liberalized dosing, based on individual patient circumstances. There may be instances where model-based doses conflict with approved doses. Clearly, there is a need for guidance within an overarching ethical framework to govern the appropriate use of prediction models in line with the approved regulations relating to the specific drug. These guidelines will serve to protect both patients and providers in the continued quest for precision drug therapy. Prediction models hold real potential to impart precision to drug therapy. Much of the current literature has focused on methodologies for generating statistically valid

models, but it is necessary to aspire towards applying predictive insights to improve clinical outcomes. Prediction models must be pushed beyond the stages of validation and productization and be brought into clinical practice with greater commitment, in an earnest attempt to address clinical uncertainty. Prediction models are unlikely to completely eliminate clinical imprecision and improvements will arrive incrementally. With the growing availability of rich and relevant data alongside advancements in modeling methodologies for efficient pattern extraction, the future of precision pharmacotherapy will be limited only by how willingly we embrace and apply them in practice. CONFLICT OF INTEREST The authors declare no conflict of interest.

C 2017 ASCPT V

1.

Tannock, I.F. & Hickman, J.A. Limits to personalized cancer medicine. N. Engl. J. Med. 375, 1289–1294 (2016). 2. Moons, K.G. et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart 98, 691–698 (2012). 3. Dorajoo, S.R. et al. A scoring model for predicting survival following primary tumour resection in stage IV colorectal cancer patients with unresectable metastasis. Int. J. Colorectal Dis. 31, 235–245 (2016). 4. Dorajoo, S.R. et al. Identifying potentially avoidable readmissions: a medicationbased 15-day readmission risk stratification algorithm. Pharmacotherapy 37, 268–277 (2017). 5. Halabi, S. et al. Updated prognostic model for predicting overall survival in first-line chemotherapy for patients with metastatic castration-resistant prostate cancer. J. Clin. Oncol. 32, 671–677 (2014). 6. van Houwelingen, H.C. Validation, calibration, revision and combination of prognostic survival models. Stat. Med. 19, 3401–3415 (2000). 7. Densen, P. Challenges and opportunities facing medical education. Trans. Am. Clin. Climatol. Assoc. 122, 48–58 (2011). 8. Bifet, A. et al. MOA: Massive Online Analysis, a framework for stream classification and clustering. J. Machine Learn. Res. 11, 44–50 (2010). 9. Goodman, K.W. Ethical and legal issues in decision support. Clin. Decis. Support Syst. 126–139 (2007). 10. Murphy, J.E., Gillespie, D.E. & Bateman, C.V. Predictability of vancomycin trough concentrations using seven approaches for estimating pharmacokinetic parameters. Am. J. Health Syst. Pharm. 63, 2365–2370 (2006).

VOLUME 00 NUMBER 00 | MONTH 2017 | www.cpt-journal.com