Understanding how nurse practitioners estimate patients ... - CiteSeerX

JAN

RESEARCH METHODOLOGY

Understanding how nurse practitioners estimate patients’ risk for coronary heart disease: a judgment analysis Jason W. Beckstead & Kelly D. Stamp Accepted for publication 22 June 2007

Correspondence to J.W. Beckstead: e-mail: [email protected] Jason W. Beckstead PhD Quantitative Methodologist/Associate Professor University of South Florida College of Nursing, Tampa, Florida, USA Kelly D. Stamp PhD RN Clinical Instructor University of South Florida College of Nursing, Tampa, Florida, USA

B E C K S T E A D J .W . & S T A M P K .D . ( 2 0 0 7 ) Understanding how nurse practitioners estimate patients’ risk for coronary heart disease: a judgment analysis. Journal of Advanced Nursing 60(4), 436–446 doi: 10.1111/j.1365-2648.2007.04406.x

Abstract Title. Understanding how nurse practitioners estimate patients’ risk for coronary heart disease: a judgment analysis Aim. This paper is a report of a study to examine how nurse practitioners combine information when estimating patient risk of coronary heart disease. Background. In the United States of America and other countries, nurse practitioners are increasingly working alongside physicians in primary healthcare settings. Given this role, nurse practitioners represent an important resource in early detection of numerous diseases. Understanding how nurse practitioners use patient characteristics (cues) to form estimates of patient risk for disease may improve general disease prevention efforts. Method. Social judgment theory and its lens model analysis are concerned with the correspondence between a person’s judgments and the environment. This approach was applied to examine how 15 nurse practitioners weighted eight risk factors for coronary heart disease, how accurate practitioners were in assessing patient risk for coronary heart disease, and how much self-insight practitioners had into their own risk estimation processes. The data were collected in 2006. Results. Nurse practitioners showed moderate to high accuracy and evinced a variety of cue-weighting strategies. Insight into their own judgment policies was modest. The lens model analysis revealed that most practitioners had lower values on knowledge of the ecology than they did on cognitive control. Conclusion. Educational efforts aimed at improving detection of patients at risk for diseases might do better to target increasing clinicians’ understanding of cue-criteria relationships, than to stress themes of consistency in evaluating patients. Keywords: clinical decision-making, clinical judgment, coronary heart disease, empirical research report, judgment analysis, lens model, nurse practitioners, nursing

Introduction Understanding how clinicians assess patient risk for disease is important for optimizing professional training and practice, and ultimately for ensuring that patients receive the highest 436

quality care possible in the most expedient and cost-effective manner. In the 1960s Kenneth Hammond conducted a series of studies on clinical inferences made by nurses (Kelly & Hammond 1964). Among the topics examined were: types

2007 The Authors. Journal compilation 2007 Blackwell Publishing Ltd

JAN: RESEARCH METHODOLOGY

Judgment analysis and coronary heart disease

of cognitive tasks nurses faced in practice (Hammond et al. 1966a), information units nurses used in problem-solving (Hammond et al. 1966b), information-seeking strategies nurses used when assessing the state of their patients (Hammond et al. 1966c), and how nurses revised their judgments when presented with new information (Hammond et al. 1967). Since that time the education of nurses and their roles in the healthcare system have been expanded considerably. Hammond went on to develop social judgment theory (SJT) and its policy capturing (PC) technique, a general approach to how humans make decisions in the face of irreducible uncertainty (Hammond et al. 1975). PC is also known as judgment analysis. Social judgment analysts studying clinical decision-making often investigate how healthcare professionals make differential diagnoses and disease management decisions. Much of the early work in this area is reviewed by Wigton (1988a, 1988b). Since then, SJT has continued to provide a useful perspective for studying judgment and decision-making in the healthcare domain. Hogge and Murrell (1994) investigated the assessment of professional competence of British nursing students by hospital nurses responsible for the supervision and evaluation of their practice. Thompson et al. (2005) employed PC to model the clinical information nurses used in critical care education. Holzworth and Wills (1999) examined nurses’ decisions to seclude and restrain psychiatric patients. Unsworth and Thomas (1993) applied PC to understand how rehabilitation professionals made accommodation decisions for stroke patients at discharge. Dhami and Harries (2001) examined physician’s decisions to prescribe lipid-lowering agents, and Smith et al. (2003) studied physicians’ prescribing decisions in the treatment of depression. Tape et al. (1991) explained regional differences in diagnosis of pneumonia using judgment analysis. Sorum et al. (2002) showed how PC could be used to compare sequential vs. independent process models linking diagnosis and treatment decisions. SJT has also proved useful for assessing the effectiveness of physician education efforts (Wigton et al. 1990).

experience; this culminates in the award of a Master’s degree. According to the American Academy of Nurse Practitioners (2005), 66% of NPs work in primary care settings that include doctors’ offices and general practice clinics, emergency departments and critical and acute care facilities. NPs manage a wide spectrum of patient conditions ranging from acute to chronic. Patient care under this doctor (MD)/NP model is comparable to the MD-alone model (Aigner et al. 2004, Horrocks et al. 2002, Kinnersley et al. 2000). Patients report greater satisfaction with the health care they receive under the MD/NP model (Horrocks et al. 2002, Kinnersley et al. 2000). Horrocks et al. (2002) report that NPs spend more time with the patients per clinic visit on average than physicians do. NPs represent an important resource in early detection of numerous diseases. Understanding how they use patient characteristics to form estimates of patient risk for disease and how such risk estimates influence treatment plans and referral decisions has the potential to improve general disease prevention efforts. Cardiovascular disease accounts for nearly half of all deaths in the developed world (World Health Organization, 2002), and coronary heart disease (CHD) is estimated to surpass infectious disease as the world’s number one cause of death and disability by the year 2020 (Gaziano 2005). CHD encompasses angina pectoris, myocardial infarction and death due to coronary disease. In 2004–2005 the American Heart Association (AHA) awarded US$146Æ6 million in research grants aimed at preventing CHD (AHA, 2006). The AHA issues evidence-based guidelines informing professionals about current standards of care and provides clinical prediction rules for identifying patients at risk for CHD. One of the most widely circulated rules in the USA is based on the ongoing Framingham study (Anderson et al. 1991). Based on its prevalence and consequences, and on organized efforts aimed at its prevention, we chose CHD as the first disease to study in the course of our investigations into how NPs estimate patient risk for disease. Our investigation is grounded in the SJT perspective, and we review SJT and its PC method in the next section.

Background

Social judgment theory

In the United States of America (USA) many primary care physicians employ nurse practitioners (NPs) to share the care of patients (Leclaire 2005). In 2004 there were an estimated 141,209 licensed NPs in the USA (U.S. Department of Health and Human Services 2005). NPs receive advanced education beyond their preregistration training, including advancedlevel course work in pathophysiology, pharmacology, health assessment and primary care, and several hours of practice

Social judgment theory evolved through the 1960s and 1970s as a method and perspective for understanding judgment as it is exercised within a particular ecological context (Cooksey 1996). Despite its name, SJT is not a theory per se, but a meta-theory offering direction to research in judgment (Brehmer 1988). SJT is based on Brunswik’s (1955, 1956) concepts of probabilistic functionalism and representative design. Probabilistic functionalism refers to how a person


437

J.W. Beckstead and K.D. Stamp

functions in an environment filled of uncertain information. Brunswik suggested that some cues may be substituted for one another, referring to this as vicarious functioning. The intent of representative designs is to sample the substance of the environment so that it retains its realistic content and feel from the participant’s point of view. Hammond et al. (1975) provided the first complete description of how these concepts could be applied to the study of judgment. SJT is concerned with the correspondence between a person’s judgments and the environment. These relationships may be illustrated using the lens model (see Figure 1). Referring to Figure 1, Xk denotes the attributes of some multi-attribute object of judgment that in the context of the judgment task are called cues. Y refers either to criteria or responses. The subscripts e and s designate the environment and the subject respectively. (Although we recognize that it is more appropriate to refer to the people studied as ‘participants’ rather than ‘subjects’, we maintain the latter word when describing the lens model because it applies to both human and animals and to maintain continuity of symbology with past scientific literature.) The hats (^) denote predicted values, usually from least squares regression models of the respective ‘sides’ of the lens. Cues are related to criteria via some weight (wek) and correspondingly to the subject’s responses (wsk). These weights may be correlation coefficients, or alternatively regression coefficients. The model allows for correlations among the cues. The term ra is the correlation between the person’s judgments and the criteria and is referred to as achievement. Two multiple correlations in the lens model, Re, the extent to which the criterion is predictable in the environment and Rs, ECOLOGY (CRITERION)

CUES

the extent to which the person applies her judgment policy in a systematic manner, dubbed cognitive control, are estimated from regression models. The amount of knowledge that the person has about the relationships of cues to the criteria is expressed using two other bivariate correlations G and C, referring to linear and nonlinear relationships respectively. G is the correlation between the predicted values from each regression equation (Yê and Yˆs) and C is the correlation between the residuals from each regression (Ye–Yê and Ys–Yˆs). The relationships among these various correlations in the lens model are summarized in the lens model equation (LME) developed by Hammond et al. (1964) in the context of studying clinical judgment data. Tucker (1964) simplified the equation somewhat, and in its current form it is represented as: ra ¼ GRe Rs þ C½ð1 R2e Þð1 R2s Þ

1=2

The LME decomposes achievement into distinct components (Stewart, 2001). According to the LME, ra is a function of: 1) the criteria’s predictability in the environment Re, represented by the multiple R resulting from the regression of the criteria onto the cues; 2) the judge’s knowledge of cue-criteria relationships G, assessed as the correlation between predictions made from the two regression models; 3) cognitive control (Rs) or how consistently a judge utilizes her knowledge of these relationships, represented by the multiple R resulting from the regression of judgments onto the cues; and 4) a final component C, representing the influence of unmodeled aspects of the ecology and of judgments, that is formed by the correlation among the residuals from these two regression equations. When linear models fit well to both the criteria and the judgments, C is typically small or zero. Under

SUBJECT'S JUDGMENTS

X1 ws1

we1 we2

e

Ye Re Predictability

we3

e

X3

we4 wek

Ye -

X2

ws2 ws3 ws4

X4 . . .

Ys

s

Rs Control

wsk Ys -

s

Xk ra Achievement G Linear Knowledge C Unmodeled Knowledge

438

Figure 1 The lens model of judgment (See text for details). 2007 The Authors. Journal compilation 2007 Blackwell Publishing Ltd


these conditions, the LME then illustrates that task accuracy = knowledge · task predictability · cognitive control. Policy capturing, or judgment analysis, is a term referring to the application of multiple correlation/regression methods to obtain a representation of a judge’s policy. PC studies often focus on small samples of experienced decision-makers such as physicians, nurses or other professionals who have domain-specific expertise (Brehmer & Brehmer 1988) [for example, Holzworth and Wills (1999) examined how nine nurses made decisions to restrain or seclude psychiatric patients, and Thompson et al. (2005) reported on 23 nurses’ use of information in critical care education]. PC studies present an experienced judge (e.g. an NP) with a series of m profiles (e.g. patients) to be judged on some relevant dimension, in this case risk of CHD. The profiles are constructed of k representative cues that can take on different values. Analysis proceeds on an individual, or idiographic, basis. The relative importance (weights) of the cues may then be determined using the standardized regression coefficients, or correlation coefficients. In the current study, we used standardized regression coefficients to measure cue importance. Policy capturing designs allow calculation of various indices of accuracy. The primary index is achievement, ra. Two other indices are precision (the amount, on average, that a judge’s responses differ from the criterion value) and elevation (the amount by which the judge’s overall mean rating of risk is too high or too low when compared with the mean of the criterion) (Cooksey 1996). In PC designs that include replicated profiles it is possible to estimate cognitive consistency or the extent to which the judge performs similarly when responding to identical profiles on different occasions. Both cognitive control and cognitive consistency represent how orderly the individual is at making sense of the environment. Insight refers to the correspondence between the individual’s self-reported cue importance and the importance weights derived via statistical analysis. Insight may be gauged by substituting self-reported weights into the regression equation and comparing the predictions from this model to those made from the statistical model.

The study Aim The aim of this study was to examine how NPs combine information when estimating patient risk of CHD. Specifically, the following research questions were addressed: 1) How do individual NPs distribute importance weights


among cues as they estimate risk? 2) How accurate are their estimates when compared with a ‘gold-standard’ prediction rule? 3) How well does the additive linear model capture NPs’ judgment policies? 4) How much agreement is there among NPs as they estimate risk? 5) How much self-insight do NPs have into their risk estimation processes?

Participants The volunteer judges used in the study were a convenience sample of 15 NPs practising in west-central Florida, USA. They were recruited through word of mouth. Two were male. NPs ranged in age from 34 to 55 years, with an average of 46Æ1. The majority (66Æ6%) worked in primary care settings. The judges had, on average, 10Æ1 years of experience practising as NPs (range 3–23).

Materials Materials were presented to the judges in booklets. Booklets contained a cover page describing the purpose of the study, instructions for the PC task, a series of patient profiles, a section asking judges to indicate how they assigned importance to the cues during the PC task, and a section requesting demographic information. These sections are described below. Policy capturing task Wigton (1988b) discusses important choices for researchers to consider when designing and analysing simulated cases in clinical judgment tasks. These include: selection of cues, choice of cue values, selection of cases, and method of analysing judgments. The investigator’s decisions on the first three can affect the relative weights clinicians assign to the cues. The fourth consideration has implications for introducing systematic error into judgment models that can result from inappropriate matching of analytic method and type of outcome variable. The current task was constructed with attention to these issues. Our choice of cues and outcome were determined by examining the literature on evidence-based practice. One of the most widely-circulated prediction rules for CHD developed by Anderson et al. (1991) is based on the ongoing Framingham study. The rule and its development are now described. The prediction equation (Anderson et al. 1991) is in the form of a non-proportional hazards Wiebull-distributed failure time regression model constructed on a sample of 5573 patients followed for 12 years. The equation includes eight patient characteristics: gender, age, smoking status,


439


total cholesterol level, high-density lipid level (HDL), systolic blood pressure, and whether or not the patient has been diagnosed with diabetes or left ventricular hypertrophy. These risk factors represent an optimal set and were selected from a larger set of potential risk factors using stepwise selection procedures (see Anderson et al. 1991 for details). The central goal of judgment analysis is to determine what cues are used in judgment and how cue utilization compares with the ideal ecological validity of those cues (Hammond & Stewart 2001). Brunswik (1956) advocated for representative designs in the experiments. The intent of these designs is to sample the substance of the ecology so that it retains its realistic content and feel from the participant’s point of view. Hammond (1966) discusses the difficulty of doing so and notes that often the best an investigator can obtain is to sample the relations of the ecology, including retaining distributions and intercorrelations among cues that the judge would ordinarily encounter. The advantage here comes in the form of greater generalizability. Following Wigton’s (1988b) suggestions, we used the distributions of these cues reported by Anderson et al. (1991) to construct a population of cases with similar means and variances on all variables. After constructing such a population of cases, we randomly selected 70 cases for presentation. The means, standard deviations, and correlations among cues were consistent with

those found in Anderson et al. (1991) and are shown in Table 1. Nurse practitioners judged the risk that each patient had for developing CHD in the next 10 years by assigning a value from 0% to 100%. To obtain an estimate of cognitive consistency, we repeated the first 10 profiles at the end of the sequence. Thus, NPs judged a total of 80 profiles. To address the face validity of the profiles, judges were asked to rate ‘Overall, how realistic were the patient descriptions you read?’ on a 0–10 scale ranging from ‘not at all realistic’ to ‘completely realistic’. Self-report of cue importance Following completion of the PC task, judges were asked to indicate how much importance they placed on each cue by allocating 100 points among the eight cues. Demographic information We asked judges to provide their age, gender, years of practice as an NP, and practice setting.

Data collection procedure Judges were tested individually and in small groups in office and classroom settings. After obtaining informed consent, instructions were read aloud to the judges. The procedure

Table 1 Characteristics of 70 hypothetical patients used in coronary heart disease risk judgment task Characteristic

Percentage

Min

Max

Mean

SD

Sex (male) Age (years) Systolic blood pressure Left ventricular hypertrophy Cholesterol level High density lipids Smoker Diabetic Risk of CHD in next 10 years (0–100 scale)

53% – – 11% – – 43% 23% –

– 32 98 – 145 28 – – 0

– 72 178 – 317 70 – – 56

– 52Æ5 132Æ5 – 214Æ1 47Æ4 – – 14Æ4

– 11Æ4 19Æ2 – 36Æ2 11Æ7 – – 12Æ6

Sex Age SBP LVH Cho HDL Smo Dia Risk

Sex 1Æ000 0Æ111 0Æ018 0Æ249 0Æ301 0Æ618 0Æ050 0Æ167 0Æ272

Age 1Æ000 0Æ427 0Æ145 0Æ333 0Æ061 0Æ058 0Æ256 0Æ663

Correlations among cues and criterion SBP LVH Cho HDL Smo

1Æ000 0Æ073 0Æ148 0Æ031 0Æ120 0Æ366 0Æ502

1Æ000 0Æ139 0Æ309 0Æ052 0Æ232 0Æ632

1Æ000 0Æ057 0Æ127 0Æ102 0Æ130

1Æ000 0Æ092 0Æ065 0Æ447

1Æ000 0Æ079 0Æ131

Dia

Risk

1Æ000 0Æ466

1Æ000

CHD, coronary heart disease; SBP, systolic blood pressure; LVH, left ventricular hypertrophy; Cho, cholesterol; HDL, high density lipids; Smo, smoker; Dia, diabetes. 440




took an average of 32 minutes (range 20–47) to complete. Data collection took place in 2006.

Results How well does the additive linear model capture NPs’ judgment policies?

Ethical considerations The study was approved by the university’s institutional review board for research. Participants were assured of anonymity and confidentiality.

Various indices are available for assessing the appropriateness of the linear model for the judgment task. To the extent that the data are consistent with the proposed model, substantive analyses and conclusions are tenable. The average Rs value across judges was 0Æ828 and ranged from 0Æ687 to 0Æ928, suggesting that the judgment policies were adequately captured (see Table 2). The degree of agreement (G) between predictions from Anderson’s equation and the NPs’ judgment policies averaged 0Æ785 and ranged from 0Æ501 to 0Æ893 suggesting that NPs had a reasonably high degree of knowledge about the relationships between the risk factors and CHD. Taken together, these indices support the use of the additive linear model for analysing NPs’ estimates of risk and justify subsequent analyses.

Data analysis Data were analysed to address the research questions raised above. We present our results below in sections corresponding to these questions. Regression analyses were conducted separately for each NP using SPSS version 15. Prior to aggregating any correlation coefficients (ra, Rs, G, and testretest r) Fisher’s Z transformation was applied; aggregated values were then transformed back to original metric for presentation. Before considering questions of accuracy, cueweighting strategies, and self-insight, we address the quality of our data. Consistency was measured for each NP by calculating the correlation of responses made to the 10 duplicate cases with those made to their counterparts in the set of 70 cases. The mean of this test-retest correlation was 0Æ838, indicating sufficient although non-uniform performance. Judges indicated that the patient profiles were realistic Mean = 7Æ8, SD = 2Æ26, offering support for the face validity of the judgment task.

Table 2 Summary of NPs’ performance on coronary heart disease risk judgment task

How accurate are NPs’ estimates as compared with Framingham equation estimated risks? Using the LME, it is possible to estimate what each judge’s ra would be, given Rs, G, and Re. Anderson et al. (1991) did not report any index of fit analogous to multiple Re. In the current study, we did not have ecological criteria (Ye, i.e. whether each patient actually developed CHD within a

Judge

Rs

G

ra

Elevation

RMSE

rt-rt

Insight

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

0Æ823 0Æ795 0Æ839 0Æ849 0Æ776 0Æ832 0Æ851 0Æ904 0Æ928 0Æ866 0Æ812 0Æ746 0Æ834 0Æ726 0Æ687







Rs, cognitive control; G, linear modeled knowledge; ra, achievement, elevation is the average of risk estimates; RMSE, Root Mean Square Error; rt-rt, test-retest measure of intra-judge consistency, Insight is the multiple correlation between judge’s risk estimates and predictions of risk obtained by replacing regression coefficients with judge’s self-reported cue- importance weights. 2007 The Authors. Journal compilation 2007 Blackwell Publishing Ltd

441


10 year period) but we did have Yê, the risk of CHD predicted by the Framingham equation. Whatever value used for Re in the LME, the relative accuracy among the judges (i.e. their rank order on achievement) will be preserved. Therefore, for computational convenience, we assume that Re = 1, and define ra as an index of the relative accuracy among judges. This definition of ra allows us to examine individual differences in accuracy using the Framingham equation as an ecological criterion. Achievement ranged from 0Æ370 to 0Æ842 (average ra = 0Æ675). Judge 7 had the highest value and judge 15 had the lowest. A second index of accuracy is the root mean square error (RMSE), which may be thought of as an index of relative precision among the judges. Judge 9 was the most accurate (RMSE = 9Æ49) while judge 11 was the least accurate (44Æ22). The average value over all judges was 25Æ91. A third way to quantify accuracy is examine the overall extent to which a judge over- or under-estimates risk relative to criterion values. This type of inaccuracy (elevation bias) was based on the average of each NP’s risk estimates compared with 14Æ40, the mean of the criterion. All NPs over-estimated risk. The average estimated risk was 35Æ00. Judge 9 showed the least bias (15Æ43– 14Æ40) while judge 6 showed the most (50Æ00–14Æ40).

How do NPs distribute importance weights among cues as they judge risk? The NPs varied in their distribution of cue importance weights as they estimated patient risk. Cue weights were standardized regression coefficients (bs). In order for these weights to be comparable across judges, they were normalized by dividing each judge’s bs by the sum of the absolute values of their eight bs and multiplying by 100. Normalized weights for all judges are shown in Table 3, along with ecological weights for comparison. Vicarious functioning makes it possible for judges to perform a task with the same degree of accuracy while weighting cues differently. For example, judges 3 and 12 showed the same achievement (0Æ747), but Table 3 reveals that they obtained this level of accuracy using very different cue-weighting strategies. Indeed, despite the positive relationship between cholesterol and risk for CHD, judge 12 showed a negative weight – perhaps because of confusion of cholesterol with HDL.

How much agreement is there among NPs in their assessment of relative patient risk? Despite using different cue-weighting strategies, there was considerable agreement among NPs in their judgments of 442

Table 3 Normalized subjective weights assigned to cues in coronary heart disease risk judgment task Judge

Sex

Age

SBP

LVH Cho HDL

Smo Dia

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 M SD Ecology weights

1 1 3 1 5 11 0 0 3 3 10 0 6 9 3 2Æ3 4Æ7 8Æ6

12 25 10 24 0 35 31 4 4 7 2 19 2 4 1 11Æ3 12Æ0 22Æ2

1 14 3 18 16 6 13 9 11 10 7 17 20 12 24 12Æ1 6Æ1 10Æ4

14 3 13 7 17 9 5 8 17 17 5 12 17 1 25 14 25 5 19 14 11 13 20 6 20 15 14 21 23 2 16Æ3 8Æ7 6Æ0 7Æ2 20Æ1 6Æ1

20 19 21 20 12 14 18 16 8 19 16 11 14 13 19 16Æ0 3Æ7 9Æ5

18 1 11 3 2 1 4 3 7 13 14 9 1 5 6 2Æ9 7Æ8 11Æ4

31 18 25 22 31 15 16 29 36 15 27 18 23 24 22 23Æ5 6Æ2 11Æ6

SBP, systolic blood pressure; LVH, left ventricular hypertrophy; Cho, cholesterol; HDL, high density lipids; Smo, smoker; Dia, diabetes. Weights are transformed standardized regression coefficients: .X jbij jÞ 100 weightij ¼ ðbij

risk. Agreement was determined by calculating the pairwise correlations among judges across profiles. There were 105 possible pairs of NPs to assess. The average correlation was r = 0Æ691 and the range was 0Æ370–0Æ873. All pairwise correlations were statistically significant (P values