Validity of Person Matching in Vocational ... - Wiley Online Library

Received 02/05/13 Revised 04/09/13 Accepted 04/12/13 DOI: 10.1002/j.2161-0045.2014.00074.x

Validity of Person Matching in Vocational Interest Inventories Stephanie T. Burns Research for more than 60 years has shown that entry into occupations can be predicted from scores on interest inventories at a rate better than chance (Donnay, 1997). The psychometric scoring methodologies used today by a majority of vocational interest inventories were developed in the 1920s and 1960s. Researchers are challenged with improving the theory and science behind vocational interest inventories to align them with current vocational constructions. In this study, validity comparisons were made between person matching and standard scoring based on 5,143 medical students who had taken a vocational interest inventory and had entered their medical residency. Person matching was found to improve differentiation between occupational groups and increase the amount of information offered in the scoring report; in addition, it could potentially increase occupational group assignment to advance vocational interest inventory validity. Keywords: person matching, career interest, narrative, assessment, hit rate

Vocational interest inventories help to clarify interests and promote occupational exploration (Creed, Patton, & Prideaux, 2006; Ihle-Helledy, Zytowski, & Fouad, 2004). Interests are permanent enough (an intraindividual correlation coefficient of .75 suggests high stability) and sufficiently unaffected by vocational training and experience to furnish a basis for predicting future behavior (Strong, 1943). Research for more than 60 years has shown that entry into occupations can be predicted from scores on interest inventories at a rate better than chance (Donnay, 1997). Follow-up studies ranging from 3 to 18 years suggest that hit rates for interest inventories can be between 32% and 69%, which is well above the chance expectancy rate (Donnay, 1997). Hit rates are operationally defined as an exact match between the individual’s chosen occupation and the occupation suggested by the vocational interest inventory. The greater the accuracy with which a scoring method calculated a person’s final occupational choice, the higher the hit rate reported. The present research investigated a rarely used psychometric scoring methodology known as person matching to potentially improve the validity of vocational interest inventories. There are two measures of the validity of a vocational interest inventory (Strong, 1943). First, validity rests on an interest inventory’s ability to differentiate between specific Stephanie T. Burns, Counselor Education and Counseling Psychology, Western Michigan University. Special thanks to the Association of American Medical Colleges for permitting the use of their data to perform this study, as well as to Mark L. Savickas, George V. Richard, and Erik J. Porfeli for their support during this process. Correspondence concerning this article should be addressed to Stephanie T. Burns, Counselor Education and Counseling Psychology, Western Michigan University, 1903 West Michigan Avenue, Kalamazoo, MI 49008-5226 (e-mail: [email protected]). © 2014 by the National Career Development Association. All rights reserved. 114

The Career Development Quarterly

ACACDQ_v62_n2_0614TEXT.indd 114

JUNE 2014 • Volume 62

5/16/2014 3:02:41 PM

occupational groups (Kuder, 1977a; Strong, 1943). Second, the interest inventory needs to assign individuals to membership in one or more occupational groups based on their interest inventory scores (Strong, 1943). Comparisons in validity between the standard scoring method of Strong (developed in 1927) and the person-match scoring method of Kuder (developed in 1977) were examined.

Standard Scoring Method Strong’s (1943) scoring method is based in a modernist paradigm and compares individuals to occupational groups. Individuals are described as “patients” who take on a passive role by waiting for answers to be provided to them by assessments, others, or the environment so that they can make an occupational selection (Brott, 2004; Bujold, 2004). Predicting occupational choice results from objectively counting individuals’ responses to items on interest inventories, which can then be matched to the world of work (Bujold, 2004; Cohen, Duberley, & Mallon, 2004). Strong (1943) discovered empirically that occupational groups produced a pattern of responses on an interest inventory, which could then differentiate between the average member of that occupational group and the average individual in general (Crites, 1969). Strong’s occupational scales eliminated similarities in interests and used only the differences in interest to set members of an occupation apart from general interest patterns (Campbell & Borgen, 1999; Case & Blackwell, 2008; Strong, 1943). He believed that creating occupational scales for each occupation was invaluable because interest in certain kinds of activities linked the test taker to realworld behavior in that occupation (Campbell & Borgen, 1999). Strong collected interest inventory scores from 250 to 500 individuals from a specific occupation and then contrasted those scores with the scores of nearly 5,000 men-in-general and women-in-general by weighting the items in terms of importance to the occupational group (Campbell & Borgen, 1999; Kuder, 1977a; Strong, 1943). By weighting an individual’s rating of activities as liked or disliked, the final score on the inventory could classify an individual as a member of an occupational group by factoring out common interests (Campbell & Borgen, 1999; Kuder, 1977a; Strong, 1943). For example, if 180 members of an occupation liked the item on the inventory, 75 were indifferent to the item on the inventory, and 45 disliked the item on the inventory, then the interest inventory item for that occupational scale would be scored 180 for like, 75 for indifferent, and 45 for dislike (Strong, 1943). The development of occupational scales with an interest inventory comes with imperfections. Guilford (1952) sharply criticized Strong (1943) for using factor analysis to generate occupational scales for interest inventories where the same items are scored for more than one scale. Furthermore, it was suggested that the results of the factor analysis of ipsative scores were likely to be misleading because the correlation between any two scales is determined largely by the scoring system and could be regarded as an artifact (Kuder, 1977a). Even Strong noted that if different individuals made up the men-in-general and the women-ingeneral reference groups, the correlations changed dramatically in his occupational scales (Kuder, 1977a). The Career Development Quarterly


JUNE 2014 • Volume 62 115

5/16/2014 3:02:41 PM

Person-Match Scoring Method In the postmodernist paradigm, it is believed that individuals make meaning through social experiences (Brott, 2001; Cohen et al., 2004). Therefore, an individual’s interests would not be the result of internal processes but would instead be the result of social and environmental systems (Brott, 2001; Cohen et al., 2004). Furthermore, individuals are seen as constructing their reality by taking action to create their preferred future (Brott, 2004; Bujold, 2004). This philosophy uses assessments to help individuals define their preferred self and preferred work environment to construct their preferred future (Brott, 2004; Cohen et al., 2004). Kuder’s (1977a, 1977b) interest inventories are based in a postmodernist paradigm, with test takers’ scores being matched to the scores of individuals from a reference group database who are enthusiastic about their work. Kuder’s method scored all items on the interest inventory so that similarities and differences could be measured at the same time (Donnay, 1997; Ihle-Helledy et al., 2004). Kuder estimated 5,000 cases to be sufficient for the reference group for an interest inventory using person matching as a psychometric scoring methodology (Kuder, 1977b). He created a reference group of people from several hundred occupations who were enthusiastic about their work, which could not be accomplished through Strong’s (1943) use of occupational group scales. Next, Kuder compared test takers’ scores with those of individuals from the reference group to offer test takers biographies (including current occupations, past occupations, lifestyles, future goals, and descriptions of what reference group members like best and least about their occupation) of the closest 20 reference group members (Kuder, 1977a). In this way, the scoring report provided the test taker the ability to consider career themes found within the narratives to improve career decision making over vocational interest inventories that offered only occupational scale scores and three-digit personality codes (Kuder, 1977b).

Comparing the Two Scoring Methods Person matching has the ability to assist any population with career exploration. In this study, medical students served as the population for examining person matching with vocational interest inventories. General vocational interest inventories (e.g., the Strong Interest Inventory [SII; Donnay, Morris, Schaubhut, & Thompson, 2005] or Self-Directed Search [SDS; Holland, 1994]) demonstrate very limited success in helping medical students to select their medical specialty, because the generic interests measured in these types of inventories share similarities across all medical specialties (Glavin, Richard, & Porfeli, 2009; Sodano & Richard, 2009). Therefore, to compare the two scoring methodologies, I selected the Medical Specialty Preference Inventory–Revised (MSPI-R; Richard, 2011) because a data set existed that included a large sample with item scores, longitudinal data, and demographic data. In addition, medicine has the most complex and diverse array of specialties that require decidedly different abilities, skills, and talents (Rogers, Creed, & Searle, 2009; Sodano & Richard, 2009; Stratton, Witzke, Elam, & Cheever, 2005). Because there are more than 100 medical specialties to choose 116




5/16/2014 3:02:41 PM

from, medical specialty selection is a crucial part of a medical student’s career development. It is the biggest and most enduring decision made during medical school and is equal to choosing medicine as a career (Borges, 2007; Reed, Jernstedt, & Reber, 2001). Medical students receive similar core classes while in medical school to prepare them to ultimately engage in their chosen medical specialty. Medical students commonly change their minds about their specialty (Scott, Gowans, Wright, & Brenneis, 2007) during the first 2 years of medical school while they are studying the basic sciences and have many questions about the different medical specialties (Gough, 1979; Reed et al., 2001). Longitudinal studies have suggested that between 60% and 75% of medical students change their specialty while still in medical school (Markert, 1983; Savickas, Alexander, Jonas, & Wolf, 1986). Many students seek counseling when making this choice because they must choose a specialty before they have sufficient experience and information (Savickas, Brizzi, Brisbin, & Pethtel, 1988). Although medical students can swap or change residencies at any time if they can find a vacancy in a new residency program, changing a residency costs hospitals time and money in training the resident. Furthermore, the resident becomes frustrated and loses time and money while finding and then completing a second residency (Borges, Gibson, & Karnani, 2005). One previous study used the Person Matching Model (Kuder, 1977b) with scores on the Sixteen Personality Factor Questionnaire (Cattell, Cattell, & Cattell, 1993) to predict the medical specialty choices of 420 medical students (Hartung, Borges, & Jones, 2005). Results indicated moderate (43% to 60%) hit-rate accuracy. The current study compared the ability of standard and person-match scoring to predict specialty selection. The MSPI-R was used to compare the validity of its current standard scoring system with the validity of scoring it using person matching. Person matching could deliver results for every medical specialty instead of only the 16 offered today with the MSPI-R. This would allow medical students to potentially be matched to the full range of medical specialties to help foster medical specialty exploration, which has been called for in the literature (Borges & Savickas, 2002). Validity implications are required before resources will be expended to study the full person-matching protocol with currently established vocational interest inventories.

Method Participants The participants were 5,143 (2,898 female and 2,245 male) medical students enrolled in medical schools across the United States who took the MSPI-R between January 2005 and April 2008. Participants represented several racial/ethnic groups: European American (n = 3,447), Asian (n = 767), African American (n = 343), Hispanic (n = 250), other (n = 53), American Indian/Alaska Native (n = 27), and Native Hawaiian/ Pacific Islander (n = 6). Two hundred fifty participants did not identify a race/ethnicity. Ages ranged between 24 and 57 years at the time of taking the MSPI-R, with an average age of 30 years (SD = 3.22). The medical students eventually practiced in 44 medical specialties ranging from internal medicine (n = 1,007) to medical toxicology (n = 1), with The Career Development Quarterly



5/16/2014 3:02:41 PM

an average of 117 medical students per medical specialty (SD = 213), as listed in Table 1. To test both psychometric scoring methodologies to be inclusive of the largest number of medical specialties (the literature calls for including Table 1 Frequencies of Medical Residency Selections Made by the Reference Group and the Criterion Group Reference Group Residency Choice Internal medicine Pediatrics Emergency medicine Family medicine Obstetrics/gynecology Surgery Anesthesiology Psychiatry Orthopedic surgery Radiology Pathology Internal medicine pediatrics Otolaryngology Ophthalmology Neurology Dermatology Physical medicine and rehabilitation Urology Neurological surgery Radiation oncology Plastic surgery Pediatrics/psychiatry/child and adolescent psychiatry Internal medicine/psychiatry Preventive medicine Psychiatry family practice Pediatrics/physical medicine and rehabilitation Child and adolescent psychiatry Infectious disease Child neurology Internal medicine/dermatology Internal medicine/emergency medicine Internal medicine/family practice Internal medicine/neurology Nuclear medicine Pediatric emergency medicine Pediatric hematology/oncology Vascular surgery Pediatrics/medical genetics Cardiovascular disease Gastroenterology Medical toxicology Pediatrics/dermatology Psychiatry/neurology Sports medicine

n

%

Criterion Group n

%

1,007 695 514 507 379 330 297 245 201 194 149 139 94 65 64 58 55 33 27 18 16

19.58 13.51 9.99 9.86 7.37 6.42 5.77 4.76 3.91 3.77 2.90 2.70 1.83 1.26 1.24 1.13 1.07 0.64 0.52 0.35 0.31

56 44 40 40 30 30 24 24 24 20 20 20 14 14 14 14 14 14 14 10 10

11.20 8.80 8.00 8.00 6.00 6.00 4.80 4.80 4.80 4.00 4.00 4.00 2.80 2.80 2.80 2.80 2.80 2.80 2.80 2.00 2.00

11 4 4 4

0.21 0.08 0.08 0.08

10 0 0 0

2.00 0 0 0

3 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1

0.06 0.04 0.04 0.04 0.04 0.04 0.04 0.04 0.04 0.04 0.04 0.04 0.04 0.02 0.02 0.02 0.02 0.02 0.02

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Note. Reference group N = 5,143 and criterion group N = 500. 118




5/16/2014 3:02:42 PM

more than 16 medical specialties), I selected a subset of the full reference group and then compared it individually to the entire reference group. By allowing all members of the reference group to be part of the random sample selection process, the study mimicked the reality that not all medical students taking the MSPI-R enter the medical specialty calculated by the inventory. A stratified random sample of 500 medical students (250 women and 250 men) was chosen from the reference group of 5,143 medical students to ensure a high confidence level and low confidence interval (CI). Twenty-two medical specialties were selected to be part of the stratified random sample because they contained at least 10 medical students (five female and five male) who had entered the medical specialty. Because person matching requires that occupations must be represented in the reference group for a match to be made, one person was removed from the entire reference group at a time so that at least nine other individuals from a medical specialty were represented in the reference group. It would significantly hinder the hit rate of person matching to randomly remove 500 people from the reference group when seven of them may have come from a specialty represented by 10 individuals. To further mimic real-world processes in the study, I matched the proportion of medical students in each medical specialty in the reference group to the proportion of medical students in the 22 medical specialties chosen to be part of the stratified random sample. The random selection of each stratum required sorting the database of 5,143 medical students first by their chosen medical specialty and then by gender. An online random-number generator based on atmospheric noise was accessed, which allowed for a minimum and maximum number to be entered; this process generated a true random number between the stated minimum and maximum range for each gender by medical specialty. Measure The MSPI-R (Richard, 2011) is an online instrument that measures interest in 18 areas of medical practice and predicts entrance into 16 major medical specialties. The MSPI-R provides information to medical students to help them choose a medical specialty appropriate to their interests following graduation from medical school. To take the MSPI-R, a medical student selects one of seven scale points to indicate the degree of desirability for each item on the inventory; the next item is displayed until the MSPI-R is completed. Medical students instantaneously receive a report of results including 16 specialty choice probabilities along with 18 medical interest scales (Glavin et al., 2009). For each of the 16 medical specialties, a percentage score is reported that indicates the likelihood that the student will enter into the specialty. The 16 medical specialty percentages, when added together, total 100% and are presented in order from the highest to the lowest likelihood that the student would enter each specialty (Glavin et al., 2009). Students are instructed to select the two or three specialties with the highest probabilities to explore further. Next, students receive their medical interest scale scores to identify their highest and lowest scoring interests in 18 areas of medical practice that are experienced in varying degrees in each medical specialty. The Career Development Quarterly



5/16/2014 3:02:42 PM

There are 150 items included in the MSPI-R; however, only 102 items are used to score the instrument (Glavin et al., 2009). Of those, 88 items are used to score the 18 medical interest scales, and 30 items are used to score the 16 specialty choice probabilities (Richard, 2011). Sixteen of the items are scored in both the medical interest scales and the specialty choice probabilities (Richard, 2011). The remaining 48 items are not scored and may be used in the future for possible replacement of items as needed to improve the predictive ability of the instrument and to support the development of new specialties (Richard, 2011). The medical interest scales represent 18 areas of medical practice that involve knowledge and information, services and procedures, and types of problems that are experienced to varying degrees in each specialty. They represent another way to assess fit with a specialty. The 18 medical interest scales are Complex Problems, Comprehensive Care, Diagnostic Precision, Emergency–Critical Care, History Taking, Home Health Care, Immediate Results, Knowledge of Anatomical Structures, Knowledge of Organ Systems, Laboratory Results, Palliative Care, Patient Counseling, Prevention and Education, Procedural Care, Psychological Care, Reproductive Care, Social Context, and Technology in Medicine. In addition, the MSPI-R calculates preferences for 16 medical specialties: anesthesiology, dermatology, emergency medicine, family medicine, internal medicine, neurology, obstetrics and gynecology, orthopedic surgery, otolaryngology, pathology, pediatrics, physical medicine and rehabilitation, psychiatry, radiology, surgery, and urology. The Cronbach’s alpha, a measure of internal consistency, indicated reliabilities ranging from .77 for History Taking and Diagnostic Precision to .94 for Psychological Care (Richard, 2011). Comparisons between the second edition MSPI factors and the MSPI-R medical interest scales suggested high positive correlations and indicated sufficient validity of the new MSPI-R medical interest scales (Richard, 2011). A study by Porfeli, Richard, and Savickas (2010) suggested a hit rate of 56% with the MSPI-R. The hit rate is higher than the hit rates of vocational interest inventories that are used to predict an individual’s general occupational choice, such as the 2005 SII with a hit rate of 38% (Gasser, Larson, & Borgen, 2007) and the SDS with a hit rate of 47% (Glavin & Savickas, 2011). Data Analysis Procedures The MSPI-R’s raw scores were analyzed in two different ways: (a) person matching using all 150 items on the inventory and (b) the standard method of scoring the MSPI-R. All person-matching analyses used Cronbach and Gleser’s (1953) difference squared (D2) values, a person-matching statistic, to determine the linear distance of profile similarity. D2, the sum of squared Euclidian distances between self- and other ratings of traits, reflects differences in elevation, scatter, and shape between individuals’ scores on the same inventory (Cronbach & Gleser, 1953). The D2 statistic is a descriptive statistic and as such does not include the concepts of statistical power or statistical precision. When one uses the D2 statistic, Score 1 (from a test taker) is subtracted from Score 2 (from an individual from a reference group), with the resulting difference being squared (Hartung et al., 2005). When the differences between the two scores are squared, the result becomes normally distributed (Cronbach & Gleser, 1953). In the D2 statistic, there is no upper 120




5/16/2014 3:02:42 PM

limit on the distance between two scores. However, the closer the D2 calculation is to zero, the closer the two individuals scored similarly on the inventory, which signifies a close person-to-person match. D2 values were calculated by subtracting all 150 item scores on the MSPI-R for each member of the stratified random sample from all 150 item scores on the MSPI-R for each individual in the reference group, with the resulting differences being squared. The squared differences comparing the two individuals were summed to obtain a final score. Scores comparing the test taker with each of the 5,142 members of the reference group were then placed into rank order from the lowest score to the highest score. Specialty choice probabilities obtained through standard scoring for the 16 medical specialties were generated for each member of the stratified random sample by using beta weights (which were derived from multinomial logistic regression analysis) with 30 out of 150 MSPI-R items. The highest probability score was documented as the first match-predicted specialty (also known as the top match) based on standard scoring. The second through fifth highest probability scores were documented as the second through fifth matches, respectively. Hit rates were recorded for person matching by listing the closest five matches from the 5,142 members of the reference group for the 500 members of the random sample. To record the calculations for standard scoring, I generated the five highest specialty choice probabilities for each member of the stratified random sample by using beta weights with 30 out of the total 150 MSPI-R items. Next, the kappa coefficient was calculated for the top match to determine the interrater agreement between the predicted and the actual medical specialty selected for the two scoring methods. Finally, chance expectancy hit rates were calculated to determine the expected hit rates that would be achieved by chance alone.

Results Table 2 shows hit rates for person matching and standard scoring based on the five top predictions for the 500 members of the random sample. Table 2 Hit Rates for Person Matching and Standard Scoring Criterion Person matching (N = 500) Top match Second match Third match Fourth match Fifth match Total of top 5 Standard scoring (N = 500) Top match Second match Third match Fourth match Fifth match Total of top 5

n

Hit Rate (%)

111 48 36 26 30 251

22 10 7 5 6 50

165 67 41 36 26 335

33 13 8 7 5 67

Note. For standard scoring, the total of the individual hit-rate percentages for the top five matches differs from the “total of top 5” hit-rate percentages because of rounding.




5/16/2014 3:02:42 PM

Data suggested that standard scoring had an 11% advantage in accuracy as the top prediction, a 3% advantage in accuracy as the second prediction, a 1% advantage in accuracy as the third prediction, a 2% advantage in accuracy as the fourth prediction, and a 1% disadvantage in accuracy as the fifth prediction. Table 3 presents kappa coefficients for top-match hit-rate accuracy for actual versus predicted medical specialty for standard scoring and person matching. A kappa coefficient of less than .20 represents poor agreement, between .21 and .40 represents fair agreement, between .41 and .60 represents moderate agreement, between .61 and .80 represents good agreement, and between .81 and 1.00 represents very good agreement beyond chance (Landis & Koch, 1977). The interrater reliability for standard scoring suggested a kappa coefficient of .33 (p < .001), 95% CI [0.28, 0.38]. The interrater reliability for person matching proposed a kappa coefficient of .18 (p < .001), 95% CI [0.14, 0.22]. Standard scoring obtained the highest kappa coefficient representing fair agreement. In a study by Glavin and Savickas (2011) with the SDS, a kappa coefficient of .29 was suggested with 2,264 participants. In a study by Gasser et al. (2007) with the SII, squared canonical correlations were used and suggested moderate agreement (.40) with 1,872 participants. It appears that vocational interest inventories in general achieve fair to moderate hit-rate accuracy, and the MSPI-R is comparable. Table 4 presents the expected hit rates that would be achieved by chance for the top match with standard scoring and person matching. Person matching and standard scoring accurately placed medical students at a rate greater than chance into 16 out of 22 medical specialties. Because standard scoring did not calculate scores for six of the 22 medical specialties used in the study, it is understandable why standard scoring fell below the chance expectancy hit rates for those six listed specialties.

Discussion Both scoring methodologies struggle with top-match hit-rate accuracy, which is much lower than the reported hit rates. This phenomenon is also seen in general vocational interest inventories. It would appear that work to improve the scoring methodology of the MSPI-R and all vocational interest inventories needs to be continued. Furthermore, the data suggest that standard scoring outperformed person matching for accuracy in four of the top five predictions. Students taking the MSPI-R are instructed to select the two or three specialties with the highest probabilities to explore further. Person matching, as performed in this study, Table 3 Kappa Coefficients for the Top Match Criterion Standard scoring (N = 500) Person matching (N = 500)

Kappa .33 .18

Rating Fair Poor

SE .03 .02

p .000* .000*

Note. Agreement of kappa coefficients was rated as follows: poor (< .20), fair (.21–.40), moderate (.41–.60), good (.61–.80), and very good (.81–1.00; Landis & Koch, 1977). *p < .001.

122




5/16/2014 3:02:42 PM

Table 4 Chance Expectancy Hit Rates for the Top Match by Medical Specialty Medical Specialty Internal medicine Pediatrics Emergency medicine Family medicine Obstetrics/gynecology Surgery Anesthesiology Psychiatry Orthopedic surgery Radiology Pathology Internal medicine pediatrics Otolaryngology Ophthalmology Neurology Dermatology Physical medicine and rehabilitation Urology Neurological surgery Radiation oncology Plastic surgery Pediatrics/psychiatry/child and adolescent psychiatry

n

Chance Standard Expectancy Scoring

Person Matching

56 44 40 40 30 30 24 24 24 20 20 20 14 14 14 14 14 14 14 10 10

.11 .09 .08 .08 .06 .06 .05 .05 .05 .04 .04 .04 .03 .03 .03 .03 .03 .03 .03 .02 .02

.63 .59 .43 .53 .40 .47 .25 .42 .29 .15 .20 .00 .29 .00 .07 .14 .14 .07 .00 .00 .00

.30 .30 .38 .35 .27 .27 .08 .25 .46 .15 .20 .05 .21 .00 .14 .21 .00 .07 .00 .00 .00

10

.02

.00

.00

Note. N = 500. For both standard scoring and person matching, the number of medical specialties below the chance expectancy hit rate was six.

fell 16% when compared with standard scoring in predictive accuracy for the closest three matches. As compared with previous findings (Hartung et al., 2005), the current study is 7% higher in predictive accuracy for the top five matches. The present findings suggest that when the goal is determining the highest hit-rate accuracy for the test taker, as is the case in a modernist paradigm, standard scoring would be preferred over person matching. Despite a lower hit rate in the present study, person matching had the ability to obtain a 50% hit-rate accuracy for the closest five matches out of 5,142 individual reference group members as compared with standard scoring, which obtained a 67% hit-rate accuracy when identifying the top five out of 16 medical specialties. This finding suggests that further research is needed to refine how person-match scoring should be ideally performed and to understand how the full person-matching protocol would affect the value of the information test takers receive from the two methodologies. The increase in information received from the person-matching scoring report (including occupations, lifestyle, future goals, typical day, needed skills, and descriptions of what reference group members like best and least about their occupation) could help clients write their preferred futures by providing a context within which to make meaning, clarify values, and solidify decisions as they coconstruct with their environment their career story. The Career Development Quarterly



5/16/2014 3:02:42 PM

Limitations There are several limitations to this research. First, this study compared validity between only standard scoring and person matching. If one were to perform a study using the full person-matching scoring report, person matching might be able to demonstrate the full range of its advantages over the report of standard scoring, and then a full comparison of the two methodologies could be made. Second, Kuder (1977a) stated that, to be included in the reference group for person matching, the individual would have to be matched to other people who were enthusiastic about their work and scored the interest inventory in the same way. The present research did not ascertain if medical students in the reference group enjoyed working in their medical specialty, which may have hindered person matching’s hit-rate performance. Third, the reference group contained an uneven number of individuals in each medical specialty. Internal medicine had the largest group with 1,007 members, and there were six medical specialties with only one member. The extreme range in the number of medical students in each medical specialty in the reference group may have caused imbalances with person matching as a psychometric scoring methodology. Fourth, the study’s sample is derived from medical students who voluntarily completed the MSPI-R. It is impossible to require every medical student in the United States to take the MSPI-R; thus, the sample may comprise a specific type of medical student who is not representative of all medical students. Finally, only medical students who attended a medical college that is part of the Association of American Medical Colleges were able to access the MSPI-R and therefore participate in the study. Consequently, the results of this study may not generalize to medical students attending medical schools outside the United States. Research and Counseling Implications In addition to studying the full person-matching protocol to compare test takers’ views on the value of the two different scoring reports, this research has prompted other specific questions. Running the full person-matching protocol to study validity for person matching with all individuals in general in the reference group versus only individuals enthusiastic about their work would help to verify Kuder’s (1977a) assertion about enthusiasm. Research needs to investigate if having large differences in the number of members of an occupation in the reference group negatively affects person matching as a psychometric scoring methodology. Finally, this study did not examine how standard scoring and person-matching hit rates compared for medical students of different genders, races/ethnicities, and cultures. Researchers are challenged with improving the theory and science behind vocational interest inventories to incorporate current vocational constructions (Armstrong & Rounds, 2010). Career counseling increasingly incorporates approaches with a foundation in postmodern and constructivist and social constructionist philosophies (Amundson, 2009; Cochran, 1997; McMahon & Watson, 2010; Peavy, 1998; Savickas et al., 2009). These methods emphasize the use of narratives to assist clients in creating their preferred futures through activating meaningmaking processes. Central to this process is story. Most narrative career 124




5/16/2014 3:02:42 PM

counseling approaches must be administered by trained counselors, thus limiting their reach. Overwhelmingly, vocational interest inventories do not contain career stories/biographies, which results in a lack of career meaning-making processes for clients using vocational interest inventories as their primary means of making career decisions. Person matching offers an alternative scoring method that is more in line with the narrative and social constructionist perspectives and practices used in career counseling today. Although the hit rate for person matching fell below the hit rate for standard scoring, it may be useful to invest time and resources with the MSPI-R to provide students with Kuder’s (1977b) full Person Matching Model to further assist medical specialty decision making. First, person matching increased the number of occupations included in the scoring of the MSPI-R to offer the greatest ability to assign medical students to membership in an occupational group. Standard scoring with the MSPI-R could make predictions based on only 16 medical specialties. When person matching was used as the scoring methodology, all 44 medical specialties in the sample could be predicted for students. In this way, person matching has a greater ability to differentiate between and assign individuals to specific occupational groups and, therefore, increases vocational interest inventory validity when compared with standard scoring. Second, person matching offers test takers the ability to receive autobiographic data, thus allowing for more robust career exploration as compared with receiving an occupational title as is common in the scoring reports of vocational interest inventories. Finally, person matching offers the flexibility of adding additional occupations that have only a handful of individuals practicing in the occupation. This differs from standard scoring, where several hundred individuals must work in an occupation in order for an occupational scale to be created for a vocational interest inventory. The added features gained when using person-match scoring become increasingly important in a global economy demanding flexibility among individuals facing outsourcing and contractual work. Further research should investigate how other vocational interest inventories could (a) improve validity, (b) increase the amount and type of vocational information offered in the scoring report, and (c) include new occupations by using person matching as the scoring method.

References Amundson, N. E. (2009). Active engagement: The being and doing of career counseling (3rd ed.). Richmond, British Columbia, Canada: Ergon Communications. Armstrong, P. I., & Rounds, J. (2010). Integrating individual differences in career assessment: The Atlas Model of individual differences and the Strong Ring. The Career Development Quarterly, 59, 143–153. Borges, N. (2007). Behavioral exploration of career and specialty choice in medical students. The Career Development Quarterly, 55, 351–358. Borges, N., Gibson, D., & Karnani, R. (2005). Job satisfaction of physicians with congruent versus incongruent specialty choice. Evaluation & the Health Professions, 28, 400–413. Borges, N., & Savickas, M. L. (2002). Personality and medical specialty choice: A literature review and integration. Journal of Career Assessment, 10, 362–380. Brott, P. E. (2001). The storied approach: A postmodern perspective for career counseling. The Career Development Quarterly, 49, 304–313.




5/16/2014 3:02:42 PM

Brott, P. E. (2004). Constructivist assessment in career counseling. Journal of Career Development, 30, 189–200. Bujold, C. (2004). Constructing career through narrative. Journal of Vocational Behavior, 64, 470–484. Campbell, D., & Borgen, F. (1999). Holland’s theory and the development of interest inventories. Journal of Vocational Behavior, 55, 86–101. Case, J. C., & Blackwell, T. L. (2008). Review of the Strong Interest Inventory. Rehabilitation Counseling Bulletin, 51, 122–126. Cattell, R. B., Cattell, A. K., & Cattell, H. E. (1993). Sixteen Personality Factor Questionnaire (5th ed.). Champaign, IL: Institute for Personality and Ability Testing. Cochran, L. (1997). Career counseling: A narrative approach. Thousand Oaks, CA: Sage. Cohen, L., Duberley, J., & Mallon, M. (2004). Social constructionism in the study of career: Accessing the parts that other approaches cannot reach. Journal of Vocational Behavior, 64, 407–422. Creed, P., Patton, W., & Prideaux, L. (2006). Causal relationship between career indecision and career decision-making self-efficacy: A longitudinal cross-lagged analysis. Journal of Career Development, 33, 47–65. Crites, J. (1969). Vocational psychology: The study of vocational behavior and development. New York, NY: McGraw-Hill. Cronbach, L., & Gleser, G. (1953). Assessing similarity between profiles. The Psychological Bulletin, 50, 456–473. Donnay, D. (1997). E. K. Strong’s legacy and beyond: 70 years of the Strong Interest Inventory. The Career Development Quarterly, 46, 2–22. Donnay, D., Morris, M. L., Schaubhut, N. A., & Thompson, R. C. (2005). Strong Interest Inventory manual (Rev. ed.). Palo Alto, CA: Consulting Psychologists Press. Gasser, C. E., Larson, L. M., & Borgen, F. H. (2007). Concurrent validity of the 2005 Strong Interest Inventory: An examination of gender and major field of study. Journal of Career Assessment, 15, 23–43. Glavin, K., Richard, G., & Porfeli, E. (2009). Predictive validity of the Medical Specialty Preference Inventory. Journal of Vocational Behavior, 74, 128–133. Glavin, K., & Savickas, M. L. (2011). Interpreting Self-Directed Search profiles: Validity of the “rule of eight.” Journal of Vocational Behavior, 79, 414–418. Gough, H. (1979). Gough Medical Specialty Preference Scales: A report for counselors. Palo Alto, CA: Consulting Psychologists Press. Guilford, J. P. (1952). When not to factor analyze. Psychological Bulletin, 49, 26–37. Hartung, P., Borges, N., & Jones, B. (2005). Using person matching to predict career specialty choice. Journal of Vocational Behavior, 67, 102–117. Holland, J. L. (1994). The Self-Directed Search. Odessa, FL: Psychological Assessment Resources. Ihle-Helledy, K., Zytowski, D., & Fouad, N. (2004). Kuder Career Search: Test–retest reliability and consequential validity. Journal of Career Assessment, 12, 285–297. Kuder, F. (1977a). Activity interests and occupational choice. Chicago, IL: Science Research Associates. Kuder, F. (1977b). Career matching. Personnel Psychology, 30, 1–4. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. Markert, R. J. (1983). Stability and change of medical specialty choice in U.S. medical schools. Journal of Medical Education, 58, 589–590. McMahon, M., & Watson, M. (2010). Story telling: Moving from thin stories to thick and rich stories. In K. Maree (Ed.), Career counselling: Methods that work (pp. 53–63). Cape Town, South Africa: Juta. Peavy, R. V. (1998). SocioDynamic counselling: A constructivist perspective. Victoria, British Columbia, Canada: Trafford. Porfeli, E. J., Richard, G. V., & Savickas, M. L. (2010). Development of specialization scales for the MSPI: A comparison of empirical and inductive strategies. Journal of Vocational Behavior, 77, 227–237.

126




5/16/2014 3:02:42 PM

Reed, V. A., Jernstedt, G. C., & Reber, E. S. (2001). Understanding and improving medical student specialty choice: A synthesis of the literature using decision theory as a referent. Teaching and Learning in Medicine, 13, 117–129. Richard, G. (2011). Medical Specialty Preference Inventory, Revised edition technical manual. Washington, DC: Association of American Medical Colleges. Rogers, M. E., Creed, P. A., & Searle, J. (2009). The development and initial validation of social cognitive career theory instruments to measure choice of medical specialty and practice location. Journal of Career Assessment, 17, 324–337. Savickas, M. L., Alexander, D., Jonas, A., & Wolf, F. (1986). Difficulties experienced by medical students in choosing a specialty. Journal of Medical Education, 61, 467–469. Savickas, M. L., Brizzi, J., Brisbin, L., & Pethtel, L. (1988). Predictive validity of two medical specialty preference inventories. Measurement and Evaluation in Counseling and Development, 21, 106–112. Savickas, M. L., Nota, L., Rossier, J., Dauwalder, J., Duarte, M. E., Guichard, J., . . . van Vianen, A. E. M. (2009). Life designing: A paradigm for career construction in the 21st century. Journal of Vocational Behavior, 75, 239–250. Scott, I., Gowans, M. C., Wright, B., & Brenneis, F. (2007). Why medical students switch careers: Changing course during the preclinical years of medical school. Canadian Family Physician, 53, 94–95. Sodano, S., & Richard, G. (2009). Construct validity of the Medical Specialty Preference Inventory: A critical analysis. Journal of Vocational Behavior, 74, 30–37. Stratton, T. D., Witzke, D. B., Elam, C. L., & Cheever, T. R. (2005). Learning and career specialty preferences of medical school applicants. Journal of Vocational Behavior, 67, 35–50. Strong, E. K. (1943). Vocational interests of men and women. Stanford, CA: Stanford University Press.




5/16/2014 3:02:42 PM