Are we as good as we think we are? Self-assessment ...

The Cognitive Behaviour Therapist, 2009, 2, 43–50 doi:10.1017/S1754470X08000081

ORIGINAL RESEARCH

Are we as good as we think we are? Self-assessment versus other forms of assessment of competence in psychotherapy Fiona M. Mathieson1∗ , Tracey Barnfield2 and Graeme Beaumont3 1

Department of Psychological Medicine, Otago University, Wellington, New Zealand Psychology Clinic, Massey University, Wellington, New Zealand 3 Blueprint Centre for Learning, Porirua, New Zealand 2

Received 23 April 2008; Accepted 29 October 2008; First published online 26 November 2008

Abstract. The Oracle of Delphi’s admonition to ‘know thyself’ may be more difficult than it seems, when it comes to self-assessment of competence in cognitive behaviour therapy (CBT). This paper investigates the accuracy of self-rating of competence in relation to other measures such as ‘direct’ assessment of videotaped sessions or supervisor ratings. Self-assessment of competence is something most of us do and arguably has an important role in professional development, but it may also be biased and unreliable. Two measures were developed, based on the Cognitive Therapy Scale – Revised, to assess student and supervisor assessments of competence at the start and end of a CBT training course. Competence data across a range of measures from a 5-year audit of a postgraduate CBT course is presented and the relationship between self-rated and other-rated measures explored. Results are discussed and it is suggested that trainee self-assessment, while not found in this study to be correlated with other measure of competence, may provide important information about confidence development, and areas where a trainee perceives they have strengths and weaknesses. Key words: CBT, psychotherapy training, self-assessment, therapist competence.

Introduction How good are we as therapists at assessing our own competence? This exploratory study investigated the reliability of self-assessment of therapeutic competence through exploring the relationships between self-assessment and direct/indirect assessment by others. Verifying competence is an important responsibility of training programmes. It is a complex task, and it is important that we find rigorous methods of assessing competence (James et al. 2001). A number of reviews have emphasized the need for multiple methods of assessment of competence particularly in relation to determining the outcomes of psychotherapy training programmes (Scofield & Yoxtheimer, 1983; Alberts & Edelstein, 1990; Milne et al. 1999;

* Author for correspondence: Ms. F. M. Mathieson, Department of Psychological Medicine, Otago University, Wellington, PO Box 7343, South Wellington 6242, New Zealand. (email: [email protected]) © 2008 British Association for Behavioural and Cognitive Psychotherapies

44

F. M. Mathieson et al.

Barnfield et al. 2007). However, little is known about the relationships between the various methods of assessment of competence. This study investigated the relationships between three different measures of therapist competence: self-report, supervisors’ assessment via indirect observation (therapist self-report which includes written formulations and case reports) and assessment by raters via direct (videotape) observation using a rating scale and asked: What are the relationships? Do they add value? Are they all necessary? A case for self-assessment Self-assessment of competence can be defined as ‘a form of appraisal that makes a comparison between one’s behavioural outcomes and an internal or external standard’ (Boekaerts, 1991, p. 11). Elks & Kirkhart (1993) argue that clinicians continually evaluate their own practice, albeit privately: we form ongoing global appraisals of our work based on intuition and experience, through monitoring our own reactions and make judgements in comparison to guidelines for good practice. Watts (1990) argues that self-assessment is valuable because through it we become more questioning, analytical, self-motivated, self-challenging and curious, while clinicians who do not may be sliding towards incompetent practice, may easily become careless or out of date without realizing it and may waste valuable efforts to learn. Moreover, Bjork (1999) suggests that self-assessment is arguably as important as our actual competence in that it affects time allocation, whether we seek further study or practice, volunteer for tasks and instil confidence in others. Self-assessment of a clinician’s skills and abilities is therefore, arguably, a crucial element in developing and maintaining competence. A case against self-assessment In his book on ‘self-insight’, Dunning (2005) suggests that it is a complicated and subtle matter to know one’s self. He argues that it is surprisingly difficult and that we come to flawed and sometimes downright incorrect conclusions about ourselves. For example, a number of studies have found a weak to non-existent connection between people’s self-estimation and actual ability, with a tendency to overestimation. Overestimation of ability by the trainee has been found to be particularly evident with less able trainees (Bjork, 1999). Ames & Krammath (2004) found that most people overestimate their social judgement and mind-reading skills and they suggest narcissism, not actual competence may predict self-estimated ability. This may fit with a broader tendency for people to overestimate their competence across a range of areas, dubbed the ‘above average effect’ (Alicke et al. 1995). Kruger & Dunning (1999), in studies of university undergraduates, found that people who overestimate their competence make poor choices; a further manifestation of their incompetence is their lack of meta-cognitive ability to identify their unfortunate choices. Kruger & Dunning argue that the skills that engender competence in a domain are often the same skills needed to evaluate competence in that domain. Miller (1998) argues that although self-assessment of one’s own therapy is clearly open to bias, in that it is subjective and potentially self-serving. However, LaSala (1997) did not find support for a self-serving bias: clinicians underrated therapeutic gains compared to client ratings.

Are we as good as we think we are?

45

Limitations of other forms of assessment The reliability and validity of assessment of students by supervisors was explored in a review of the literature by Ellis & Ladany (1997). They noted that supervisees may be evaluated primarily qualitatively via their indirect report of therapy and suggest that perceptions of the supervisee by the supervisor may influence evaluation. They conclude that using this process to assess supervisee competence may be unreliable, bringing into question its usefulness. The unreliability of indirect supervisee report was evident in a study by Ladany et al. (1996) who found that supervisees frequently did not disclose key material from therapy during supervision. Trainees’ reasons for not disclosing included a perception that the material was too personal; that they had a poor alliance with the supervisor, and that they felt uncomfortable about disclosing the material (embarrassed by a lack of empathy). The limitations of self-assessment and indirect assessment by supervisors underscore the importance of incorporating external, direct observation techniques (video-/audiotapes) in obtaining valid and reliable judgements of therapeutic competence, as direct observation may provide a more objective and accurate measure of competence. This method would generally involve observation of therapy sessions via a two-way mirror, videotape or audiotape and the therapist’s skills being rated using a structured rating scale (Ellis & Ladany, 1997). This is what the Cognitive Therapy Scale – Revised (CTS-R) (Blackburn et al. 1997, 2001; Milne et al. 1997), used in the current study, is designed for. The limitation with this approach, however, as noted by Miller (1998) is that it only draws on limited data from a single session. Relationships between the measures Within the psychotherapy field there is little research that examines the relationships between subjective self-assessment of competence, indirect (supervisor) assessment measures and the more objective assessment via direct observation. Acosta (1995) investigated psychologists’ self-assessment of multicultural competence, in comparison to an objective measure (a multicultural awareness-knowledge-skills survey). Results revealed the subjective self-assessment to be positively related to the objective scores. The relationship between measures of self-assessment of competence and those involving objective assessment is unclear. This lack of clarity supports the argument that self-assessment of competence, as a sole measure of competence, is not a good stand-alone measure if we want valid, reliable judgements of therapeutic competence. The purpose of this exploratory study was to examine the relationships between self-assessed competence, supervisor-rated competence via indirect observation and other-rated competence via direct observation. Two exploratory measures were devised, the Student Self-Rating Form and the Supervisor Rating Form (T. V. Barnfield, unpublished course materials; both available as Supplementary material on the Journal’s website). It was expected that measurement of competence based on direct observation would be superior to measurement of competence based on indirect observation or self-assessment. Hypotheses (1) There would be a weak but positive relationship between self-assessment of competence and other forms of assessment of competence.

46


(2) There would be a positive relationship between ‘other’ measures, i.e. between direct and indirect (supervisor) external measures

Method Participants Participants were 34 experienced mental health professionals enrolled in a half-time (20 hours per week on average) 30-week postgraduate-level CBT course. There were 24 females and 10 males. Thirty participants were identified as European (Pakeha) New Zealanders, two as Maori New Zealanders and two from other unspecified ethnicities. The age range was from 24 to 61 years (mean age 39 years). Fourteen were qualified social workers, 12 were registered nurses, five were occupational therapists and the remaining three were from other mental health disciplines. Fifteen participants had more than 6 years’ clinical experience, nine had between 4 and 6 years, and ten had 3 or fewer years of experience. Participants were selected for the course on the basis of years of mental health experience, interpersonal effectiveness in a selection interview, referee endorsement and curriculum vitae. None of the course participants had undergone prior formal training in CBT.

Procedure CBT training course The training consisted of a half-time postgraduate 30-week course. It was designed to provide workforce development for experienced mental health professionals and has been running for 5 years. The course consisted of two components, a 300-hour academic component consisting of lectures, tutorials, reading and assignments, and a 300-hour practicum. There were three block-taught (40-hour) weeks at intervals throughout the course. A mixture of teaching modalities were used during block weeks, including didactic lectures, practical exercises, role-plays, videotapes of experts demonstrating specific skills and small group discussions. The approach taught was based on the work of Dr Aaron Beck. A special aspect of the course was the inclusion of a component on working with Maori, the indigenous people of New Zealand. For the practicum component, each trainee worked full time in a mental health setting and developed CBT skills while working with patients as part of their regular client load. Participants received individual supervision during their training from clinical psychologists with expertise in CBT. Each participant received 50 hours of supervision. Supervisors also received supervision and further training from the CBT course lecturers.

Measures Cognitive Therapy Scale – Revised (CTS-R) The CTS-R (Blackburn et al. 1997, 2001) is a 14-item scale designed to measure competence across a range of therapeutic skill areas. Items are rated on a 7-point likert scale ranging from


47

incompetent (0) to expert (6). The CTS-R has been shown to have high internal consistency and adequate inter-rater reliability (Blackburn et al. 2001). Supervisor Rating Form (SRF) This measure was developed for the present study, due to the absence of similar supervisor measures of competence in CBT. It is based on the CTS-R (Blackburn et al. 1997), but is based on supervisor observations from supervision sessions and several observed therapy sessions, rather than rating a single session, as occurs with the CTS-R. The form requested the supervisor to rate the student’s skills, knowledge and behaviours across a range of areas pertinent to CBT. The twenty-four items measuring skill areas sampled on the CTS-R and additional skills and behaviours were rated on the same 6-point likert scale (0, poor performance; 5, excellent performance). This measure also allowed supervisors to comment on the strengths and weaknesses of the student, and asked whether supervisors would refer selected patients to the student. For the purposes of the present study, only the 24 skill development items were used. Participants could obtain a score between 0 and 120, with higher scores indicating greater levels of skill. Student Self-Rating Form (SSRF) This measure was also developed for the present study, again, due to the absence of a suitable available measure. It is the student self-rating version of the SRF. The items were identical to the SRF but were self-rated by the students. The students were also asked to comment on their perceived strengths and weaknesses and state whether they were comfortable using CBT with selected patients. Again, only the 24 skill development items were used in the present data analysis, and were scored out of a maximum of 120. Higher scores indicating greater levels of skill.

Design/data collection A videotape of each trainee’s therapy session was assessed using the CTS-R after 2 and 6 months of training. Three clinical psychologists, experienced in CBT training, therapy, supervision, and using the CTS-R, independently rated the videotapes. The supervisors and course participants completed the SRF and SSRF, respectively, after both 2 and 6 months of training.

Results The inter-relationships between CTS-R, SRF, and SSRF were examined. Pearson productmoment correlations were calculated and are presented in Table 1. Although none of the correlations presented in Table 1 reached significance, they are low and in the predicted direction. CTS-R ratings have low and positive relationships with the SRF and SSRF ratings at both time 1 and time 2. It is of interest to note that the CTSR has a greater correlation with the SRF than it does with the SSRF. This relationship is consistent at both time 1 and time 2. The SRF and SSRF have moderate and positive


48

Table 1. Pearson product–moment correlations between CTS-R, SRF and SSRF at time 1 and time 2 Measures Time 1

1. CTS-R 2. SRF 3. SSRF

Time 2

CTS-R

SRF

CTS-R

SRF

0.32 0.15

0.21

0.23 0.10

0.27

CTS-R, Cognitive Therapy Scale – Revised; SRF, Supervisor Rating Form; SSRF, Student Self-Rating Form.

relationships with each other at time 1 and time 2, although these correlations do not reach significance. Discussion The results of this study were that self-assessment had a non-significant relationship with supervisor-rated competence and with other-rated competence assessed via the CTS-R but there was a trend in the hypothesized direction. In a recent article by Bennett-Levy & Beedie (2007) it is suggested that it is likely that self-perception of competence is related to actual therapeutic performance. The results of the present study suggest that this relationship may be weaker than was anticipated. The lack of a significant correlation between supervisor-rated competence and competence assessed via the CTS-R is of interest. It is possible that supervisor ratings were affected by the fact that they often shared their evaluations with their supervisees, so supervisors may have been kinder as a result. It may also have been that supervisors’ views of their students were influenced by their experiences of the student’s competence in other areas, such as in a social work or nursing role. Even if student self-assessment, on further investigation, proves unreliable as a measure of competence, it may be important to pay attention to it because it is likely to reflect students’ confidence in their competence as CBT therapists, which is likely to affect whether they take on cases, what cases they choose to take on, etc. Student self-assessments may also prove useful for course lecturers, through making it possible to identify students who are overestimating their competence in a way that may lead them to work in a harmful way or get out of their depth. Although the results of this study suggest that self-assessment may not have a strong relationship to more (supposedly) objective measures of competence, such as the CTS-R, it must be emphasized that reliability/validity data for the two measures developed for this study are not available. It may be that these tools did not measure what they were intended to measure. A future investigation into the reliability and validity of the SSRF and SRF would clarify this. The measures of competence used in this study differed between the supervisors, the students and the independent raters. A future study where students, supervisors and independent raters rate the same video sessions with the same measure, e.g. the CTS-R may also assist in further clarifying the relationship between self-assessment and competence.


49

Summary of main points • It is important that we find ways to verify psychotherapeutic competence as a result of training. • Current evidence suggests trainee self-assessment is not necessarily reliable. • Other measures such as supervisor ratings and direct assessment of videotaped sessions also have their limitations. • There is little evidence of relationships between these measures. • This exploratory study compared these three forms of assessment of competence and found no significant relationship between the measures. • Trainee self-assessment may have an important role in professional development, through the development of confidence and the identification of strengths and weaknesses. • Follow-up reading suggestion: Bennett-Levy & Beedie (2007).

Note Supplementary material accompanies this paper on the Journal’s website (http://journals. cambridge.org).

Declaration of Interest None.

References Acosta J (1995). Psychologist’s self-assessment of multicultural competence. Dissertation Abstracts International, Part B: Science and Engineering 56, 2850. Alberts G, Edelstein B (1990). Therapist training: a critical review of skills training studies. Clinical Psychology Review 10, 497–511. Alicke MD, Mark D, Klotz ML, Breitenbecher DL, Yurak TJ, Vredenberg DS (1995). Personal contact, individuation, and the better-than-average effect. Journal of Personality and Social Psychology 68, 804–825. Ames DR, Kammrath LK (2004). Mind-reading and metacognition: narcissism, not actual competence, predicts self-estimated ability. Journal of Nonverbal Behavior 28, 187–209. Barnfield TV, Beaumont GR, Mathieson FM (2007). Assessing the development of competency during post-graduate cognitive-behavioural therapy training. Journal of Cognitive Therapy: An International Quarterly 21, 140–147. Bennett-Levy J, Beedie A (2007). The ups and downs of cognitive therapy training: what happens to trainees perceptions of their competence during a cognitive therapy training course? Behavioural and Cognitive Psychotherapy 35, 61–75. Bjork RA (1999). Assessing our own competence: heuristics and illusions. In: Attention and Performance XVII: Cognitive Regulation of Performance: Interaction of Theory and Application (ed. D. Gopher and A. Koriat), pp. 435–459. Cambridge, MA: MIT Press. Blackburn I, Milne D, James I (1997). How are therapeutic competencies evaluated? Paper presented at the Annual Conference of the British Psychological Society (Division of Clinical Psychology). Herriot–Watt University, Edinburgh.

50


Blackburn I-M, James IA, Milne DL, Baker C, Standart S, Garland A, Reichelt FK. (2001). The revised cognitive therapy scale (CTS-R): psychometric properties. Behavioural and Cognitive Psychotherapy 29, 431–446. Boekaerts M (1991). Subjective competence, appraisals and self-assessment. Learning and Instruction 1, 1–17. Dunning D (2005). Self-insight: Roadblocks and Detours on the Path to Knowing Thyself. New York, NY: Psychology Press. Elks M, Kirkhart K (1993). Evaluation effectiveness from the practitioner perspective. Social Work 38, 554–563. Ellis MV, Ladany N (1997). Inferences concerning supervisees and clients in clinical supervision: an integrative review. In: Handbook of Psychotherapy Supervision (ed. C. E. Watkins Jr.), pp. 447–507. NJ, USA: John Wiley & Sons Inc. James IA, Blackburn I-M, Milne DL, Reichfelt FK (2001). Moderators of trainee therapists’ competence in cognitive therapy. British Journal of Clinical Psychology 40, 131–141. Kruger J, Dunning D (1999). Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology 77, 1121–1134. Ladany N, Hill CE, Corbett MM, Nutt EA (1996). Nature, extent, and importance of what psychotherapy trainees do not disclose to their supervisors. Journal of Counselling Psychology 43, 10–24. LaSala MC (1997). Client satisfaction: consideration of correlates and response bias. Families in Society 78, 54–64. Miller RR (1998). Practitioners as privileged evaluators of their own practice. Smith College Studies in Social Work 69, 78–83. Milne DL, Baker C, Blackburn I-M, James I, Reichelt FK (1999). Effectiveness of cognitive therapy training. Journal of Behavior Therapy and Experimental Psychiatry 30, 81–92. Milne DL, Blackburn I-M, James I (1997). Patient and therapist characteristics and competence. Paper presented at the Annual Conference of the British Psychological Society. Heriot–Watt University, Edinburgh. Scofield ME, Yoxtheimer LL (1983). Psychometric issues in the assessment of clinical competencies. Journal of Counselling Psychology 30, 413–420. Watts N (1990). Handbook of Clinical Teaching: Exercises and Guidelines for Health Professionals who Teach Patients, Train Staff, or Supervise Students. New York: Churchill Livingstone.

Learning objectives • Knowledge of the current state of the evidence regarding self-assessment of therapeutic competence. • Knowledge of different measures of assessing competence and their strengths and limitations. • Awareness that despite little evidence of correlation with other measures of competence, it may be useful to include self-assessment measures in training programmes.