DEVELOPMENTAL MEDICINE & CHILD NEUROLOGY
ORIGINAL ARTICLE
Development of a parent-report computer-adaptive test to assess physical functioning in children with cerebral palsy II: upperextremity skills | KATHLEEN MONTPETIT MSCOT 2 | NATHALIE BILODEAU MSCOT 2 | HELENE M DUMAS MS PT 3 | MARIA A FRAGALA-PINKHAM MS PT 3 | KYLE WATSON DPT 4 | GEORGE E GORTON BS 5 | PENGSHENG NI MD 6 | RONALD K HAMBLETON PHD 7 | MJ MULCAHEY OT PHD 4 | CAROLE A TUCKER
STEPHEN M HALEY
PHD PT PCS
PHD PT
1
6
1 Temple University, Philadelphia, PA, USA. 2 Shriners Hospitals for Children, Montreal, Canada. 3 Franciscan Hospital for Children, Boston, MA, USA. 4 Shriners Hospitals for Children, Philadelphia, PA, USA. 5 Shriners Hospitals for Children, Springfield, MA, USA. 6 Health and Disability Research Institute, School of Public Health, Boston University, Boston, MA, USA. 7 University of Massachusetts Amherst, MA, USA. Correspondence to Dr Carole A Tucker at College of Health Professions, Temple University, 3307 North Broad Street, Philadelphia PA 19140, USA. E-mail:
[email protected]
PUBLICATION DATA
Accepted for publication 26th November 2008. Published online 11th March 2009. LIST OF ABBREVIATIONS
ASK CAT MACS PEDI
Activities Scale for Kids Computer-adaptive test Manual Ability Classification System Pediatric Evaluation of Disability Inventory PODCI Pediatric Outcomes Data Collection Instrument UE49 49-item upper-extremity item bank ACKNOWLEDGEMENTS
Supported by the Shriners Hospital for Children Foundation (grant no. 8957) and an Independent Scientist award to Dr Haley (National Center on Medical Rehabilitation Research ⁄ NICHD ⁄ NIH, grant no. K02 HD45354-01A1).
The specific aims of this study were to (1) examine the psychometric properties (unidimensionality, differential item functioning, scale coverage) of an item bank of upper-extremity skills for children and adolescents with cerebral palsy (CP); (2) evaluate a simulated computer-adaptive test (CAT) using this item bank; (3) examine the concurrent validity of the CAT with the Pediatric Outcomes Data Collection Instrument (PODCI) upper-extremity core scale; and (4) determine the discriminant validity of the simulated CAT with Manual Ability Classification System (MACS) levels and CP type (i.e. diplegia, hemiplegia, or quadriplegia). Parents (n=180) of children and adolescents with CP (spastic diplegia 49%, hemiplegia 22%, or quadriplegia 28%) consisting of 102 males and 78 females with a mean age of 10 years 6 months (SD 4y 1mo, range 2–21y), and MACS levels I through V participated in calibration of an item pool and completed the PODCI. Confirmatory factor analyses supported a unidimensional model using 49 of the 53 upper-extremity items. Simulated CATs of 5, 10, and 15 items demonstrated excellent accuracy (intraclass correlation coefficient [ICCs] >0.93) with the full item bank, had high correlations with the PODCI upper-extremity core scale score (ICC 0.79), and discriminated among MACS levels. The simulated CATs demonstrated excellent overall content coverage over a wide age span and severity of upper-extremity involvement. The future development and refinement of CATs for parent report of physical function in children and adolescents with CP is supported by our work.
The ability to perform functional tasks with the upper extremities is an important predictor of success in daily activities and participation in children and young adults with cerebral palsy (CP).1 Assessment of upper-extremity skills can be challenging, as functional activities are composed of fundamental subcomponents of upper-extremity movements (e.g. proximal and distal movements, in-hand manipulations) over a range of different activities (e.g. self-
care, writing) within different functional contexts (e.g. school, home). Often, several measures of upper-extremity function are used concurrently to capture changes due to maturation, progressive impairment, or pre- ⁄ post-intervention.2 Performance-based upper-extremity assessments such as the Quality of Upper Extremity Skills Test,3 Assisting Hand Assessment,4 Shriners Hospital Upper Extremity Evaluation,5 and the Melbourne Assessment of Unilateral ª The Authors. Journal compilation ª Mac Keith Press 2009 DOI: 10.1111/j.1469-8749.2009.03267.x
725
Upper Limb Function6 have been used to capture change in upper-extremity skills in children with CP. Such performance-based assessments require clinician administration and focus on a limited set of specific task-related aspects of upper-extremity function. In contrast, the Pediatric Outcomes Data Collection Instrument (PODCI),7 the Pediatric Evaluation of Disability Inventory (PEDI),8 and the Activities Scale for Kids (ASK) focus on assessing activity performance within the natural setting using a parentreport format.9 The PODCI includes eight items specific to upper-extremity function (upper extremity core scale), the ASK has eight items within a personal care ⁄ dressing category, and the PEDI contains 63 dichotomous response upper-extremity-related items in the self-care domain. In these measures with relatively few items or items with only dichotomous responses that specifically characterize upper-extremity function, ceiling and floor effects may exist, and adequate precision and responsiveness to detect changes in upper-extremity skills may be inadequate. The ABILHAND-Kids, a 21-item Rasch analysis-based upperextremity function questionnaire for children with CP, has been reported.10 The reported psychometrics of the ABILHAND-Kids indicate that it may provide a more responsive measure to capture change in manual ability over time than existing measures. However, there remains a requirement for measures that balance the need for more precise measurement with greater ease in the burden of administration. This balance is even more difficult to achieve in a single instrument when trying to account for wide differences in both ages and skill levels. With the advent of contemporary measurement technology such as the computer-adaptive test (CAT), better precision and depth may be obtainable while reducing the response burden. The use of a CAT to measure upper-extremity skills is further supported by recent work that demonstrated the feasibility of measuring selfcare in children with physical disabilities using a CAT adaptation of the PEDI.11 The application of CAT technology could be of great benefit in the development of a parent-report instrument that provides sufficient coverage of the full range of upper-extremity skills across the continuum of degrees of disability and ages. The specific aims of this study were to (1) the examine psychometric properties (unidimensionality, differential item functioning, scale coverage) of an item bank of upperextremity skills for children and adolescents with CP; (2) evaluate a simulated CAT using this item bank; (3) examine the concurrent validity of the CAT with the PODCI upper-extremity core scale; and (4) determine the discriminant validity of the simulated CAT with Manual Ability Classification System (MACS) levels and CP type (i.e. diplegia, hemiplegia, or quadriplegia). Our long-term goal 726 Developmental Medicine & Child Neurology 2009, 51: 725–731
was to create a multifaceted item bank that could assess global physical health, upper- and lower-extremity skills, and activity via CAT technology for the monitoring of the functional outcomes of clinical programs and the assessment of change with interventions.
METHOD Participants A convenience sample of parents (n=180) of children and adolescents with CP was recruited from the clinical programs in four pediatric hospitals (Shriners Hospital for Children [SHC] Philadelphia, SHC Montreal, SHC Springfield, Franciscan Hospital for Children). Ethnic representation corresponding to the current US census was targeted for recruitment; however, only respondents with fluency in English were included. Parents of children who had not undergone any major interventions (e.g. surgery, botulinum toxin injections) in the past 6 months were eligible for participation. Participants ranged in age from 2 to 21 years of age, with MACS12 levels of I to V. The demographics of the children are summarized in Table I. The Table I: Demographics and characteristics of participants (n=180) Age (y)
n (%)
1.4), four items were not included in the final item bank because they were not well fit by the item response theory model. Sometimes misfitting items are retained when they are needed to ensure content coverage but in this instance, the content (movement subcomponents) of the four test items was covered by other items that were fit by the model. Thus the initial item pool of 53 items was reduced to an item bank of 49 items (UE49). The comparative fit index value of 0.906 indicated an excellent fit and indicated that 90.6% of the covariance between pairs of items in the data is reproducible by the unidimensional Rasch partial credit model (Appendix SI). The Tucker–Lewis index value of 0.992 further supports this indication of adequate unidimensional model fit. The root mean square error of approximation of 0.1 is acceptable. If the 49 items were to be given as a single test, score reliability as reflected by coefficient alpha would be 0.981 in samples similar to the one used in this study. The UE49 item set demonstrated a reasonable distribution of items with our respondent sample across the full continuum of upper-extremity performance. There was an adequate match between person score distributions 728 Developmental Medicine & Child Neurology 2009, 51: 725–731
100 90 80
Ceiling: n=12 (6.7%)
70 60 50 40 30 Floor: n=2 (1.1%) 20 10 10 PARTICIPANTS
10 CATEGORIES
0
Figure 1: Item map for upper-extremity (UE) scale. Representation of the frequencies of estimated `skill' levels for participants compared with the item response levels along the difficulty continuum of UE skills. The vertical scale represents the relative difficulty of UE functioning with lower numbers representing easier skills and the higher numbers representing more difficult skills. The left-hand column of bars represents the distribution of our sample participants along the skill continuum of UE functioning with each bar representing 10 participants. The right-hand column of bars represents the distribution of available response categories of our UE49 item set. Ideally these two columns should appear similar with the distribution of possible item response categories and participants spread across the construct scale.
and expected item category values (Fig. 1), although small ceiling (6.7%) and floor (1.1%) effects were noted. Only one of the UE49 items (‘Using both of his ⁄ her hands together, my child can tear or rip paper’) showed significant differential item functioning, and then only for type of CP, but not for other known groups (i.e. age, sex, and MACS). ICCs between simulated score estimates on the 5-, 10-, and 15-item CATs and the UE49 indicated a high degree of correspondence (Table II). The 15-item upper-extremity CAT and the 10-item CAT reproduced UE49 scores, with little information lost in going from a 15- to a 10-item CAT. However, the accuracy of the 5-item CAT relative to the UE49 form was less than for the other simulated CAT forms (Table II). Comparison between the performance of simulated CATs of 5, 10, 15, and all 49 upper-extremity items with the PODCI upper-extremity physical function core scale core demonstrated high correlations of 0.78 for the simulated CAT of five items, and a correlation of 0.79 for the simulated CATs of 10, 15, and 49 items.
Table II: Performance comparison based on intercorrelation coefficients (ICCs) of simulated 5-, 10-, and 15-item computer adapted tests with each other and the upper extremity 49 full item set. ICCs
Full set
5-CAT
5-CAT
0.9580 (0.941–0.967)
10-CAT
0.9852 (0.980–0.989)
0.9754 (0.966–0.981)
15-CAT
0.9898 (0.986–0.992)
0.9693 (0.957–0.976)
10-CAT
0.9956 (0.994–0.997)
Score
Values are derived using an ICC (3,1), two-way mixed effects model, with the 95% confidence interval in parentheses.
90 80 70 60 50 40 30 20 10 0
Full Item Set CAT-15 CAT-10 CAT-5
Level I
Level II Level III Level IV, V MACS Levels
Figure 2: Discriminant validity of the upper extremity domain with the Manual Ability Classification System (MACS). Comparison is made between computer-adaptive test (CAT) scores based on simulated CATs with four different stopping routes (using either the full item set, 5 items, 10 items, or 15 items).
The full upper-extremity skill scale with 49 items was able to discriminate across type of CP [F(2,177)=35.83, p
