Oct 22, 2010 - signed specifically for the study; and did not report .... (Cowan & Turner-Smith, 1999). In order to .... sign and interpretation of research studies,.
Assistive Technology
ISSN: 1040-0435 (Print) 1949-3614 (Online) Journal homepage: http://www.tandfonline.com/loi/uaty20
Psychometric and Administrative Properties of Measures Used in Assistive Technology Device Outcomes Research James A. Lenker MS and OTR/L and ATP , Marcia J. Scherer PhD , Marcus J. Fuhrer PhD , Jeffrey W. Jutai PhD and CPsych & Frank DeRuyter PhD To cite this article: James A. Lenker MS and OTR/L and ATP , Marcia J. Scherer PhD , Marcus J. Fuhrer PhD , Jeffrey W. Jutai PhD and CPsych & Frank DeRuyter PhD (2005) Psychometric and Administrative Properties of Measures Used in Assistive Technology Device Outcomes Research, Assistive Technology, 17:1, 7-22, DOI: 10.1080/10400435.2005.10132092 To link to this article: http://dx.doi.org/10.1080/10400435.2005.10132092
Published online: 22 Oct 2010.
Submit your article to this journal
Article views: 372
View related articles
Citing articles: 38 View citing articles
Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=uaty20 Download by: [University of Ottawa]
Date: 26 June 2017, At: 11:57
REVIEW
Ass! Techn ol 2005;17:7-22 © 2005 RESNA
Psychometric and Administrative Properties of Measures Used in Assistive Technology Device Outcomes Research *J ames A. Lenker, MS, OTRIL, ATP, , Marcia J. Scherer, PhD, +Marcus J. Fuhrer, PhD, §Jeffrey W. Jutai, PhD, CPsych, and [Frank DeRuyter, PhD "Department of R ehabil itation S cience, Sc hool of Public Health and Health Professions, University at Buffalo, Buffal o, New York ; "flns titute for Mat ch ing Person and Techn ology, Webster, New York; :;N ational In stitute of Ch ild Health and Hu man Developm ent , National In stitutes of Health, Dama scus, Maryl and; §Depa rtment of Phy sical Medi cine and R ehabilitati on, Unive rsi ty of West ern Ont ari o, Lond on, On tario, Canada ; and l!Division of Sp eech Pathol ogy and Audiology, Du ke University Medi cal Center, Durham, North Carolina
Although there have been numerous calls for increasing the quantity and quality of assistive technology outcomes research, no one has analyzed the nature of data that the field has been accumulating. This article su m m ar izes our evaluation of 82 outcome studies, published between 1980 and 200 I , addressing assistive te chnology devices (ATDs). Our data indicate that the "typical" ATD outcomes study published in the past 20 years is one that (a) used a sample population that was diverse in terms of age, di sability population, and type of ATD being used; (b) measured user-reported d ependent variables with instruments designed specifically for the study; (c) did not report adequate information on the reliability and validity for the measure m e n t instruments that were used; (d) did not discuss the staff workload associated with learning, administering, and scoring its data collection tools; and (e) did not differentiate its findings in terms of distinguishable participant subgroups. Several suggestions are provided to guide future development of assis tive technology outcome measures in the domains of usability, quality of life, and social role performance . In addition, seven recommendations are offered to outcomes researchers, policy makers,journal ed itors , and reviewers in order to improve the reporting of assistive technology outcomes research. Key Words: Assistive technology-Outcome measu res- O u tco m e s research-Psychometric properti es-Evidence-based practice.
Address correspondence a nd repri nt requests to J ame s A. Lenker, MS, OTRIL, ATP, 515 Kimball Tower, Univer sit y at Buffalo, Buffalo, NY 14214-3079.
The increasing demands for accountability of services throughout medicine and allied health indicate a need for research that focuses on quantitative as well as qualitative accounts of functional outcomes (Bedrosian, 1999; Fuhrer, Jutai, Scherer, & DeRuyt er , 2003 ). Use of outcome s data to support service delivery practices, so-called evidence-based pr actice, has the potential to inform and guide clinical practice so that health care practitioners "do the rights things right" (Gray , 1997 ). Thi s implies existence of a research literature that allows researchers and practitioners to learn from , and build upon, the previous effort s of others. Accumulating such a body of knowledge requires credible data (i.e., data that have been gathered in a reliable and efficient fashion using measurement tools that have demon strated validity; Dijkers et al. , 2002 ; Frytak, 2000 ), Althou gh there have been numerous calls for increa sing the quantity and quality of assistive technology (AT) outcome s research (DeRuyter, 1997 ; Fuhrer, 2001 ; Light, 1999 ; Smith, 1996), no one ha s analyzed th e na ture of data th at the field has been accumulating. Thi s article summa rizes our evaluation of 82 outcome studies, published between 1980 and 2001 , addressing assi stive technology devices (ATDs), a subset of AT (Gitli n, 2002 ). Seeking a diverse representation of ATDs, including ones with higher cost, we looked for studies concerned with mobility, posture and seating of wheelchair users, augmentative communication, 7
computer accommodation, and environmental control devices. Three principal characteristics were examined: the composition of sample populations, the outcome domains that were measured, and the reported psychometric and administrative properties of the measurement instruments that were used . KEY CONCEPTS IN AT OUTCOMES RESEARCH Definitions of AT Outcomes
DeRuyter (1995, 1997) and Smith (1996) describe the scope of outcomes measurement in terms of the needs and interests of various stakeholders in the process . At the individual level, end users, families, and caregivers need to know which devices will work best for their own circumstances. At the program level, third-party payers (e.g., insurance companies, HMOs, Medicaid , vocational rehabilitation agencies, and school districts) want to know that their funds are being allocated effectively. At the research and development level, manufacturers are interested in knowing how well their products work for end users, whereas service providers are constantly seeking information that improves the quality of their assessments and recommendations. At the societal level, federal agencies use outcomes information to justify continued support for related research programs. At the simplest level, outcomes are the result of an intervention (Scherer , 1998b). AT outcomes can be defined in terms of changes produced by assistive devices in the lives of users and their environments, including less dependence on other people and a variety of societal resources (Fuhrer et al., 2003). In the context of AT service provision , DeRuyter (1997) defines outcomes measurement as , "The evaluation process in the service delivery system that is designed to measure and establish a baseline of what works; how well something works; for which clients it works; and at what level of economic efficiency it works ." Importance of Measurement and Reporting Standards in Outcomes Research
Measurement science is concerned with accurate representation of domain attributes as they exist in the world, making it possible to draw mathematical comparisons of system attributes and performance (Law, 1987). The properties of reliability, validity, and administrative burden (i.e., the staff workload required for learning, administering, and scoring a measurement tool) that are associ8
ated with a particular tool provide a backbone that supports the credibility of data gathered using a measurement tool (Dijkers et al. , 2002; Frytak, 2000). Those properties of measurement instruments are considered to be essential components of the formal reporting of research results (Andresen, 2000; Johnston, Keith, & Hinderer, 1992; Law, 1987). The results of a recent study, however, suggest that these properties are substantially underreported by authors in the medical rehabilitation literature (Dijkers et al., 2002). PURPOSE Related Research
Previous authors have synthesized portions of the AT outcomes literature in terms of AT usage (Rogers & Holm, 1992; Wielandt & Strong, 2000) and quality of life (QOL) impact (Pape, Kim, & Weiner, 2002; Scherer, 1996). A three-volume guide on AT outcomes measurement was developed by RESNA that includes a compendium of extant measurement tools (DeRuyter, 1998; Scherer, 1998a; Smith, 1998; Smith & Scherer, 1998). It does not address application of the tools in outcomes studies, nor does it review the tools in a manner that allows comparisons within common measurement domains. A recent issue of Technology and Disability (Gelderbloom & de Witte, 2002) includes developmental summaries of individual measurement tools. Contributing authors described and discussed the measurement properties of their instruments with reference to a common set of criteria. This review provides useful information for specific tools, but it does not examine how they have been applied and reported by other researchers. Dijkers et al. (2002) evaluated the reliability and validity of 651 outcome measures as reported in 171 papers in the medical rehabilitation research literature. Information about reliability was provided for 20% of the outcome measures; information about validity was reported for less than 7% of them. Research Questions
The purpose of this study was to answer the following questions: • How adequately have published ATD outcomes studies reported the nature of the sample populations that were used? • How well have published ATD outcomes studies provided evidence of reliability, validity, and adASSISTIVE TECHNOLOGY , VOL. 17, NO.1
mini strative burden for measurement tools used to measure outcome variables? • How frequently have various outcomes domains been assessed in published ATD outcomes studies? METHODS Sample
Inclu sion Cri teri a
The objective was to identify all research studies publi shed from 1980 to 2001 that involved a followup st udy of ATD users and that measured one or more domains of ATD imp act. Both prospective and retrospective st udies were consider ed, as were studies that used either an experim ent al or nonexperimental design. Minimally, a subset of th e st udy participants must ha ve been users of one or more devices in an y of th e following categories: mobility aids (including crutches, canes, walk ers, and wheeled mobility devices); sea ting and postural su pport devices for wheelchair users; augmentative and alte rn ative communication (AAC); computer-based ATDs; or environmenta l control units (ECDs ). Exclu sion Criter ia
Among th e articles excluded were case studies, ethnographic st udies for which no outcome variables were identifi ed a priori , review articles, meta-an alyses, conference papers, and th ose for which an ATD did not constitute an ind epend ent variable. Studies including only children und er age 18 yea rs were also excluded, in part becau se the conceptual th eory und erlying outcomes for childre n is not yet well developed. The absence of a cont rol group, control peri od, or detailed specification of th e treatment/a) was not a reason for disqualifying an article if it sa tisfied all other criteria. S earch Strategy
A lit erature search, restricted to English langua ge articles, was conducted usin g several bibliogra phic databases (CINAHL, Medlin e, EMBASE , ERIC, an d PsychInfo) and th e following key words: assisti ve technology, rehabilitation technology, as sistive technology devices, ass istive devices, assistive computing technol ogy, environmenta l control uni ts, electronic aids to daily living , mobility aids, wheeled mobility aids, wheelchairs, augmentative an d alternative communic ation devices, outcomes, and outcomes measurement. Studies were also inMEASURES USE D IN ATD OUTCOMES RESEA RCH
eluded that were cited in review articles or in the selected study articles , but not identified through th e origina l electronic search. Coding Syllabus
The coding syllabus used in this st udy was ada pte d from a form at used by Dijkers et al. (2002) to evalu ate th e medical reha bilitation lit erature. The coding form used to ana lyze each article was expanded to includ e several demograph ic cha racteristic s: th e study's country of origin; age , disability populati on, and type of ATD used by study participants; data source and level; and context of data collection . In addition, our form includes three categories of validity an d three categories of administrative burden (vs. one category of each used in th e Dijker s et al., 2002, form) in order to evalua te an article's reporting of its measuremen t tools. The appropriateness of the psychometric and administra tive evaluation categories is born e out by th eir repeated men tion in articles assessing the meri ts of clinical measurement tools (Andresen, 2000; J ohnston et al., 1992; Law, 1987). For ana lysis of an individua l article, all re ported outcome variables, and their corres ponding measur ement tools, are recorded on the coding form. The reporting of reliabili ty, validity, and administrative burd en for each of the measurement tools is evaluated on a 3-point scale. For example, the scoring options for interrater reliability are the following:
o=
Th e authors did not p rovide evide nce of thi s psychom etric property; 1 = Th e authors provided evide nce of thi s p sychometric property ; houieoer, applica tio n of th is da ta to the current st udy is questio nable; and 2 = Th e aut hors p rovided evide nce of this psychom etric property that appears applicable to this par tic ular st udy appl ication.
Each outcome variable is also class ified into one of seven domain s: usabili ty, use, user satisfaction, QOL, role par ticipati on, functiona l level, and cost. The conceptual bas is for these domains is described in the following section. Domains of Outcome
Outcome domains re prese nt the objects of measur ement . In effect, they sha pe the conte nt of measure ment tools (Keith, 1984). A number of conceptu al domain s are commonly associated with ATD outcomes measurement . Consistent with recen t discussions (Fuhrer et al., 2003; Lenker & Paqu et , 9
2004), the following domains are the focus of this article. Device Usability
Product usability is not a property that resides within a device. Rather, usability emerges from interactions between the user, device, and environment during task performance. Usability is comprised of effectiveness, efficiency, and user satisfaction (International Organization for Standardization, 1993). Examples of usability indicators include device usage; physical and cognitive effort associated with use; comfort or pain associated with use; environments of use; safety; benefits of use ; reasons for nonuse; relative advantage of an ATD compared with alternative devices or personal assistance options; user satisfaction with devices; task completion; time required for task completion; errors committed during performance; level of assistance required; quality of performance (technique, skill); and quality of the end product (Lenker & Paquet, 2004). User Satisfaction
Although subsumable under usability as defined above (International Organization for Standardization, 1993), user satisfaction was coded separately here because of its prominence in the ATD outcomes literature. Consistent with Hulin's definition (1977), ATD satisfaction can be understood as being users' cognitive, affective, and evaluative responses (on a like-dislike dimension) to the perceived characteristics of ATDs and their impacts. Some satisfaction measures seek user judgment of global satisfaction with devices, whereas others seek satisfaction ratings about specific device features or byproducts of device interaction (e.g., comfort, ease of use, and reliability). Quality of Life
QOL impact is perhaps the most important outcomes indicator from the ATD user's perspective. One could make the case that all outcomes variables are, in fact, indicative of some slice of life quality. QOL domains have been variously characterized in terms of health-related QOL, health status, quality-adjusted life years, material status, social relationships, and subjective well-being (Dijkers, 1999; Fuhrer, 2000; Gill & Feinstein, 1994; Keith , 1994; Renwick & Brown, 1996; Ryff, 1995). Consistent with the analysis and recommendations of Fuhrer (2000) and Gill and Feinstein (1994), QOL was coded exclusively in terms of respondents' subjective well-being. 10
Social Role Performance
Although regarded by some to be an aspect of QOL, social role performance was coded separately because of its centrality in gauging the impacts of rehabilitation (Keith , 1995) and assistive technology (Gitlin, 1998). Roles related to family , work , and community define an individual's existence and shape the composition of their daily activities (Dijkers, Whiteneck, & EI-Jaroudi, 2000). Measures of social role participation include activity patterns, location of residence, employment, education (e.g., academic standing, workload , or writing quality), users' overall goals , and unmet needs. Functional Level
This domain includes indices bearing on discharge destination, functional status, capacity, assistance, independence, hours without assistance, and productive activity (Keith, 1994, 1995). Cost
This domain is typically measured in terms of costs associated with devices, services, reduced assistance, and deferred health care. Cost may be monetized (e.g., costs of devices, practitioner services, or overall health care ) or expressed in terms of surrogates such as caregiver time. Procedure
Initially, eight candidate articles were reviewed by the first three authors in order to verify the completeness of our inclusion and exclusion criteria. Next, the same three authors coded three articles in parallel in order to evaluate the coding instructions, uncover limitations in usability of the coding form, refine the coding categories, and develop interpretations that would affect coding. As an example of the latter, some articles reported use of measurement instruments in the Methods sections , but did not subsequently report the corresponding data in the Results sections. For our purposes, it was agreed that dependent variables would be coded only if they were described in an article's Methods section and reported in the Results section. In other instances, the Methods section described data being collected at an ordinal level, whereas the Results section reported nominal data. For the latter cases, it was decided that the level of measurement would be coded based on the method of data measurement. These and other consensus interpretations were incorporated into a second revision of the coding form and syllabus. To further sharpen the coding process, five adASSISTIVE TECHNOLO GY, VOL. 17, NO.1
TABLE 1. Country
United States United Kingdom Can ada Sweden Australia Th e Netherlands Belgium Germany Italy Total
Country of origin
TABLE 2.
Modal disabling condition of the study samples
No. of articles (%) 46 (56) 12 (14 ) 8 (10 ) 6 (7) 4 (5)
3 (4) 1 (1 )
1(1)
Disabling condition
Neurologi c/neuromuscular Orthopedic Vision Learning disability Mental retardation Unable to determine Total
No. of articles ( %) 35 (42) 9 (11) 7 (9 )
3 (4) 3 (4)
25 (30)
82 ( 100)
1 (1 ) 82 (100 )
Year of Publication
ditional articles were coded independently by the same three individuals. For the demographic factors associated with each article (e.g., country of origin, disability population, AT device, data collection context), the preponderance of ratings were in unanimous agreement, with two thirds or greater agreement in all cases. Similarly, there was near unanimity for the ratings of reported reliability, validity, and administrative burden of tools used in each study. Identification of the outcome variables elicited somewhat greater diversity . There was two thirds or greater agreement for 25 of 42 dependent variables that were reported in the five articles. In seeking to understand the reasons underlying areas of disagreement, the raters discussed each case and found that they generally deferred to the rationale presented by the first author. For example, the first author had identified 14 of 17 dependent variables that were recorded by only one rater. Additional discussion among the three raters revealed coding insights that resulted in a final revision of the coding form and syllabus. The authors were comfortable that the coding process had reached the level of readiness necessary for the analytic phase of this review. Using the final versions of the coding form and sylla bus, the first author read and coded all articles meeting the inclusion criteria. The coding form and coding syllabus are available from the first author and are viewable at http://www. atoutcomes.com/. RESULTS
The inclusion criteria were met by 82 articles (Appendix) involving a total of 212 ATD outcome variables. An additional 102 articles were read but not coded because they accorded with the exclusion criteria. MEASURES USED IN ATD OUTCOMES RESEARCH
The number (n = 52) of ATD outcome articles published in the 1990s was more than three times the number (n = 15) in the 1980s . Output (n = 15) in the first 2 years of the current decade is on pace to surpass the total published in the 1990s . Country of Origin
The majority of studies (56%) were conducted in the United States (Table 1) , with the rest based in Canada, Europe, and Australia. Sample Characteristics
The study samples in each article were evaluated in terms of three characteristics: disabling condition, type of ATD used, and age group. Disabling Condition
Nine categories of disabling condition were used to classify the studies' participants. For each article, all disabling categories represented in the sample were recorded. The most frequently occurring impairment category (i.e., the modal impairment category) was also recorded. Table 2 summarizes the number of articles for which each impairment category was the modal category. The largest category by far was neurologic/neuromuscular impairments. The modal impairment category could not be determined for 30% ofthe articles due to imprecise descriptions of the proportions of disabling conditions present in sample populations. Type of ATD
Nine categories were used to categorize the ATDs that were studied. All of the ATDs that were used in a given study were recorded, as was the ATD type that was most frequently associated with the study sample (i.e., the modal device category ). Table 3 summarizes the number of articles 11
TABLE 3.
Modal ATD associated with the study samples
ATD ECU Computer access, non-AAC AAC Self-care Seat cushion Manual wheelchai r Mobility aid, nonwheeled Power wheelchair Unabl e to determine Total
Frequency of domains measured by AT outcome variables No. of outcome variables (% )
No. of articles (%) 10 10 8 8
(12 ) (12) (10 ) (10 )
5 (6) 4 (5) 4 (5) 2 (2)
31 (38) 82 (100 )
Note: ATD = assistive technology device; ECU = environmental control unit; AAC = augmentative and alternative communication .
for which various ATDs were the modal device category. The modal device category could not be determined for 38% of the studies because insufficient information was presented to determine the article's dominant device type. Age Group
Three categories were used to record the mix of age groups present in each study: children (birth to age 17 years), adults (ages 18 to 65 years), and elders (older than age 65 years ). Although studies were excluded if the participant sample included only children, coding accommodated studies with samples featuring children and adults and/or elders. For a plurality (n = 35; 43%) of studies, the adult category predominated. For 21 studies (26%), the modal age category could not be determined because insufficient information was presented to make that determination. Homogeneity of Sample Characteristics
Other characterizations of the study samples are as follows: • 63% (52 of 82) of the studies concerned two or more categories of disabling condition; • 62% (51 of 82) of the studies featured a sample drawn from two or more age categories; • 65% (53 of82) of the studies concentrated on two or more ATD categories; • only 10% of the studies concerned participants whose disability and age fell into a single category and who used a single category of ATD; • 21% (17 of 82) of the studies were homogeneous for two of those three characteristics, and 39% (32 of 82) were homogeneous for one of them; and 12
TABLE 4.
Domain Usability Use User satisfaction Functional level Quality of life Role part icipation Cost Total
71 49 24 24
(34) (23) (11) (11)
17 (8) 16 (8) 11 (5)
212 (100 )
Note: AT = assistive technology.
• 30% (25 of 82) of the studies used a sample that included multiple categories of disabling condition and age, and who used multiple categories of ATDs. ATD Outcomes Domains
The 82 studies included a total of212 dependent variables relating to the impacts of ATDs being investigated. The outcome variables were categorized into one of seven different domains, which are summarized in Table 4. The domains of ATD usability, usage, user satisfaction, and functional level comprised approximately 80% of the variables identified. Usability was assessed in terms of a variety of factors : ease or difficulty of use , comfort or pain associated with use , environments of use , safety, task effectiveness and benefits of use , reasons for nonuse, and relative advantage compared with various options . Use was most often evaluated dichotomously (e.g., used/not used ), but was also measured in terms of frequency of use , hours used per day, and total number of devices used. User satisfaction was measured in terms of overall satisfaction with ATDs or ratings of specific device features. QOL variables accounted for 8% of identified outcome variables and included measures of health-related QOL and subjective wellbeing . Social role participation, functional level, and cost each accounted for less than 10% of identified outcome variables. Origin of Measurement Tools
Of 212 outcome variables reported in the 82 articles, 44 (21%) variables were measured using previously published measurement tools. One hundred sixty-eight (79%) were measured using study-specific instruments (i.e., tools that were deASSISTIVE TECHNOLOGY, VOL. 17, NO. 1
TABLE 5.
Data sources for outcome variables
Dat a so u rce
No. of outcome variables gathered using thi s source (%)
1. ATD user 2. Clinicia n 3. Caregiver or familiar othe r 4. Electromecha nical ins t ru ment 5. Medical records 6. Unclea r or not reported Combina tion of I , 2, a nd/or 3
Total
149 (70) 28 (13) 3 «3) 2 « 3) 5 « 3) 10 (5)
TABLE 6.
Data levels of outcome variables No. of outcome variables (% )
Data level
Qualitati ve Open-ended Nominal Ordinal Ratio Unable to det ermine
4 (2) 12 (6) 67 (32) 68 (32)
37 (17) 24 (11)
212 (100 )
Total
15 (7) 212 (100 )
Note : ATD = ass istive technology devices .
veloped to serve the purposes of the particular study). Source of Data
Table 5 summarizes the sources of data used to capture the 212 outcome variables. One hundred forty-nine (70%)were obtained via participant selfreport, and 28 (13%) were based on clinician observation and rating. Fifteen variables (7%) were measured using a combination of user, clinician, and/or caregiver report, with the latter two serving as a proxy for participant-reported data. Most of the clinician-rated variables represented mea sures of Acti vity and Parti cipation (World Health Organization, 2001), for example, as assessed by the Functional Independence Measure (Uniform Data System, 1993) and OT FACT (Smith, 2002). Data Level
Tabl e 6 summarizes the level of data represented across the 212 identified variables. Almost two thirds of the vari able s were measured with either nominal or ordinal data. Approximately 12% of the variables were not defined clearly enough to iden tify th e data level.
Psychometric Evidence
Very few authors offered adequate evidence supporting the reliability or validity of the tools being used to collect data (Table 8). Evidence of test-retest reliability information was not provided in 92% of the cases, and evidence of interrater reliability information was not presented in 88% of them. No evidence of validity was presented in similarly large proportions: content validity (90%), criterion validity (93%), and construct validity (92%). Administrative Burden
As indicated in Table 8, most articles offered no information regarding the staff time required for learning to use the tools (98%), administering the tools to research participants (85%), or scoring the tools (> 99%). DISCUSSION
The "typical" ATD outcomes study published in the past 20 years used a sample population that was diverse in terms of age , disability population, and type of ATD being used ; measured user-reported dependent variables with instruments designed specifically for the study; and did not report the psychometric or administrative properties of TABLE 7.
Data Collection Locus or Modality
Six categories described the locus or modality of data collection used in each study. Table 7 summari zes the frequency with which each was used as the primary data-collection context. Data were collected most often in community settings (40%), such as participants' homes. Mail surveys (21%), clinically based studies (21%), and phone surveys (9%) comprised most of the remainder. MEASU RES USE D IN ATD OUTCOMES RESEAR CH
Principal locus or modality of data collection
Method
Community site Clinical site Mail surv ey Phone Population survey Unable to determine Total
No. of articles ( %) 33 (40) 17 (21) 17 (21) 7 (9) 4 (5) 4 (5)
82 (100)
13
TABLE 8.
Presence of evidence for eight psychometric and administrative properties of outcome measures
(n
Type of ps ychometric /administrative evidence
Test-ret est reliability Interrater reliability Cont ent validity Criterion valid ity Const ru ct validity Training required to administer tool Time required to administer t he tool Time required to score the tool
No report, no. (%) 195 (92) 187 (88) 191 (90) 196 (93) 196 (92) 207 (98) 181 (85) 211 (99)
the measurement tools, regardless of whether they were previously published or study-specific. The studies' sample populations exhibited substantial heterogeneity, characterized by merging a number of distinguishable subgroups (e.g., categories of age , disabling condition, or type of ATD used ) and not differentiating the findings in terms of those groups. Such heterogeneity has been mentioned by previous reviewers (DeRuyt er , 1995; Fuhrer, 2001; Smith, 1996), but it has not previously been quantified. Sample heterogeneity is desirable in studies hypothesizing that an intervention is robust across a range of subgroups. However, it poses hazards for exploratory studies that lack such hypotheses, because it allows the results to be confounded by factors that have neither been identified nor controlled. The effect is to muddle interpretation of the findings and limit their contribution to a cohesive body of evidence-based literature (J ongbloed, 1990). A preponderance of the studies' measurement tools had not been published independently and lacked supporting documentation describing their developmental history, psychometric properties, and suggested applications (J ohnston et al., 1992). The tools did not travel the traditional developmental pathway in which a "deliberately converging strategy" is used to build a progressively stronger case for their reliability, validity, and applicability (Keith, 1984). In many cases this was probably due to the immaturity of the AT outcomes research area that forced researchers to develop their own measures. Very few articles reported adequate information on the reliability, validity, or administrative requirements for the instruments that were used. Some authors credited past research demonstrating reliability without specifying the category of 14
= 212)
Some report, but questionably or incompletely applicable to the study, no. (%) 10 (5) 11 (5)
Satisfactory presentation of evidence sup po r ting this characteristic, no. (%) 7 (3 )
7 (3)
14 (7) 14 ( 7 )
6 (3 )
10 (5)
7 (3)
1 « 1) 17 (8) 1 « 1)
9 (4 ) 4 (2 )
14 (7) 0(0)
reliability, strength of reliability, or the applicability of the reliability data to their current study. Likewise, others credited past research for demonstrating validity without describing the nature of validity , the strength of su pporting evidence, or the contexts for which the measure is valid (J ohnston et al., 1992). In a very few instances, low re porting of psychometric and administrative characteristics may have represented a failure to report information that was actually available from previous reports by the instruments' developers. In most cases, however, it can be as sumed that the information was simply none xistent. The former explanation perhaps reflects the youth of the field, which is still developing a collective expertise through qualified training and mentorship. Serious questions arise from using psychometrically unproven instruments. Genuine treatment effects may go undetected if the reliability of measures is actually weak. Tools lacking validity can result in systematic underestimation, overestimation, or misrepresentation of treatment effects . Neglect in reporting psychometric background data is not unique to the AT field. Dijkers et al. (2002) found similar results in their evaluation of medical rehabilitation research literature published in the late 1990s. A scant few articles discussed the pragmatic issues associated with learning, administering, and scoring measurement tools . The absence ofthat information hampers researchers wishing to replicate those studies or to use their measures in another context. The high percentage of outcome variables measured with user-reported data reflects three things: (a) researchers, clinicians, and government funding agencies have shown great interest, and rightly so, in capturing the end user's perspective ASSISTlVE TECHNOLOGY, VOL. 17, NO.1
regarding ATD impact; (b) the domains of device usage , device usability, and QOL impact are most easil y captured through user self-report; and (3) there are very few options for measuring ATD usage and usability characteristics. The generalizability of our findings is qualified for at least two reasons. First, non-English language articles were excluded. Second, article selection focused on research involving specific types of ATDs. RECOMMENDATIONS FOR DEVELOPMENT OF MEASUREMENT DOMAINS
Measurement tools can only be successful to the exte nt that their underlying conceptual domains are fully developed and described (Dijkers et al. , 2002; Frytak, 2000; Keith, 1995). With that in mind, several suggestions are offered as partial guidance for the future conceptual and methodological development of ATD outcome measures. Device Usability
The AT field needs a dialogue about the measurement of usability dimensions. Four themes ar e suggested below as starting points. Uniform ity, Where Possible
The field should seek a consistency of usability domains and item-scaling that would be appropriate acros s age groups, reimbursement models, disability populations, and , most importantly, types of ATDs. The diversity of devices seems to necessitate some device-specific usability domains and mea sures thereof. For example, factors reflecting th e usability of a manual wheelchair differ from tho se associated with using an augmentative communication device. Notwithstanding the need for device-specific usability measures, a concerted attempt should be made to establish a common set of usability domains and measures that can then be employed for evaluating the outcome s of particular classes of ATDs. Measuring Use
Usage has often been measured dichotomously in terms of use and nonuse (or use and abandonment). Th e field lacks an explanatory theory for the mechanism underlying ATD use and nonuse (J utai, 2002), perhaps partly due to the imprecise manner in which use has been measured. Additional dimensions of device usage, if captured more systematically, could lend inferential clues . Frequ ency of us e can be measured in terms of number MEASURE S USED IN ATD OUTCOMES RESEARCH
of days used per week or per month. Duration of us e can be measured in terms of minutes or hours per day (Weiss-Lambrou, Tremblay, LeBlanc, Lacoste , & Dansereau, 1999). Environments of use can reveal the breadth of ATD impact in terms of social (Murphy, Markov, Collins , & Moodie, 1996), community (Angelo, Jones, & Kokoska , 1995; Trefler & Crislip, 1985), or treatment (DeRuyter & Kennedy, 1990) contexts of use. Delineating th e task s for wh ich a device is used (e.g., writing reports, entering data, or composing e-mail) would add depth to characterizing the usage of computerbased ATD. These dimensions have appeared sporadically in the ATD outcomes literature and ha ve been measured inconsistently, making it difficult to compare usage data across studies (Rogers & Holm, 1992). A related discussion is also needed regarding interpretation of usage data (Gitlin , 1998). The common inference has been that device usage is "good" and that nonusage is "bad." Although this may describe a subset of situations, there are numerous studies (Clemson & Martin, 1996; Cushman & Scherer, 1996; DeRuyter & Kennedy, 1990; Garber, Bunzel, & Monga , 2002; Geiger, 1990; Phillips & Zhao, 1993; Wielandt & Strong, 2000 ) showing that users have "abandoned" ATDs because their condition had improved such that they performed satisfactorily without the device. Among individuals who continue to use a device, it remains to be established empirically whether frequent users achieve better outcomes than infrequent ones . Also, most studies do not explore users' unmet needs or if they are using devices to full capacity (Cowan & Turner-Smith, 1999). In order to interpret usage data more meaningfully, it is necessary to identify what constitutes "expected or optimum device use " (Kraskowsky & Finlayson , 2001), including circumstances of nonuse that do not merit pejorative interpretation (Rogers & Holm, 1992). Measuring Objecti ve Aspects of Usability
Historically, there have been few options for objectively measuring the interaction between ATD and end user. A number of performance-oriented tools are in development, however. COMPASS (Koester et al. , 2003) is a software program that measures speed and errors associated with keyboard and mouse performance. Several companies market sophisticated pressure-mapping systems for measuring the pressure distribution effectiveness of wheelchair cushions, and several tools are being developed to measure functional mobility and skills of wheelchair users (Kirby, Swuste, Du15
puis, MacLeod, & Munroe, 2002; Stanley, Stafford, Rasch, & Rodgers , 2003). The Language Acquisition Monitor (Romich & Hill, 1999) is a data logging system, embedded within an AAC device, which captures user utterances over time. Although there are privacy issues associated with data collection systems that are integrated within ATDs, embedded data acquisition methods represent an area of opportunity for the field. Development of these and other techniques is needed in order to measure performance in a manner that complements the well-established tradition of obtaining user-reported data. Measuring Subjective Aspects of Usability
Subjective aspects of ATD usability are tapped by the Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST; Demers, Weiss-Lambrou, & Ska, 2000), Psychosocial Impact of Assistive Devices Scale (PIADS; Day & Jutai, 1996), and the Device Section of the Assistive Technology Device Predisposition Assessment (ATD-PA; Scherer, 1998c). Each offers strengths with respect to measuring subjective aspects of ATD usability, but it would be worthwhile to consider development of a single measure that captures the breadth of the usability construct with a minimized number of items. Physical and cognitive effort , esthetics, hassles, learnability, dependability, discomfort, speed, accuracy, and reasons for continued use (or nonuse) are a few of the indicators that might be included in that measure or others. Quality of Life
A variety of questions confront the developers of future instruments intended to assess the impacts of ATDs on users' QOL. Should the QOL domains be consistent with those specified by the International Classification of Functioning, Health, and Disability (ICF ), for instance, body structure and functioning, activity and participation, personal factors , and environmental factors (World Health Organization, 2001)? Should some non-ICF domains be tapped as well, and if so, from what other taxonomy of domains should they be drawn? Should respondents rate their status in QOL domains, or should they rate the ATD's impact on their QOL status? Should a premium be placed on the brevity of the item set? If so, using item-response theory to develop the items is an especially attractive strategy. It is a set of statistical models for analyzing multiple questionnaire items that measure the same concept (Hambleton, Swami16
nathan, & Rogers , 1991; Van der Linden & Hambleton, 1997). The PIADS (Day & Jutai, 1996) is a 26-item scale that has been used to measure the impact of ATD on the subjective well-being of diverse ATD users (Day, Jutai, Woolrich, & Strong, 2001; Demers, Monette, Lapierre, Arnold , & Wolfson, 2002; Jutai & Gryfe, 1998; Jutai, Rigby, Ryan , & Stickel, 2000). The ATD-PA (Scherer, 1998c; Scherer & Cushman, 2001) includes a subscale that corre lates positively with the Satisfaction with Life Scale (Diener, Emmons, Larsen, & Griffin , 1985). Few measurement alternatives exist to assess the subjective well-being of children or people with cognitive disabilities. Social Role Performance
Multistakeholder dialogue is needed to identify appropriate objective indicators of role performance that might vary with ATD usage. At a minimum, the areas covered should include the family, employment, education, and the community. Given the methodological difficulties associated with conducting longitudinal studies with low incidence populations, it would be especially valuable to identify indicators of short-term ATD impacts. Areas of "small gain" may not necessarily relate to increased functional independence, but nonetheless carry personal value and meaning as part of an individual's "role enactment" (Gitlin, 1998). CONCLUSIONS
AT outcomes research is challenging because of the diversity of ATDs, the breadth of ages and disabilities that characterize ATD users, and the transactional nature of ATD use in everyday situations (Light, 1999), including work , school, leisure, and self-care activities. The context-dependency of ATD usage distinguishes ATD outcomes research from parallel research in physical medicine and rehabilitation, which generally measures outcomes under relatively controlled clinical environments (DeRuyter, 1997; Fuhrer, 2001; Smith , 1996). A number of additional factors make it especially difficult to characterize ATD outcomes that occur as a result of routine service provision. They include the small volume of consumers seen at most AT service delivery sites; inadequate reimbursement for evaluation, system configuration and integration, training, and follow-up; insufficiently standardized practices for service delivery programs; and the absence of uniform curriculum standards for professional preparation (DeRuyter, 1997). ASSISTIVE TECHNOLOGY, VOL. 17, NO.1
The above pragmatic alibi s not withstanding, th e field will not thrive ifthese challenges are not met. As Holm (2000) suggested to a national group of occupational therapists, clinical competency is imperiled unless a field develops the wherewithal to conduct re search that can be integrated into professional practice. Reimbursement for practitioner services and ATDs for end users will be threatened unl ess a more substantive demonstration is mad e supporting ATD effectiveness at reasonable levels of cost (DeRuyter , 1997). Based on the results of thi s study, we offer several recommendations to outcomes researchers, journal editors, review ers, and policy makers in order to elevate the quality of ATD outcomes research reporting. • Sample populations should be described as clearl y as possible in terms of age , impairment, and ATD being used. Authors should also characteriz e the delivery model under which participants received their AT in order to clarify the relationships between that intervention and its outcomes. Authors should also indicate, where possible , the length of time that participants have been using their ATDs. • The ra tionale for selecting each outcome instrument should be included. This would be helpful for clinicians and researchers who are consideri ng similar tools for their own use . • If an out come measure is developed anew for a particular investigation, study resources should be devoted to at least an initial assess ment of the mea sure's reliability and validity. Regardless of domain, new tools should strive for ease of implementation and be suitable for everyday use by practitioners (J utai, Ladak, Schuller, Naumann, & Wright, 1996). • Article s should provide information about the psychometric properties (e.g., reliability, validity) of each measurement tool. This may consist of references to appropriate publications, or the pr esentation of such data if they are otherwise un available. Authors should address the applicability of the psychometric data to the context of th e current study. If psychometric inform ation is nonexistent, authors should attach strong caveats to their findings , subject to establishing that information. Alth ough journal space is a precious commodity, authors can mak e study-specific tools and supporting documentation available from electronic repo sitories (i.e., web sites) to enhance interpretation of reported results, promote awareness of promi sing new tools, and facilitate replication studies. MEASURES USED IN ATD OUTCOMES RESEARCH
• Information should be provided about the workload asso ciated with using particular outcome instruments (e.g., the time required for administering and scori ng them ). Such information would add practical value for consumers of AT research lit erature (i.e., clinicians and researchers considering use of tho se tools in th eir own settings). • A simple checklist of mea surement and reporting principles based on publi shed sta nda rds (Andresen, 2000; Johnston et al. , 1992; Law , 1987) should be developed. A checkli st for design and interpretation of research studies, based on standards from the epidemiolo gy field, was developed for the ergonomics field (Heacock, Koehoorn , & Tan, 1997). • Governmental research prioritie s affect nextgeneration measurement practices. Agencies such as the National Institute of Disability and Rehabilitation Research and the National Center for Medical Rehabilitation Research need to assign higher priority to the development of ATD outcome mea sures for usability domains for which avail able mea sures are substandard. The domains requiring especially urgen t attention include usage, role participation, and QOL. Acknowledgments: Funding was provided in part by Grant H133A010401 from the Na tional Institute of Disability and Rehabilitation Research. The authors are all memb ers of the Consortium of Assisti ve Technology Outcome s Researchers (CATOR). The authors would like to thank Dr. Marcel Dijkers for sharing th e coding form and syllabus adapted for the purposes of this study. We are grateful to Dr. Louise Demers for her comments in respons e to an early dr aft of thi s manuscrip t. The first author (J .A.L.) extend s heartfelt appreciation to Crystal Yalch for gathering most of the articles reviewed her e. The technical assistance of Gordon Sumner facilitated completion of the manuscript.
REFERENCES Andresen, E. M. (2000). Criteria for assessing the tools of disability outcomes research. A rchi ves of Physical Medi cine and Rehabilitati on, 81(Sup pl. 2), SI5- S20. Angelo, D. H., Jo nes, S. D., & Kokoska, S. M. (1995). Fam ily perspective on augmentative and alternative communication: Families of young children. Augmentative and A lternati ve Comm unication, 11, 193- 202. Bedrosian , J . 1. (1999). AAC efficacy research: Challenges for th e new cent ury . Augmentative and A lternative Comm unicatio n, 15, 2-3. Clemson, 1. , & Mart in, R. (1996). Usage and effectiveness of rail s, bathing and toileting aids. Occupational Therapy in Health Care, 10(1),41-59.
17
Cowan , D. M., & Turner-Smith, A. R (1999 ). The user's perspective on the provision of electronic a ssistive technology: Equipped for life? Br iti sh Journal of Occupational Therapy, 62,2-6. Cus hman, L. A., & Scher er , M. J . (1996). Mea suring th e relati onship of assistive technol ogy use, function al status over tim e, and consumer -th erapist perceptions of ATs. A ssist ive Techn ology, 8, 103-109. Day, H., & Jutai, J . W. (1996). PIADS: The Psychosocial Impact of A ssistive Devices S cale. Toronto, Ontario, Canada: Authors . Day, H., Jutai, J . W., Woolrich , W., & Strong, G. (2001). Th e stability of impact of assist ive devices. Disability and Rehabil itat ion, 23 , 400-404. Demer s , L., Monette, M., Lap ierre, Y , Arnold, D. L., & Wolfson, C. (2002). Reliability, validity, and applicability of the Quebec User Evaluation of Satisfaction with Assistive Tech nology (QUEST 2.0) for adults with multiple sclerosis . Disability and Rehab il itat ion, 24, 21- 30. Dem er s, L., Weiss-Lambrou, R , & Ska, B. (2000). Quebec User E valuation of Satisfaction with A ssisti ve Technology (QUES T version 2.0): An outcome measure for assisti ve devices. Webst er , NY: In stitute for Matching Person & Technology. DeRu yt er , F. (1995). Evaluating outcomes in ass istive technology: Do we und erstand th e commitment? Assisti ve Technology, 7, 3-8. DeRuyt er , F. (1997). The importance of outcome measures for assistiv e techn ology service delivery sys te ms . Technol ogy and Disability, 6, 89- 104. DeRu yter , F. (1998). Concepts and rational e for accountability in assist ive technology. In L. Vitaliti (Ed.), RESNA resource guide for assist ive technol ogy outcomes:Measurement tools (Vol. 1, pp. 2-14). Arlington, VA: RESNA. DeRuyt er , F., & Kennedy, M. R (1990). Augmentative communi cation followin g brain inj ury. In D. R Beukelman & K. Yorkston (Eds.), Communication disorders following traumatic brain injury. Austin, TX: Pro-Ed. Diener , E. , Emmons, R, Larsen , R , & Griffin , S. (1985). The satisfaction with life scale. Journal of Personal ity Assessment , 49, 71-75. Dijker s, M. (1999). Measuring quality of life: Methodological iss ues. American Journal of Physical Medicine and Reha bil itation, 78, 286-300 . Dijkers , M., Kropp , G. C., Esper , R M., Yavuzer, G., Cullen, N., & Bakd alieh , Y. (2002). Reporting on reliability and validity of outcome measures in medical rehabilitation research. Disab ility and Rehab il itat ion, 24, 819- 827. Dijker s, M., Whiteneck, G. G., & El-Jaroudi, R (2000). Measures of social outcomes in disability research. Archives of Physical Med icine and Rehab ili tat ion, 81(12, Suppl. 2), S563-S580. Frytak, J . (2000). Mea su remen t. Journ al of Rehabilitation Outcomes Measurement , 4, 15-31. Fuhrer , M. J . (2000). Subjectifying quality oflife as a medical rehabilitation outcome. Disability and Rehabilitation, 22, 481-489. Fuhrer , M. J . (2001). Assistive technology outcomes re search: Cha llenges met and yet unmet. A merican Journal of Physical Medi cine and Rehab ilitation, 80, 528-535. Fuhrer , M. J ., Jutai , J . W., Scher er , M. J ., & DeRuyter, F. (2003). A fram ework for the conceptual modeling of assi sti ve techn ology device outcomes . Disability and Rehabilitation, 25, 1243-1251. Garber , S. L., Bunzel, R , & Monga , T. N. (2002). Wheelchair utilization a nd satisfaction following cerebral vascula r ac-
18
cident. Journal of Rehab il itation Research and Develop· ment , 39, 521-534. Geiger , C. (1990 ). Th e utilization of assistive devices by patients discharged from an acute reh abilitation setti ng. Physical & Occupational Th erapy in Geriatrics, 9, 3-25. Gelderbloom, G. J ., & de Witte, L. P. (2002). Specia l issu e: Th e assessm ent of assist ive techn ology outcomes effects and cost s. Techn ology and Disabili ty, 14, 89- 145. Gill, T. M., & Feinst ein , A. R (1994 ). A criti ca l appraisal of th e quality of quality-of-life measurem ents. JAM A, 272, 24-31. Gitlin, L. N. (1998). The role of social science research in und erstanding techn ology use among older adults. In M. G. Ory & G. H. DeFri ese (Eds .), Self-care in later life: Research prog ram and policy issues (pp. 142-169). New York: Springer. Gitlin, L. N. (2002). Assisti ve techn ology in th e home a nd community for older people : Psychological and socia l consi derations. In M. J. Scherer (Ed.), Assistive techn ology : Match ing device and cons umer for successful rehab il itation (pp, 109-122). Wash ington, DC: Ameri can Psych ological Association. Gray, J . A. M. (1997 ). E vid ence-based health care: How to make health policy and management decisions. New York : Churchill-Livings tone . Hambleton, R K., Swaminathan, H., & Roger s, J . (1991). Fundam ent als of item response theory. Newbu ry Park, CA: Sage. Heacock , H ., Koeh oorn , M., & Ta n, J . (1997). Applyin g epidemiological principles to ergonomics: A checklist for incorp orating sound design and interpret ation of studies. Applied Ergonomi cs, 28(3) , 165-172. Holm , M. B. (2000). Our mandate for th e new millennium : Evidenc e-based practice. American Journal of Occupationa l Therapy, 54, 575-585. Hulin, C. L. (1977 ). Satisfaction and othe r job a ttit udes. In B. B. Wolman (Ed.I, Int ernati onal encycloped ia of psychiatry, psychology, psychoanalysis & neurology (Vol. 10, p. 12). New York: Aesculapius . International Organization for Standardizati on. (1993 ). ISO CD 9241 -11.2. In Ergonom ic requirements for office work with visual displa y term in als (VDTs): Guidan ce on usabili ty specificat ion and measures. Genev a , Switze rland: Author. Johnston, M. V., Keith, R A., & Hinder er , S. R (1992 ). Mea sureme nt st andards for interdi sciplin a ry medical reh ab ilitation . Archives of Physical Medicine and Rehabil itation, 73(Suppl.), 3-23. Jongbloed , L. (1990 ), Prob lem s of methodological het er ogen eity in studies predicting disability a fter stro ke . Stroke, 2 1(Su ppl. II-32-II-34. Jutai, J . W. (2002). Occupation al th erapy a nd assistive technology: The res earch chall enge. Israel Journ al of Occupa tional Th erapy, 11(1), E3-E22. Jutai, J. W., & Gryfe, P. (1998 ). Imp act s of ass istive tec hnology on clients with ALS. Proceedings of th e RESNA Annual Conference, 18, 54-56. Jutai, J . W., Ladak, N., Schuller , R , Naumann, S., & Wright, V. (1996). Outcomes measurem ent of ass istive techn ologies: An ins tit utional case study. A ssistive Techn ology, 8, 110120. Jutai, J . W., Rigby, P., Ryan, S., & Stickel, S. (2000). Psychosocial impact of electronic aids to da ily living . A ssistive Techn ology, 12, 123-131. Keith, R A. (1984). Functional assessm ent mea sures in medical rehabilitation : Cu rre nt status . Archi ves of Physical Med icine and Rehab ilitation, 65, 74-78. Keith, R A. (1994). Function al health and heal th stat us . Ar -
rn.
ASSISTIVE TE CHNOLOGY, VOL. 17, NO. 1
chive s of Physical Medicin e and Rehabilitation, 75, 478483 . Keith, R A. (1995 ). Conceptual ba sis of outc ome measures . American Journ al of Physical Medi cin e and R ehabilitation, 74, 73- 80. Kirby, R 1., Swu ste, J ., Dupuis, D. J., MacLeod, D. A., & Mu nroe, R (2002). Th e wheelchair skills test: A pilot st udy of a new outcome measu re. Ar chi ves of Physical Medi cine and Reh abilitat ion, 83 , 10-18. Koeste r, H. H., LoPresti , E ., Ash lock, G., McMill an , W., Moor e, P ., & Si mpson, R (2003, March ). Compass : So ftwa re for comp uter skills assessment. Pap er pr esented at th e CSUN 2003 International Con fere nce on Technology and Per son s wit h Disab ilities, Los Angeles , CA. Kraskowsky , 1. H., & Finlayson , M. (2001). Fac tors affecting olde r a dults ' use of ass ist ive devices : Review of th e literature . A merican Journ al of Occupati onal Th erapy, 55, 303310. Law, M. (198 7). Measurem en t in occupatio na l th erapy: Scienti fic crite ria for eva luatio n. Cana dian Journ al of Occup atio nal Therapy, 54(3), 133-138. Lenker , J . A., & Paquet , V. L. (2004). A new conceptual model for ass istive techn ology out comes resear ch and practice. Assisti ne Techn ology, 16, 1-10 . Light, J. (1999). Do a ugme ntati ve and alte rnative communication interv entions reall y make a difference?: Th e challenges of efficacy research. Augment ative and Al ternati ve Com mun icatio n, 15, 13-24. Murphy, J ., Ma rk ov, 1., Collins, S., & Moodie, E. (1996 ). AAC systems: Obs ta cles to effective use. Eu ropean Journal of Disorders of Com munication, 3 1,3 1-44. Pap e, T. 1. B., Kim , J., & Wein er , B. (2002). Th e shaping of individua l mean ing s assigned to assist ive techn ology: A review of per son al factors. Disability an d Rehabilitation, 24, 5-20. Ph illips, B., & Zhao , H. (1993 ), Predictor s of ass ist ive techn ology abandonme nt . A ssist ive Techn ology, 5, 36-45. Renwic k, R , & Brown , 1. (1996). Th e Cent re for Health Promoti on's conceptu al approach to th e qualit y oflife. In R Renwick , 1. Brown , & M. Nag ler (E ds.), Qual ity of life in health promotion and rehab il itati on: Conceptual approaches, is sues , and applications. Th ousand Oaks , CA: Sage. Rogers , J . C., & Holm , M. B. (1992). Assi stive technology device use in pati ents wit h rh eumatic disease: A lit er ature review. Am erica n J ourn al of Occup ational Th erapy, 46, 120-127. Romic h , B. A., & Hill , K J. (1999 ), A language act ivity monitor for AAC a nd writing sys te ms : Clinical intervent ion, outcomes measurem ent, and research. Proceedings of the RE S NA '99 A nnual Conference, 19- 21. Ryff, C. D. (1995 ). Psychological well-being in adult life. Cur rent Directions in Psychologi cal Sc ience, 4(4),99- 104. Scherer, M. J . (1996 ). Outcomes of assist ive techn ology use on quality of life. Disabil ity and R ehabilit ation, 18,439-448. Sche rer, M. J . (1998a ). Chara cte ris tics of a meaningful outcome assessme nt. In RES NA resource g uide for assist ive tech nology outcomes: Measurement tools (Vol, I, pp. 50-59). Ar lingto n , VA: RESNA Press. Scherer, M. J . (1998b). Th e impact of assist ive techn ology on th e lives of people with disabili t ies. In D. B. Gray, 1. A. Qu atrano, & M. L. Lieberman (Eds.), Design ing and using as sistive technology : Th e human perspective. Bal timore, MD: Pau l H. Brookes. Scherer. M. J . (1998c ). Matching person & technology. Webste r , NY: In stitute for Matching Person & Techn ology. Sche re r, M. J ., & Cus hman , 1. A. (2001). Mea suring subje ctive qu ality of life followin g s pinal cord injury: A validat ion study
MEASURE S USED IN ATD OUT COMES RESEARCH
of th e Assistive Techn ology Device Predisposition Assessment. Disabili ty and R ehabilitation, 23, 387-393 . Smith, R O. (1996). Measuring th e outcomes of assist ive tec hnology: Challe nge and innova t ion . Assistive Techn ology, 8, 71-81. Smith, R O. (1998 ), Accountability in assisti ve tec hnology inter ven tions: Mea su rin g outcomes. In L. Vitaliti (Ed.), RES NA resource g uide for assist ive technology outcomes: Mea suremen t tools (Vol. 1, pp . 15-42). Arlington, VA: RES NA Pr ess . Smi th , R O. (2002). OT FACT : Multi-level perform anc e-oriented softw are wit h an assisti ve te chnology outcomes assessme nt pr otocol. Techn ology an d Disability, 14, 133-139. Sm ith, R 0 ., & Sch er er , M. J . (1998). Wher e are we head ed with ass ist ive technology outcomes? In L. Vitaliti (Ed.), RESNA resource g uide for assisti oe techno logy outcomes: Measu rement tools (Vo\. I, pp . 60-67). Arlingt on , VA: RESNA Press. Stanley, R K , Sta fford, D. J., Rasch , E., & Rodger s, M. (2003). Developm ent of a funct iona l assessme nt mea sure for man ual wh eelch air user s. Journ al of Rehabilitation R esearch and Developm ent, 40(4 ),301-308. Tr efler , E. , & Crislip, D. (1985 ). No aid , an Etran , a Min sp eak: A comparison of efficiency and effect iveness du ring st ru ctu red use . Augm entative and Alte rnative Comm un ication , 1, 151- 155. Uni form Data System. (1993 ). Guide for th e un ifor m da ta set for medi cal rehabilitation (Adult FIM J, versio n 4.0. Buffa 10: State Uni ver sity of New York . Van der Lind en , W., & H amblet on , R K (1997). Han db ook of modern item response theory. New York : Spri nge r. Weiss-Larnbrou, R , Tremblay, C., LeBl an c, R , Lacoste, M., & Dansereau, J . (1999). Wheelch air se at ing aids : How sat is fied are consum er s? Assi stive Technol ogy, II , 43-53. Wielandt, T., & St rong , J . (2000l . Compliance wit h pr escrib ed ada pti ve equipment: A literature review. Br it ish Journ al of Occupational Th erapy, 63(2 ),65-75. World Health Organiza t ion. (2001). ICF: Int ern at ional classi fication of fun ctioning, disability and health . Genev a , Switzerl and: Author.
APPENDIX
Research Articles Included in the Coded Sample Aaronson , D., & Gabias , P. (1987). Compute r use by th e visually imp aired . Beha vior Resear ch Methods, In strument s, & Comp uters, 19, 275-282. Agr ee, E. M. (1999). Th e influ en ce of per sonal care and assisti ve devices on th e mea surem en t of disability. Social S cience & Medi cine, 48, 427-443. Allen , S. M., Foster , A., & Ber g, K (2001). Receiving help at hom e: The interplay of human and techn ological ass istance. Journal of Gerontology: Socia l Scie nces, 56B, S37 4-S382 . Andrich, R , Ferrario, M., & Moi, M. (1998 ). A model of costoutcome analysis for assist ive technology. Disab ilit y an d Rehab ilitation, 20 , 1-24. Barnes, M. P . (1994). En vir onmental con trol sys te ms : An audit and review. Cli nical Rehabilitat ion, 8, 362-366. Ber gen , A. F. (1996 ). Th e effect of th e mobil ity assessme nt pro cess on outcomes : A beginning effort. Technology and Disability, 5, 17- 23. Brooks, N. A. (1991). Use rs ' responses to assistive devices for physical disab ility. S ocial Scie nce & Medicin e, 32 , 14171424.
19
Bryen , D. N., Slesaransky, G., & Baker, D. B. (1995). Augmentative communication and empowerment supports: A look at outcomes. Augmentative and Alternative Communication , II , 79-88. Buning, M. E., Angelo, J ., & Schmel er , M. R (2001). Occupational performance and the transition to powered mobility. Am erican Journal of Occupat ional Therapy, 55, 339-344. Buning, M. E., & Hanzlik, J . R (1993). Adaptive computer use for a person with visual impairment. Am erican Journal of Occupational Th erapy, 47, 998-1008. Caudrey, D. J ., & Seeger, B. R (1983). Rehabilitation engineering service evaluation: A follow-up survey of device effectiveness and patient acceptance. Rehabilitation Literatur e, 44, 80-85. Cowan , D. M., & Turner-Smith, A R (1999). The user's perspective on th e provision of electronic assistive technology : Equipp ed for life? British Journal ofOccupat ional Th erapy, 62,2-6. Croser, R , Garrett, R , Seeger , B., & Davies, P. (2001). Effectiveness of electronic aids to daily living: Increased independence and decreased frustration . Australian Occupational Therapy Journal, 48 (1), 35-44. Culp, D. M., Ambrosi , D. M., Berniger, T. M., & Mitchell, J . O. (1986). Augmentative communication aid use : A follow-up study. Augm entative and Alternative Communication, 2, 19-23. Cushman, 1. A., & Scherer, M. J . (1996). Measuring the relationship of assisti ve technology use, functional status over tim e, and consumer-therapist perceptions of ATs. Assistive Technology, 8, 103-109. Demers , 1. , Monette, M., Lapierre, Y., Arnold , D. 1. , & Wolfson, C. (2002). Reliability, validity, and applicability of the Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST 2.0) for adults with multiple sclerosis . Disability and Rehabilitation, 24, 21-30. Demers, 1. , Ska , 8. , Giroux , F., & Weiss-Lambrou, R (1999). Stability and reproducibility of the Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST). Journal of Rehabilitation Outcomes Measurement , 3(4), 42-52. Demers , 1. , Wessels , R , Weiss-Lambrou, R , Ska , 8., & De Witte , 1. P. (2001). Key dimensions of client satisfaction with assistive technology: A cross-validation of a Canadian measure in The Netherlands. Journal of Rehabilitation Medi cine, 33, 187-191. Dowden, P., Beukelman, D. R , & Lossing , C. (1986). Serving nonsp eaking patients in acute care settings: Intervention outcomes. Augm entative and Alternative Communication, 2, 38-44. Edward s, N. I., & Jones, D. A. (1998). Ownership and use of assistiv e devices among st older people in the community. Age and Ageing , 27, 463-468. Efthimiou, J ., Gordon, W. A., Sell, G. H., & Stratford, C. (1981). Electronic assistive devices: Their impact on the quality of life of high level quadriplegic persons. Archives of Physical Medicine and Rehabilitation , 62, 131-134. Forbes, W. F., Hayward, L. M., & Agwani, N. (1993). Factors associated with self-reported use and non-use of assistive devices among impaired elderly residing in community. Canadian Journal of Public Health, 84, 53-57. Garber, S. 1. , Bunzel, R , & Monga , T. N. (2002). Wheelchair utilization and satisfaction following cerebral vascular accident. Journal of Rehabilitation Research and Development, 39, 521-534. Garber, S. 1. , & Gregorio, T. L. (1990). Upper extremity assistive devices: Assessment of use by spinal cord-injured pa-
20
tients with quadriplegia. Am erican Journal of Occupation al Th erapy, 44, 126-131. George, J ., Binns, V. E., Clayden , A. D., & Mulley, G. P. (1988). Aids and adaptation for the elderly at home : Underprovided, underused, and undermaintained. British Medical Journal, 296, 1365-1366. Gitlin, 1. N., Luborsky, M. R , & Schemm, R L. (1998). Emerging concerns of older stroke patients about assistive device use. Gerontologist, 38, 169-180. Gitlin ,1. N., Mount, J ., Lucas, W., Weirich, 1. C., & Gramberg, L. (1997). The physical costs and psychosocial benefits of travel aids for persons who are visually impaired or blind. Journal ofVisual Impairment and Blindness, 91, 347-359. Gitlin , L. N., Schemm , R L., Landsberg, L., & Burgh, D. (1996). Factors predicting assi stive device use in the home by older people following rehabilitation. Journal of Aging and Health , 8, 554-575. Goodman, G., Tiene , D., & Luft , P. (2002). Adoption of assistive technology for computer access among college students with disabilities. Disability and Rehabilitation, 24, 80-92. Hamm el, J ., Lai, J ., & Heller, T. (2002). The impact of assistive technology and environmental interventions on function and living situation status with people who ar e ageing with developmental disabilities. Disability and Rehabilitation, 24, 93-105. Harmer, J ., & Bakheit, AM. O. (1999). The benefits of environmental control systems as perceived by disabled user s and their carers. Briti sh Journal of Occupational Th erapy. 62, 394-398. Hartke, R J ., Prohaska, T. R , & Furner, S. E. (1998). Older adults and assistive devices. Journal of Aging and Health, 10(1),99-116. Hass, U., Ander sson , A , Brodin, H., & Per sson, J . (1997). Assessment of computer-aided assistive technology : Analysis of outcomes and costs. Augmentative and Alternative Communication , 13, 125-135. Hass, U., Brodin, H., Andersson, A., & Perrson, J. (1997). Assistive technology selection: A study of participation of users with rheumatoid arthritis. IEEE Transa ction s on Rehabil itation Engineering, 5, 263-275. Hass, U., Persson, J ., Brodin, H., Freden-Karlsson, I., Olsson, J ., & Berg, 1. (1995). Assessment of rehabilitation technologies in stroke: Outcomes and costs . Int ernational Journal of Technology As sessment in Health Care, 11, 245-261. Haworth, R J. (1983). Use of aids during the first three months after total hip replacement. British Journal of Rheumatology, 22, 29-35. Haworth, R J ., & Hopkins, J . (1980). Use of aids following total hip replacement. British Journal of Occupational Th erapy, 43, 398-400. Hesse , S., Gahein-Sama, A. L., & Mauritz, K. H. (1996). Technical aids in hemiparetic patients: Pr escription, costs, and usage. Clini cal Rehabilitation, 10, 328-333. Higgins , E. 1. , & Raskind, M. H. (1995). Compensatory effectiven ess of speech recognition on the written compositi on performance of post-secondary students with learning disabilities. Learning Disability Quarterly, 18, 159-174. Higgins, E. L., & Raskind, M. H. (1997). The compensatory effectiveness of optical character recognition/speech synthesis on reading comprehen sion of post-second ary students with learning disabilities. Learning Disabilities, 8, 75-87. Jinks, A , & Sinteff, B. (1994). Consumer response to AAC devices: Acquisition , training, use , and satisfaction. Augm entat ive and Alternative Commun ication , 10, 184-190 . Jutai, J . W., Rigby, P., Ryan , S., & Stickel , S. (2000). Psycho-
ASSISTIVE TECHNOLOGY, VOL. 17, NO.1
social impact of electronic aid s to dail y living. Assistive Techn ology, 12, 123-13I. Ket tl e, M., Rowley, C., & Chamberlain, M. A. (1992). A national survey of wheelchair users. Clini cal Rehabilitation, 6(1 ), 67-73. Kohn , J . G., End er s, S., Preston, J ., & Motloch , W. (1983). Provision of assisti ve equipment for handicapped per sons. Arch ives of Physical Medicine and Rehabilitation, 64, 37838 I. Kohn, J . G., LeBlan c, M., & Mortol a, P. (1994). Measuring quality and performance of assistive technology: Results of a pr ospecti ve monitoring pr ogram. A ssist ive Techn ology, 6, 120-125. Mann , W. C., Hurren , D., Charvat, B., & Tomita, M. (1996 ). Probl ems with wheelchairs experienced by frail eld ers. Technology and Disability, 5, 101-11I. Mann, W. C., Hurren , D., & Tomit a , M. (1995). Assistive devices used by home-based elde rly per sons with a rt hritis. A merican Journ al of Occupational Th erapy, 49, 8 10-820. Mann, W. C., Hurren , D., Tomita, M., & Charvat, B. (1997 ). A 2-yea r study of coping strategies of home-based frail elder s with vision imp airment. Techn ology and Disabil ity, 6, 177189. Mann , W. C., Karuza, J ., Hurren, D., & Tomita, M. (1993). Needs of home-based older per sons for assistive devices. Techn ology and Disability, 2, 1-11. Mann, W. C., Ottenb ach er , K J ., Fraas , 1. , Tomita, M., & Gra nge r, C. V. (1999). Effecti vene ss of assist ive techn ology and envi ron me nta l interv entions in maintaining independence and redu cing hom e care costs for th e frail elderly: A randomized controlled tri al. Archives of Family Medicine, 8, 210-2 17. McDonald , D. W., Boyle, M. A., & Schumann, T. 1. (1989). Environmental control un it util izati on by high -level spina l cord injured pati ents. Archives of Phy sical Medicine and Rehabilitat ion, 70, 621-623. Murph y, J. , Markov, 1., Collins, S., & Moodie, E. (1996). AAC systems: Obst acles to effective use. Eu ropean Journal of Disorders of Communication, 3 1,31-44. Par ette, H. P., & Van Biervli et , A. (1992). Tentative findings of a st udy of th e te chn ology needs and use patterns of persons with mental retardation. Journal of In tellectu al Disability Research, 36(1 ), 7-27. Pa rette, H. P. , Van Bier vliet , A., & Holbro ok, M. C. (1990). Tech nological needs of visually imp aired a nd blind per sons in Ark an sas. Journ al of Visual Impairment and Blindness, 84, 534-538. Parker, M. G., & Thorslund, M. (1991). Th e use of technic al aids among community -bas ed elderly. American Journal of Occupational Th erapy, 45, 712-718. Pell , S. D., Gillies, R. M., & Ca rss, M. (1997). Relationship between use of techn ology a nd employme nt rates for people with physical disabiliti es in Australia: Impl icat ions for education and training programmes. Disabil ity and Rehab il ita tion, 19, 332-338. Pell, S. D., Gillies, R. M., & Ca rss, M. (1999). Use of technol ogy by people with physical disabilities in Aus tralia . Disability and Rehabili tation, 2 1, 56-60. Phillips , B., & Zha o, H. (1993). Predictor s of ass istive techn ology aba ndonment. Assistive Techn ology, 5, 36-45. Post , M. W. M., van Asbeck, F. W. A., van Dijk, A. J ., & Schri jve rs, A. J . P. (1997). Ser vices for spina l cord inj ure d: Availability and sa tis faction. Spinal Cord, 35, 109-115. Pr an gr at, T., Mann, W. C., & Tomita, M. (2000). Imp act of un ilat er al neglect on assistive device use. Techn ology an d Disabili ty, 12, 53- 69.
MEASURE S USED IN ATD OUTCOMES RESEARCH
Raskind, M. H., & Higgin s, E. L. (1995). Effects of speech synth esis on the proofread ing efficiency of post-second ary st udents with learning disabilities. Learn ing Disabil ity Qua rterly, 18, 141-158. Riemer-Re iss, M. 1. , & Wacker , R. R. (2000). Factors associ ated with assistive techn ology discontinuanc e among ind ividuals with disabilities. Journ al of Rehabilitation, 66, 4450. Roelands, M., Van Oost , P., Depoorter, A., & Buysse, A. (2002). A social-cogniti ve model to pr edict th e use of assistive devices for mobility and self-care in elde rly people. Geronto logis t, 42( 1), 39-50. Rowley, C. M., St owe, J ., Bryan t , J. , & Cha mberlain, M. A. (1988). Communication aids pr ovision 1983-1986. British Journal of Disorders of Comm unication, 23, 1-12. Scherer, M. J . (1990). Assisti ve device utili zati on a nd quality of life in people with spina l cord injuries or cer ebr al palsy two yea rs later. Journ al of Applied Rehabil itat ion Coun seli ng, 2 1(4),36-44. Scherer , M. J ., & Cus hma n, 1. A. (2000 ). Pr edicting sa tis faction with ass istive techn ology for a sa mple of a dults wit h new spina l cord injuries. Psychological Reports, 87, 981987. Sh aw , G. (1991). Wheelchair seat comfort for the institutionaliz ed elderly. A ssisti ve Techn ology, 3, 11- 23. Sh aw , G., & Ta ylor, S. J . (1991). A survey of whee lchai r seating pr oblems of the inst itu tionalized elderly. Assistive Technology, 3, 5-10. Shell, D. F., & Horn, C. A. (1991). Compensatory computer technol ogy for disabled college studen ts: Applications a nd an evaluation of student use, sa tisfaction, a nd aca demic out comes. Journal ofPostseconda ry Edu cation & Disability, 9, 282-290. Sonn , U. (1996). Longitudinal st udies of dependence in daily life acti viti es among elderly per sons. Scand inavian Journ al of Rehabil itati on Medicine, 34(Su ppl.l , 1-35. Sonn , U., Davegardh, H., Lind skog., A. C., & Ste en , B. (1996 ). Th e use and effectiveness of assistive devices in a n elderly urb an popul ati on. Ag ing : Cli nical and Experim ental Research, 8, 176-183. St eele, R. D., Goodri ch, G. L., Henn ies , D., & McKinley, J. A. (1989 ). Reading aid technology for blind persons: Responses to a que st ionnaire of experie nced user s. Assistiv e Techn ology, 1, 23-30. Stickel, M. S., Ryan , S., Rigby, P. J ., & Jutai , J . W. (2002). Towa rd a comprehe ns ive evaluation of the imp act of elect ronic aid s to dail y living: Evaluati on of consumer sa tisfactio n. Disabil ity and Rehab ilitation, 24, 115-1 25. Symington, D. C., Lywood, D. W., Lawson , J . S., & MacLean, J . (1986 ). En vir onm ental control systems in ch ronic care hospitals and nu rsing homes. Archives of Physical Medicine and Rehab ili tat ion, 67, 322-325. Tolley, K , Leese, B., Wright, K , Henn essey, S., Rowley, C., Stowe, J ., & Cha mberla in, A. (1995 ). Communication aids for th e speech impaired: Cost a nd quality-of-life out comes of assessme nt pr ogram s provided by specialist comm unic ati on aid s centers in th e Unite d Kingdom. Int erna tional Journa l of Techn ology Assessment in Health Care, 11, 196-213. Trefler , E., & Crislip, D. (1985 ). No aid, a n Etran , a Min speak: A compa rison of efficiency and effectiveness during structured use. Aug ment ativ e and Alternative Communication, 1, 151-155. van del' Heide, A., J acobs , J . W. G., van Albada-Kuipers, G. A., Kraaimaat, F. W., Geenen , R., & Bijlsma, J . W. J . (1993 ). Self-report functi onal disabil ity scores a nd th e use of de vic-
21
es: Two distinct aspects of physical function in rheumatoid arthritis. Annals of the Rheumatic Diseases, 52, 497-502. Verbrugge, L. M., Rennert, C., & Madans, J. H. (1997). The great efficacy of personal and equipment assistance in reducing disability. A merican Journal of Publ ic Health , 87, 384-392. Weiss-Lambrou, R., Tremblay, C., LeBlanc, R., Lacoste , M., & Dansereau , J . (1999). Wheelchair seating aids : How satisfied are consum ers? Ass istive Technology, 11, 43-53.
22
Wright, P., Rogers, N., Hall , C., Wilson, B., Evans , J., & Emslie, H. (2001). Enhancing an appointment dia ry on a pocket computer for use by people aft er brain inju ry. Int ernat ional Journ al of Rehabilitat ion Research , 24, 299-308. York, J . (1989). Mobility method s selected for use in home and community environments. Physical Th erapy, 69, 736-747. Zimmer , Z., & Chappell, N. L. (1994). Mobility restriction and the use of devices among seniors. Journal of Agi ng and Health, 6, 185-208.
ASSISTIVE TECHNOLOGY, VOL. 17, NO.1
