Conducted in Spanish or English; translators used as ... 5 11 1 2 79% 7% 14% 6 2 0 75% 25% 0%. A 22. 6 9 2 3 63% 14% 21% 6 0 2 75% 0% 25%. Filipino.
Three Methodologies to Improve Question Development for Different Cultural and Linguistic Groups Presentation to the American Association for Public Opinion Research May 2010 Susan Sprachman, Sally Atkins-Burnett, Nikki Aikens, and Margaret Caspe
Research was conducted as part of the Universal Preschool Child Outcomes Study (UPCOS), a descriptive study of the Los Angeles Universal Preschool Program (LAUP).
Special thanks to our clients at First 5 LA for giving us the opportunity to do some very interesting methodological work as part of our contract.
2
Do Behaviors Have Same Meaning for Different Groups?
3
Background of UPCOS
LAUP serves a diverse population.
Among the 10 million residents of Los Angeles County, an estimated 224 languages are spoken.
Latinos are the majority of the population—but some are recent immigrants and others have lived in LA for generations and speak English as their first language.
Likewise, Asians come from a variety of countries and have varying degrees of acculturation.
4
Descriptive Study Design Tasks
Go beyond the standard measures used in other studies of preschoolers.
Ensure that the proposed battery is appropriate for the diverse population of LA County.
Try to increase the reliability of parent report measures of social-emotional development.
5
What Do We Mean by Social-Emotional Development?
Increase in: Empathy Independence Self-regulation Ability to relate to others Coping with challenges Decrease in: Aggression Disobedience Sadness
6
Design Questions
1. Which of the standard parent report measures was the best choice for our study?
2.
What types of items are most problematic?
3. How could we adapt items or measures to increase reliability?
7
Social-Emotional Measures
Developed by clinicians as clinical tools
Often used as part of survey research projects to assess aspects of development that cannot be easily observed in a single point in time
Typically completed by both a parent and teacher with very few, if any, definitions for the terms used
8
Different Interpretations by Teachers and Parents
CHILD stands up for his rights.
9
Possible Sources of Bias
Wording of items: Must be understandable across groups
Meaning of items: Should be similar across groups
Response scales: Must not introduce bias for some groups but not others
Translations
10
Basic Methodologies
1. Card sort 2. Focus groups 3. Cognitive interviewing
11
Methodology #1: Card Sort
Six groups
Participants: 57 parents of preschoolers from 5 cultural /linguistic groups
Parents were not connected with LAUP
Conducted in Spanish or English; translators used as necessary
12
Format of Cards
51 items from 5 standardized rating scales or developed by the researchers
Items printed on 2"x4"-size cards
Dice groserías a otros niños Calls people names
Cards were either English only or bilingual English/Spanish
13
2
Card Sort Instructions
―… You can help us by telling us if a question is confusing, if there are any words that are unclear or inappropriate, and if this is the way that you would ask the question if you were talking with other parents in the program or with your friends.‖
Respondents sorted the cards into envelopes labeled:
Questions that are easy to answer Questions that are confusing and/or hard to answer Questions that you are uncomfortable answering
Percentage of Cards Sorted into Different Categories Percentage
90
79
75 62
61
60
56
60
31
30 17
22
24
20
20
16 9
11
15
12 9
0
Easy
16
Hard
Uncomfortable
Example of Item with Divergence Among Groups
"Acts impulsively–uncomfortable answering"
57% of Latino-English speakers
56% of Chinese/Cambodian speakers
33% of African American speakers
10% of Latino-Spanish speakers
0% of Filipino speakers
17
Methodology #2: Focus Groups
Focus group participants were the same as the card sorters.
Parents discussed social skills and problem behaviors of importance and concern to them after they completed the card sort.
18
Focus Group Guided Discussion
Which cards did you feel strongly about one way or the other?
Which were hard to answer? Why?
Which questions made you feel uncomfortable? Why?
19
Some Items Were Hard to Understand for All Groups
"Is difficult to comfort when upset"
Issue: Is the problem how difficult it is for parent to comfort child or that the child has difficulty being comforted?
Resolution: Revised item and tested it in the cognitive interviews: "CHILD has difficulty calming down when upset."
20
Example: Stealing
One Filipino parent said she would feel uncomfortable talking about that because she doesn’t want to think about her child doing that sort of thing. However, one English-speaking Latino parent felt that some behaviors are really surprising at this age (stealing and so on). Decision: Do not include in final battery.
21
Methodology #3: Cognitive Interviewing Interviews were conducted in English or Spanish with parents of children in pilot study programs.
Parents completed a self-administered questionnaire with selected items from the social-emotional measure that seemed the best for the study.
We asked parents their rationale for some of their ratings.
We also asked them to give us examples of behaviors that caused them to choose a particular rating.
22
Example 1: ―Is anxious and tense‖
Problem: Some parents misinterpreted ―tense‖ as meaning hyperactive or muscle tension. Resolution: In the final parent rating scale, we adapted this item to ask about worrying excessively about things.
23
Example 2: ―Clings to parent or caregiver‖ Problem: Some parents interpreted this as a negative behavior, as it was intended.
Others interpreted it as a positive behavior: ―When I come home my child runs to me and wants to be with me.‖
Resolution – Tested and used 2 versions of the items: ―Holds onto a parent or caregiver when strangers are around‖
―Stays very close to a parent or caregiver in a new place‖ 24
Lessons Learned
We cannot assume that, because a respondent speaks the language we are using, he or she fully understands the concepts we are asking about.
We need to make sure that items have similar meaning across groups – particularly if translated.
We need to make sure that the clinical or technical meaning of an item on a scale remains in the adaptation or translation.
