Using Card Sorts, Focus Groups, and Cognitive Interviews to Improve ...

Three Methodologies to Improve Question Development for Different Cultural and Linguistic Groups Presentation to the American Association for Public Opinion Research May 2010 Susan Sprachman, Sally Atkins-Burnett, Nikki Aikens, and Margaret Caspe

Research was conducted as part of the Universal Preschool Child Outcomes Study (UPCOS), a descriptive study of the Los Angeles Universal Preschool Program (LAUP).

Special thanks to our clients at First 5 LA for giving us the opportunity to do some very interesting methodological work as part of our contract.

2

Do Behaviors Have Same Meaning for Different Groups?

3

Background of UPCOS 

LAUP serves a diverse population.



Among the 10 million residents of Los Angeles County, an estimated 224 languages are spoken.



Latinos are the majority of the population—but some are recent immigrants and others have lived in LA for generations and speak English as their first language.



Likewise, Asians come from a variety of countries and have varying degrees of acculturation.

4

Descriptive Study Design Tasks



Go beyond the standard measures used in other studies of preschoolers.



Ensure that the proposed battery is appropriate for the diverse population of LA County.



Try to increase the reliability of parent report measures of social-emotional development.

5

What Do We Mean by Social-Emotional Development?

Increase in:  Empathy  Independence  Self-regulation  Ability to relate to others  Coping with challenges Decrease in:  Aggression  Disobedience  Sadness

6

Design Questions

1. Which of the standard parent report measures was the best choice for our study?

2.

What types of items are most problematic?

3. How could we adapt items or measures to increase reliability?

7

Social-Emotional Measures



Developed by clinicians as clinical tools



Often used as part of survey research projects to assess aspects of development that cannot be easily observed in a single point in time



Typically completed by both a parent and teacher with very few, if any, definitions for the terms used

8

Different Interpretations by Teachers and Parents

CHILD stands up for his rights.

9

Possible Sources of Bias

 Wording of items: Must be understandable across groups



Meaning of items: Should be similar across groups



Response scales: Must not introduce bias for some groups but not others

 Translations

10

Basic Methodologies

1. Card sort 2. Focus groups 3. Cognitive interviewing

11

Methodology #1: Card Sort

 Six groups 

Participants: 57 parents of preschoolers from 5 cultural /linguistic groups

 Parents were not connected with LAUP 

Conducted in Spanish or English; translators used as necessary

12

Format of Cards



51 items from 5 standardized rating scales or developed by the researchers

 Items printed on 2"x4"-size cards



Dice groserías a otros niños Calls people names

Cards were either English only or bilingual English/Spanish

13

2

Card Sort Instructions

―… You can help us by telling us if a question is confusing, if there are any words that are unclear or inappropriate, and if this is the way that you would ask the question if you were talking with other parents in the program or with your friends.‖

Respondents sorted the cards into envelopes labeled:

 Questions that are easy to answer  Questions that are confusing and/or hard to answer  Questions that you are uncomfortable answering

14

1 2 3 4 5 6

9 12 12 11 11 9

15

21% 7% 14% 21% 14% 21%

6 7 6 4 6 6

1 0 1 3 2 0

13% 13% 13% 13% 0% 25%

Easy - %

Korean 1 75% 13% 1 88% 0% 1 75% 13% 1 50% 38% 0 75% 25% 2 75% 0%

Uncomfortable

Hard

Easy

Uncomfortable - %

Hard - %

Easy - %

Uncomfortable

Hard

2 1 0 0 1 2

Uncomfortable - %

31 26 04 39 15 22

Filipino 3 64% 14% 1 86% 7% 2 86% 0% 3 79% 0% 2 79% 7% 3 63% 14%

Hard - %

A A A A A A

Easy

Label

Card

How Data Were Compiled

Percentage of Cards Sorted into Different Categories Percentage

90

79

75 62

61

60

56

60

31

30 17

22

24

20

20

16 9

11

15

12 9

0

Easy

16

Hard

Uncomfortable

Example of Item with Divergence Among Groups

"Acts impulsively–uncomfortable answering"



57% of Latino-English speakers



56% of Chinese/Cambodian speakers



33% of African American speakers



10% of Latino-Spanish speakers



0% of Filipino speakers

17

Methodology #2: Focus Groups

 Focus group participants were the same as the card sorters.



Parents discussed social skills and problem behaviors of importance and concern to them after they completed the card sort.

18

Focus Group Guided Discussion



Which cards did you feel strongly about one way or the other?



Which were hard to answer? Why?



Which questions made you feel uncomfortable? Why?

19

Some Items Were Hard to Understand for All Groups

"Is difficult to comfort when upset"

 Issue: Is the problem how difficult it is for parent to comfort child or that the child has difficulty being comforted?



Resolution: Revised item and tested it in the cognitive interviews: "CHILD has difficulty calming down when upset."

20

Example: Stealing

One Filipino parent said she would feel uncomfortable talking about that because she doesn’t want to think about her child doing that sort of thing. However, one English-speaking Latino parent felt that some behaviors are really surprising at this age (stealing and so on). Decision: Do not include in final battery.

21

Methodology #3: Cognitive Interviewing  Interviews were conducted in English or Spanish with parents of children in pilot study programs.

 Parents completed a self-administered questionnaire with selected items from the social-emotional measure that seemed the best for the study.

 We asked parents their rationale for some of their ratings.

 We also asked them to give us examples of behaviors that caused them to choose a particular rating.

22

Example 1: ―Is anxious and tense‖

Problem: Some parents misinterpreted ―tense‖ as meaning hyperactive or muscle tension. Resolution: In the final parent rating scale, we adapted this item to ask about worrying excessively about things.

23

Example 2: ―Clings to parent or caregiver‖ Problem:  Some parents interpreted this as a negative behavior, as it was intended.



Others interpreted it as a positive behavior: ―When I come home my child runs to me and wants to be with me.‖

Resolution – Tested and used 2 versions of the items:  ―Holds onto a parent or caregiver when strangers are around‖

 ―Stays very close to a parent or caregiver in a new place‖ 24

Lessons Learned



We cannot assume that, because a respondent speaks the language we are using, he or she fully understands the concepts we are asking about.



We need to make sure that items have similar meaning across groups – particularly if translated.



We need to make sure that the clinical or technical meaning of an item on a scale remains in the adaptation or translation.

25