Engaging Students in Qualitative Research Through Experiential Class Activities Lisa Aronson Fontes School of Education University of Massachusetts, Amherst
Fred P. Piercy Human Development Virginia Polytechnic Institute and State University
Experiential activities help psychology students experience firsthand the richness and depth of meaning of qualitative data. Through these activities students grapple with pragmatic, philosophical, and ethical questions like those facing researchers in the field. These activities concern focus groups, observation, data collection, cultural sensitivity, ethnomethodology, data analysis, and ethics. In a final exercise students develop answers to the common challenges posed by skeptics of qualitative methodology. Instructors can select single exercises for broad psychology courses or use all of them, as we do, in a course dedicated to qualitative research in psychology. Social scientists increasingly rely on qualitative data as both an adjunct to quantitative data and a valid form of information on its own (Patton, 1996). Some psychology departments teach a separate qualitative research course, and others include qualitative research orientations in more traditional research methods courses. We teach a one-semester graduate course on qualitative research to help students know when qualitative approaches are most useful, have the skills to conduct rigorous qualitative studies, and know how to evaluate qualitative research—none of which are as intuitive as they might seem. In this article, we describe exercises that we use in our courses to convey the nature and variety of qualitative research in psychology, in the hope that other teachers of qualitative research will find them useful. Given the large number of approaches used in qualitative research, these exercises cover selected techniques only and are not exhaustive. Rossman and Wilson (1985) described the major ways of combining qualitative and quantitative data (words and numbers) as initiation, corroboration, and elaboration. Frequently, researchers begin their query by using qualitative methods to observe and describe the phenomenon of interest, which they then test empirically (initiation). Corroboration occurs when researchers bring together data collected through more than one method to determine whether there is convergence in the findings. Finally, one type of data can be used to elaborate on the findings of another, either lending strength to an argument or uncovering contradictions (elaboration). At the end of our course, many students express interest in conducting studies with both qualitative and 174
quantitative components. Others decide that one approach only fits better with the problems that interest them. When we teach qualitative methods, we try to highlight the advantages of qualitative research without constantly referring to its more established quantitative cousin. Different methods are valuable for different kinds of questions, and we hope our students will learn to judge study designs on their appropriateness for the problem—not on some a priori preference for one approach or another. Over the years, we have seen journals become more receptive to qualitative studies (Banyard & Miller, 1998), and we have encountered increasing numbers of students who hope to conduct at least one qualitative study—without necessarily connecting it to a quantitative study—during the course of their graduate education. For this reason, we have developed this graduate course dedicated exclusively to the methods and techniques of qualitative research. Our class meets for one 2½ hr session each week, which enables us to conduct at least one experiential exercise in each class. Instructors whose classes follow different schedules will have to adapt the exercises accordingly. Instructors may wish to use specific exercises described here in classes on research methods or other topics. For instance, teachers of health psychology courses can use the focus-group exercise to familiarize students with this common health psychology research technique. Similarly, teachers of community psychology courses may discover that the exercise exploring the cultural sensitivity of qualitative research fits well (Maton, Hrabowski, & Greif, 1998). Setting the Stage By the time they enter our graduate course in qualitative research, most students have been taught that their usual ways of knowing the world—through observation and conversation—are not suited to the academic study of psychology and that facts necessarily include numbers (Hubbard, 1988). They have learned that scientific knowledge may be obtained only through reductionist procedures and statistical manipulation and that rigor comes with large sample size, standardized validated instruments, and numerical measures of reliability and Teaching of Psychology
validity. Some students attend our classes with the suspicion that qualitative research is somehow weaker (or easier) than quantitative research. Immersion in the challenging exercises described here relieves students of these notions. Techniques that actively involve students facilitate learning (Older, 1979). We begin our courses in qualitative research with a series of demonstrations of the value of nonnumerical information to help counteract some of the “the truth is only in the numbers” messages that our students have received in prior psychology classes. For instance, we ask them how they chose their program of study (their comments inevitably include “impressions” and discussions with students and faculty but rarely include statistics). We ask them how employers choose among job candidates. Again, students remind us that interviews are usually paramount. Finally, we ask them if they would trust any standardized battery of instruments sufficiently to choose their life partner based solely on the results. By this point, students begin to see how much they rely everyday on primitive qualitative inquiry and usually express a greater openness to learning about how to make qualitative research more rigorous. In the first class, we sometimes ask each student to rate a teacher, department, or job on a 5-point scale ranging from 1 (very poor) to 5 (excellent) and explain their answer to a classmate. Then, in the large group, we ask for reactions to the two types of data (rating and verbal explanation). Usually, the discussion highlights the depth, complexity, and contextual information that become available in descriptions as compared to the breadth afforded by numerical ratings. Also, at the beginning of the semester we ask class members to introduce themselves by telling their name and describing a turning point in their lives. We then discuss the power of this information and how qualitative research can capture that power and convey it directly to readers. In this way, even skeptical students begin to see the value of qualitative data. Using Focus Groups to Introduce Focus Groups Psychology researchers frequently use focus groups to determine community needs, obtain group interview data, and evaluate proposed research protocols and mental health services. We want our graduates to know how to conduct focus groups, either as a major source of data or as an adjunct to other methods. In this exercise, students gain experience conducting or participating in a focus group. After a short introduction to the definition and history of focus groups (Krueger, 1994), we ask for enough volunteers for each to be a leader of a minifocus group of four or five class members. The purpose of the focus group, we tell them, is to explore components of effective teaching. We then briefly tell the volunteers about their role as focus group leaders (e.g., present ground rules, put the group at ease, encourage different opinions, bring the group back to the topic at hand). Our instructions take no more than 10 min. The volunteers then use a question guide that we give them (available from authors) to lead their minifocus groups. We stop the groups after about 25 min, although they have not yet discussed all the questions. We ask for personal reactions and answer questions that arose from the experience (e.g., “What do I do Vol. 27, No. 3, 2000
if someone doesn’t talk?” or “Is it OK if one group member asks another a question?”). We have found that this firsthand experience makes students more receptive to our minilectures that follow on the uses, strengths, weaknesses, and mechanics of focus groups (Krueger, 1994; Piercy & Nickerson, 1996). Students understand the advantages and limitations of focus group interviews and choose to conduct studies using focus groups with greater frequency after learning about them in this course. Observing and Writing Field Notes As part of their training in the social sciences, students learn to draw inferences. Although drawing inferences may be helpful at the data analysis stage, this is a habit to keep at bay while gathering data. Students need to learn to differentiate between their observations and inferences. For example, students trained in the rudiments of clinical psychology who see a couple sitting on opposite ends of the couch with their arms crossed are likely to conclude that there is marital conflict; if they see a child with bruises, they may suspect child abuse; and if they see someone who is frowning, with downcast, tearful eyes, they may conclude that this person is depressed. Although these inferences may be correct and may be useful in a clinical situation, they can interfere with the students’ ability to simply describe what they see. In this exercise, we encourage students to become sophisticated nonjudgmental observers without interpreting or drawing conclusions. First, we have them compare the following two reports on the same person: 1. A rich man got onto the elevator. He looked like he was rushing to a job interview. 2. The first person to step into the elevator was a bearded brown-haired man who wore a dark blue suit, white shirt, and shiny shoes. He clutched a polished leather briefcase to his chest and looked at his watch twice while on the elevator. After pressing the button for the eighth floor, he examined his reflection on the metal elevator wall and straightened his tie. He shifted from foot to foot and squirted breath spray into his mouth. Obviously, the writer of the second report described without inferring. This, we tell them, is what we want them to do in the observation exercise. We give students 15 min in which they are to stand or sit in a place where they can see people (e.g., in a hallway, in front of an elevator, under a tree) and simply observe. We tell them to interact with people at the place as little as possible. No more than two students from the class can be at the same place. We ask them to write down what they observe, sticking as close to actual description as possible. After they return to the classroom, we ask them to write field notes for 10 min about the experience. We instruct them to include both practical information (e.g., where they conducted the observation, for how long, their reactions to it) and their feelings about it (e.g., “I was nervous about what people would think about my just standing there”). We ask several 175
students to read the field notes about their experiences to the entire class. They usually describe the observation task as both much harder and much richer than they anticipated. We then have several volunteers read their observation descriptions aloud. We talk about what makes a description vivid and how to choose which details to record at the moment and which to save for the field notes. We also spend some time talking about the “self” issues that emerge in an observation (e.g., feeling awkward, feeling like joining in a conversation, feeling affinity for some people and enmity for others). After this exercise, student reports show greater attention to relevant details in their observations. They also demonstrate increased awareness of their use of self in research and an improved ability to describe without editorializing. Experiencing Various Methods of Data Collection In this class activity, students learn about different types of qualitative data and gain practice collecting data. We organize a focus group discussion on an engaging topic and then assign five groups to observe the discussion and collect five different types of qualitative data (focus-group discussion guide available from authors). We discuss how these data might be used in a qualitative research study. First, we ask for five or six female volunteers who are willing to take part in a short (about 30 min) minifocus group on the subject of gender bias (another topic may be chosen, but we have found this one generates lively discussions). We then divide the rest of the class into five teams and arrange the minifocus group in the center as a fishbowl with the five observing teams on the outside. The first of the five teams observes the focus group discussion and inductively derives observer-constructed categories that emerge as the discussion proceeds (i.e., they are to take notes on the discussion and then group the topics that emerge in ways that seem to make sense). The second team observes and identifies the indigenous typologies (Patton, 1990, p. 393) that emerge (e.g., the speakers themselves may describe three categories of sexist bosses as “the kind of boss who stares at your breasts,” “the kind of boss who can’t take your work seriously,” and “the kind of boss who’s always asking you to make coffee”). The third team writes field notes in which they describe the setting and maintain a general record of the interaction dynamics and the focus-group topics. The fourth team writes a reaction/feeling diary of their reactions as the focus group discussion unfolds. The fifth team writes metanotes, field notes focusing on the process of the learning activity itself. After the focus-group discussion and a brief follow-up discussion by the members about their experience, each of the observing teams meets separately to discuss what they recorded individually. Each observing team picks a spokesperson who summarizes for the class their team’s experience and the specific data that their team collected. We then lead a discussion on the various types of data and how these might be used in an article on this focus-group study. Instructors may also want to take advantage of this opportunity to discuss how the researchers’ relevant identity groups (in this case, men or women) might affect the data-gathering and analysis process. After this exercise, students demonstrate 176
improved ability to differentiate among the various categories of information researchers can gather during a study, which makes their research proposals more precise. Cultural Sensitivity of Qualitative Research As a field, psychology is increasingly concerned with developing methods that address humanity’s cultural diversity (Matsumoto, 1996). In this exercise, students see how qualitative methods are well suited for studying cultural issues because participants have opportunities to elaborate on the meanings behind their cultural practices. In qualitative studies, participants have an opportunity to establish their own categories and place emphases where they wish. People from all cultural groups habitually engage in the most common methods of data gathering used in qualitative research: observation and conversation. We share with students our observation—obtained through fieldwork in Latin America, Asia, and among immigrants in the United States as well as our reading of the crosscultural literature—that researchers often err when they assume that the same quantitative instrument can be translated and used crossculturally. Written instruments with predetermined categories, particularly when they are used alone, might miss the mark or seem foreign or alienating to people from different cultures (Fontes, 1997; Loos, 1995). In this exercise, we ask the class how they might learn about affection toward children in an ethnic subculture in their community. They usually describe observing families and conducting interviews with their members, individually or collectively. We then discuss the advantages and disadvantages of following up on this initial fieldwork by asking open-ended qualitative questions (e.g., “How do you show your child you’re proud of him or her?”) or quantitative questions (e.g., “How many times a day do you hug your child?” “On a scale of 1–5, how important is it to hug your child?”). As long as they are implemented in a culturally sensitive manner, neither method is better; they simply provide different kinds of information. If there is sufficient time in the course, students implement this exercise with a small number of participants. This exercise leads students to incorporate attention to culture into their research. Ethnomethodology We want our students to understand the implicit cultural assumptions that shape their and others’ world views. In this way, we hope they will be able to conduct research that acknowledges and explores different cultural positions. The following ethnomethodology exercise provides a fine vehicle for this lesson. Ethnomethodologists study ordinary, routine, everyday behavior that is usually invisible because people take it for granted (e.g., ethnomethodologists might consider what a Martian would need to learn about flirting to function appropriately at a high school dance). To access unspoken norms, ethnomethodologists often conduct ethnomethodological experiments that disrupt normal routines by doing something out of the ordinary (Garfinkel, 1967). We ask students to choose one of the subsequent tasks or to propose one of their Teaching of Psychology
own and write a short paper describing their reactions to conducting the out-of-the-ordinary experiment, identifying the tacit knowledge or rules that might have surfaced. Possible ethnomethodology experiments include the following: 1. Get into a crowded elevator and stand with your back to the door. 2. Challenge someone to a game of tic-tac-toe. When the other person marks an X or O, erase it and put it in another place, then proceed with your own mark. 3. Stand within 6 in. of someone you know slightly and begin a normal conversation. 4. Start singing in a public bus. 5. Introduce yourself and shake the hands of strangers in a line to buy movie tickets. 6. Talk to the fruits or vegetables in a supermarket. 7. Wear pajamas all day as you go about your regular activities. Students should discuss all proposed experiments with the instructor or the entire class because students sometimes come up with ideas that are inappropriate or ethically questionable. These discussions provide fruitful ground for more general conversations about research ethics and acquaint students with the importance of peer review and prior approval. As an example of an ethnomethodological investigation, one of our students got approval both from us and the manager of a supermarket to sample various foodstuffs (e.g., drinking from a milk carton) prior to buying the items. The manager agreed as long as the student subsequently paid for the items he sampled. The student was surprised both at how hard it was for him to engage in this mildly taboo activity and also at the strength of the stares of reprobation that he encountered from other shoppers. We have found these exercises to be useful, engaging ways to sensitize our students to implicit rules and norms, leading them to conduct research that takes into consideration their own and their participants’ implicit cultural biases. Data Analysis As the major assignment in the course, students conduct and write up a small-scale qualitative study. Students often use this assignment as an opportunity to test an approach for their thesis or dissertation. A student who expects to interview 12 clinical psychologists and supervisors for a dissertation, for example, might interview 1 of each for this project. To gain practice in data analysis, we have a student bring in two pages of an interview transcript from this project, disguising identifying information and leaving plenty of room in the right margins. (Students volunteer eagerly to bring in data because it gives them a head start in analyzing data for their project.) We distribute copies of the transcript to everyone in the class. We then ask the students to do a line-by-line reading of a page, individually listing all of the themes they can identify. After 10 min, we ask them to break into small groups and compare notes. As a class, we list all of the themes identified by each group. We have found that the themes identified often vary widely. We then ask the students to reVol. 27, No. 3, 2000
turn to their annotated transcripts and process the full two pages this time, determining which themes could appropriately be grouped into more abstract or larger categories (e.g., anger, helplessness, and frustration could be grouped under the heading “feelings about the work”). We compare notes again as a class and discuss the integrity of the categories, what is lost and gained by collapsing subcategories, which subcategories should be split into still smaller categories, and so on. This exercise accomplishes a number of objectives. First, it gives students the hands-on experience of analyzing a transcript. They are able to see how themes emerge from the data, and at the same time it becomes clear how readers’ perspectives affect the themes they identify. They also learn about the process of working with others on data analysis. They are able to see how many more themes emerge when several people work on the data and how much rich thinking can emerge from such discussions. Data analysis ceases to be a mystical process and is revealed to students for what it is: a laborious process involving hard work, keen attention, collaboration, persistence, and tolerance for ambiguity. This exercise alerts students to the various levels at which data can be coded and the need to read manuscripts numerous times. Recently, we have found that some students dedicate more time to learning complicated qualitative data analysis programs than to immersing themselves in the data, which we believe is a mistake. This in-class intensive immersion in two pages of data helps them see what they can mine manually. They learn that in the final analysis (figuratively and literally) they are the most important instrument in qualitative data analysis. Moreover, they learn to trust their ability to analyze data and are able to approach this phase of research with greater competence and confidence. Morals and Ethics in Research Reynolds (1979) distinguished between moral concerns in research, defined as “acting in accordance with accepted notions of right and wrong” (p. ix) and ethical concerns, which generally relate to conformity to a code or set of principles established by a professional organization. Moral principles are more personal and likely to stem from family, religious, and cultural values. Institutional review boards and the Ethical Principles of Psychologists (American Psychological Association, 1992) concern themselves with ethical decisions, which are often narrower and easier to resolve than the moral dilemmas involved in research. We want our students to grapple with both ethics and morals in research. Through the following exercise, we prepare our students to think deeply about moral and ethical issues, seek direction in professional guidelines and through discussion with colleagues and supervisors, and ultimately make their research decisions with as much ethical and moral sensitivity as possible. Prior to this exercise, we assign reading materials on research ethics in psychology (e.g., Fontes, 1998; Keith-Spiegel & Koocher, 1998). In class, we present for discussion a series of 15 scenarios that represent ethical dilemmas (available from the authors). In the interest of time, we divide students into small groups to focus on two dilemmas each. After the groups have enough time to discuss their dilemmas as well as 177
what they would do and why, each small group presents the main point of their discussion to the entire class. Often students comment that the ethical guidelines that they have read do not apply easily to the qualitative research dilemmas they are assigned. Three of the dilemmas we use for this exercise follow: 1. You are volunteering or working in a nonresearch setting (e.g., at a rape crisis center or at a restaurant) and in the course of your work decide that this would be a good place to collect qualitative data through informal interviews and observations. You do not want to tell people about your study because you would be speaking with them and observing them anyway. What are the ethical implications of this situation? (see Fine, 1992) 2. You are meeting with scholars in another country where you are planning to conduct research interviews. The experts from that country advocate hiding the tape recorder during the interviews to avoid making the participants nervous. There is no Human Subjects Review Board in that country. What do you do? 3. You are evaluating an educational program for preschool children, a program that you think is useful, if not perfect. In the process of conducting your study you learn that the directors of the program are falsifying rates of completion to maintain their levels of funding. If this information were to become public, the program would be forced to shut down. What do you do with this “guilty knowledge?” (Gottfried, 1996) After completing this exercise, students report that questions of ethics and morals are more complicated than they had realized. They discover that review boards sometimes approve projects that fall within ethical guidelines but are morally questionable, different cultural groups often hold different moral views of a project (Fontes, 1998), and the moral correctness of a project is often disputed by the various stakeholders. Defending Qualitative Research We hope that by the end of the course the students will be able to provide a credible rationale for using qualitative research in psychology. This skill may help them handle the challenges qualitative researchers sometimes confront in their thesis and dissertation committees, grant proposals, and submissions of articles for publication in psychology journals. Toward the end of the course, we require our students to read several articles that outline strategies for defending qualitative methods (e.g., Dreher, 1994; Marshall & Rossman, 1989, chap. 5). In the next class, we hand the students a brief description of a qualitative study in psychology. We like to use a paragraph about a study that one of the students in the class is proposing, with that student’s permission. (We tell the students they can invent information about the study that may be missing from the brief description.) We divide the class into small groups and ask each group to prepare responses to one of the categories of the subsequent questions. The task for these small groups, then, is to develop answers to 178
their assigned questions that adequately and realistically defend qualitative methodology in general and this study in particular. Some of their answers may involve debunking common misperceptions about qualitative research, whereas others involve ensuring the rigor of the specific proposed study. The readings and discussions of the entire course provide them with the information they need (we cite particularly relevant articles next to each question). When we come back together as a large group, students (in the spirit of fun) play the role of skeptical adversaries, asking each group, in turn, the questions they have been assigned. The entire class then discusses or elaborates on their answer before the next question. The questions we assign follow. 1. Post-modern perspective: How can you say anything definitive about your qualitative research findings? Much qualitative research relies on the postmodern perspective, which contends that “reality is in the eye of the beholder.” If everything goes and no reality is privileged, how can you be sure of anything? What’s the point? (Gergen, 1988) 2. Generalizability and sample size: If qualitative research can’t be generalized, what good is it? How can we have any faith in findings that can’t be generalized? Wouldn’t it be better to consider this study a pilot and to conduct a large scale survey afterward? With such a small sample, how can this study be useful? (Kennedy, 1979) 3. Researcher bias: What the researcher attends to, the questions the researcher asks, even the quotes the researcher chooses to put into a report could all bias that report. So, how can we trust a qualitative research report? What can you do to reduce researcher bias? Isn’t it true that qualitative studies will always be able to produce exactly what the investigator expects? (Lather, 1986; Phillips, 1990) 4. Rigor: Social scientists need solid quantitative proof to gain equal footing with physical scientists. Good quantitative instruments have been validated. Qualitative research is just opinion. Shouldn’t we seek objective, factual data instead of subjective opinions? What is the difference between qualitative research and journalism? (Reason, 1994) 5. Reliability and validity: Is this study replicable? If not, how can it be considered scientific? How do you address reliability and validity in qualitative research? (Kirk & Miller, 1986) 6. Subjectivity of the participants: If you use qualitative evaluation methods, how do you know whether a program is truly effective? Maybe a program is good for the people who use it, but they may not like it. If you use qualitative methods you might get an overly negative view of this effective program. Don’t you agree? (Maguire, 1987) 7. Legitimacy: Real science uses statistics. We need to teach our students how to conduct legitimate scientific experiments. What is the point in allowing you to proceed with this study, which would never get funded or get published anywhere? Isn’t it true that qualitative methods do not have legitimacy or acceptance today? Don’t you want tenure? What would you do to increase Teaching of Psychology
the chances that this qualitative proposal will be funded and that an article about the research will be accepted for publication? (Denzin & Lincoln, 1994) Providing the answers to these questions is beyond the scope of this article. However, after a full semester’s study of qualitative research, our students approach these questions with authority and relish. They show pride in their ability to debunk common misperceptions about qualitative research, develop rigorous study designs, and explain their design decisions with convincing rationale. Conclusions A number of indicators point to the effectiveness of this class. First, students tend to rate the course positively, giving it high marks both for usefulness and interest. In their written evaluations, they make comments affirming the utility of the class, such as “You taught me that interviewing well is a lot more complex than just asking and listening.” Second, our colleagues have affirmed the importance of the course by refusing to chair qualitative dissertations unless the student has taken a graduate-level course in qualitative research. Finally, graduates of the class often choose to write rigorous theses and dissertations based solely or partly on qualitative data. Articles based on these studies have been accepted into refereed journals. Most of the activities in this article reflect our belief that people learn best through direct experience. It is not enough to explain that qualitative data are rich and culturally sensitive—students need to experience that richness and sensitivity. It is not enough to explain qualitative procedures, present the rationale for qualitative research, and discuss ethics. The instructor should also provide challenging activities that encourage students to use qualitative methods, reflect on them, and explore complex dilemmas. We hope that the activities presented here will be helpful to other teachers of qualitative research in psychology. We also hope they will stimulate creative instructors to develop additional ways to immerse psychology students in the fascinating, multi-faceted process of qualitative inquiry. References American Psychological Association. (1992). Ethical principles of psychologists (Rev. ed.). Washington, DC: Author. Banyard, V. L., & Miller, K. E. (1998). The powerful potential of qualitative research for community psychology. American Journal of Community Psychology, 26, 485–505. Denzin, N. K., & Lincoln, Y. S. (1994). Handbook of qualitative research. Thousand Oaks, CA: Sage. Dreher, M. (1994). Qualitative research methods from the reviewer’s perspective. In J. Morse (Ed.), Critical issues in qualitative research methods (pp. 281–297). Thousand Oaks, CA: Sage. Fine, M. (1992). Disruptive voices: The possibilities of feminist research. Ann Arbor: University of Michigan Press. Fontes, L. A. (1997). Conducting ethical cross-cultural research on family violence. In G. K. Kantor & J. L. Jasinski (Eds.), Out of the darkness: Contemporary perspectives on family violence (pp. 296–312). Thousand Oaks, CA: Sage. Vol. 27, No. 3, 2000
Fontes, L. A. (1998). Ethics in family violence research: Cross-cultural issues. Family Relations, 47, 53–61. Garfinkel, H. (1967). Studies in ethnomethodology. Englewood Cliffs, NJ: Prentice-Hall. Gergen, K. J. (1988). Feminist critique of science and the challenge of social epistemology. In M. M. Gergen (Ed.), Feminist thought and the structure of knowledge (pp. 27–48). New York: New York University Press. Gottfried, H. (1996). Feminism and social change: Bridging theory and practice. Urbana: University of Illinois Press. Hubbard, R. (1988). Some thoughts about the masculinity of the natural sciences. In M. M. Gergen (Ed.), Feminist thought and the structure of knowledge (pp. 1–15). New York: New York University Press. Keith-Spiegel, P., & Koocher, G. P. (1998). Ethics in psychology: Professional standards and cases (2nd ed.). London: Oxford University Press. Kennedy, M. M. (1979). Generalizing from single case studies. Evaluation Quarterly, 3, 661–678. Kirk, J., & Miller, M. L. (1986). Reliability and validity in qualitative research. Beverly Hills, CA: Sage. Krueger, R. A. (1994). Focus groups: A practical guide for applied research (2nd ed.). Thousand Oaks, CA: Sage. Lather, P. (1986). Issues of validity in openly ideological research: Between a rock and a soft place. Interchange, 17, 63–84. Loos, G. P. (1995). Foldback analysis: A method to reduce bias in behavioral health research conducted in cross-cultural settings. Qualitative Inquiry, 1, 465–479. Maguire, P. (1987). Doing participatory research: A feminist approach. Amherst: Center for International Education, University of Massachusetts. Marshall, C., & Rossman, G. B. (1989). Designing qualitative research. Newbury Park, CA: Sage. Maton, K. I., Hrabowski, F. A., III, & Greif, G. L. (1998). Preparing the way: A qualitative study of high-achieving African American males and the role of the family. American Journal of Community Psychology, 26, 639–668. Matsumoto, D. (1996). Culture and psychology. Pacific Grove, CA: Brooks/Cole. Older, J. (1979). Improving the introductory psychology course. Teaching of Psychology, 6, 75–77. Patton, M. Q. (1990). Qualitative evaluation and research methods (2nd ed.). Newbury Park, CA: Sage. Patton, M. Q. (1996). Preface. In M. B. Sussman & J. F. Gilgun (Eds.), The methods and methodologies of qualitative family research (pp. xvii–xxiii). Binghamton, NY: Haworth. Phillips, D. C. (1990). Subjectivity and objectivity: An objective inquiry. In E. Eisner & A. Peshkin (Eds.), Qualitative inquiry in education (pp. 19–37). New York: Teachers College Press. Piercy, F. P., & Nickerson, V. P. (1996). Focus groups in family therapy research. In D. H. Sprenkle & S. M. Moon (Eds.), Research methods in family therapy (pp. 173–185). New York: Guilford. Reason, P. (1994). Participation in human inquiry. Thousand Oaks, CA: Sage. Reynolds, P. D. (1979). Ethical dilemmas in social science research. San Francisco: Jossey-Bass. Rossman, G. B., & Wilson, B. L. (1985). Numbers and words: Combining quantitative and qualitative methods in a single large-scale evaluation study. Evaluation Review, 9, 627–643.
Note Send correspondence to Lisa Aronson Fontes, School of Education, SDPPS, University of Massachusetts, Amherst, MA 01003; e-mail:
[email protected]. 179
