Assessing EFL Learners’ Cultural Awareness in a Web-based Active Learning Context Wen-chi Vivian Wu Associate Professor of the Department of English Language, Providence University The Republic of China (Taiwan)
[email protected] Yu-chaun Joni Chao Associate Professor of the Department of English Language, Providence University The Republic of China (Taiwan)
[email protected] Michael W. Marek Associate Professor if the Department of Communication Arts, Wayne State College Wayne, Nebraska, USA mailto:
[email protected]
Abstract: This paper presents an innovative EFL teaching instruction in which an advanced English Conversation class of 17 students was structured with a learner-centered orientation, and cutting-edge technology was used to allow Taiwanese EFL students to talk “live” with a native speaker of English in America via Internet videoconferencing. Given such a learning context, the purpose of this study included the use of a holistic model to evaluate students’ learning outcomes quantitatively in terms of students’ cultural awareness as a result of this web-based English program. Multiple forms of quantitative evaluation were used to assess student progress during the class, including using portfolio assessment to measure cultural awareness of students. In this case, portfolios assessment, involving the students in the assessment process, was employed to measure the degree of cross-cultural awareness by locating evidence of critical thinking in student-generated texts. This study produces a set of guidelines, directions, and “best practices” for instructors in any language learning context.
Note: This document is a pre-press version of this conference paper, presented in 2011 at the AACE EdMedia conference in Lisbon. Cite as:
Wu, W.c.V., Chao, Y.C.J. & Marek, M. (2011). Assessing EFL Learners’ Cultural Awareness in a Web-based Active Learning Context. In T. Bastiaens & M. Ebner (Eds.), Proceedings of EdMedia: World Conference on Educational Media and Technology 2011 (pp. 1254-1259). Association for the Advancement of Computing in Education (AACE). Retrieved from http://www.editlib.org/p/38031.
There may be slight formatting and editorial differences from the published version. Although Michael Marek is not listed as an author on page 1 of the published version of this paper, he is credited as third author at http://www.editlib.org/p/38031.
Introduction 21st Century Internet technologies have led to more and more experimentation with using the Internet to connect learners of English as a Foreign Language (EFL) with native speakers, or speakers of other languages around the world, for the purpose of cross-cultural communication or social/professional networking. Besides, innovative instructors constantly look for new methods and techniques to make the best use of what students do well and to teach new skills effectively, and this has led to a growing body of literature about use of the Internet and technology to enrich EFL instruction (Liu & Chen, 2007; Payne & Ross, 2005; Wu & Marek, 2009). Such network based cross-cultural interactions are often interesting and enjoyable for learners, but do they lead to actual improvements in English cultural awareness? This study presents a model of multi-dimensional assessment in such learning contexts using Critical Text Analysis (CTA).The site of this study is Taiwan, where the educational system usually treats English as a classroom subject to study, similar to math or geography. English learning happens only in certain locations or places, like schools or English cram (special tutoring) schools, and these settings rarely include meaningful interaction with native English speakers or authentic materials that relate to the target culture (Kwon, Shih, Renandya, & Koike, 2000). In the 2009-2010 academic year, Taiwanese investigator of this research collaborated on a project in which the American researcher who consistently, via Internet videoconference, spoke to and interacted with learners in an advanced English conversation class taught by the Taiwanese researcher. The Taiwanese collaborator is a native speaker of Mandarin, teaching a class in Conversational English at a large private university in central Taiwan. The American collaborator is a native English speaker teaching at a public college in the central United States. Both have educational philosophies that downplay traditional lecture and emphasize student-centered active learning. The researchers began this project in the belief that further research is warranted into assessment of EFL instruction that uses student-centered instruction, active learning, and authentic materials in the form of Internet videoconferences with a native speaker. Purpose of the Study The learning objectives of the conversation class were to enhance their cultural knowledge, and to develop students’ critical thinking ability. Based on the purpose of assessing cultural awareness as a result of this computer mediated communication (CMC) English program, the research goal this related research project was: 1.
To demonstrate assessment of cultural awareness and critical thinking of students about other cultures
There are a number of studies which have attempted to measure outcomes of cross-cultural leaning programs or effectiveness of an inter-cultural curriculum (Seelye &Wasilewski, 1996). The commonly used methods of assessment have been tests, surveys, published inventories, instruments with open-ended reactions to statements. However, Kauffman, Martin, and Weaver (1992, p.144) argue that these approaches, which have dominated assessment of cross-cultural leaning experiences, tend to be too “limiting” and “reductionistic” and call for new approaches which they regard as holistic, synergistic, and multifaceted. In view of this, the researchers decided to adopt the portfolio as an assessment tool, which included a quantitative instrument to measure the degree of cross-cultural awareness by locating evidence of critical thinking in a student-generated text. A holistic authentic approach like this one is certainly at the cutting edge of measuring knowledge about the garget culture.
Literature Review The power of technology to facilitate active, cross-cultural learning experiences While study-abroad intensive English programs are usually unaffordable for the majority of Taiwanese students, and an EFL environment cannot teach real, authentic language usage, either, unless it can provide opportunities for authentic interaction and cultural learning, only creating an environment as close to the target culture as possible will enable and motivate students to really learn English. Even though a student may be able to produce a perfectly grammatical sentence, the usage and connotations are often dissimilar from that of native speakers, whose use of the language is more highly nuanced. Online learning, employing authentic interaction with native English speakers with a focus on cross-cultural learning awareness as a curriculum goal, can provide a solution, but the prerequisite is that the students must see the potential long-term value of learning English for membership in the broader community of the world (Dörnyei, 2005), and thus experience integrative motivation. The importance of program evaluation The researchers were inspired by the John M. Norris position paper, “The Why (and How) of Assessing Student Learning Outcomes in College Foreign Language Programs.” Norris (2006) mentions the challenges and opportunities that the movement to assess student learning outcomes creates for foreign language instructors. According to Norris, the potential of assessment, if done right, can facilitate student learning and improve programs, indicating the need for and the importance of a suitable assessment approach. Accordingly, foreign language instructors must recognize the important role assessment plays in a successful language program. In order to evaluate their programs, instructors assess students by gathering information about their learning (Sullivan, 2006). In the study of Brislin and Yoshida (1994), awareness as a single factor among many is the approach for assessing student learning outcomes in their cross-cultural training program because for them both attitudes toward the target culture and skills in communication play an equally important role. With the new knowledge of the culture, students become cognizant of similarities and differences. The information used for this study to measure student learning outcomes in terms of cultural awareness and linguistic competency came in the forms of pre-oral tests, post-oral tests and student portfolio materials. Critical Text Analysis in a Portfolio Critical Text Analysis (CTA) is a process of analyzing a text (which can take the form of any print, aural, or visual content) and subjecting it to multidimensional analysis. Teaching and literacy skills are multimodal and speech and writing are two of the modes used in the classroom for learning. Thus, CTA scrutinizes the text for ideas and meaning, which are, themselves, socially constructed (Bourne & Jewitt, 2003). Portfolio assessment, which uses Critical Text Analysis, is a notable assessment tool that has rarely been used to date in Taiwan. Even though the technique has been widely adopted by Japanese and American educators for a specific context, portfolio assessment is not widely regarded by Taiwanese instructors as an alternative to traditional testing for core courses. However, because the portfolio constitutes a cross-cultural encounter in itself, Taiwanese instructors must be persuaded of its great value. For example, this approach will allow for a wide range of individual variation and a full expression of the personal nature of a cross-cultural experience. Part of the attraction of portfolio assessment is that students are involved in deciding what to include in their portfolios to demonstrate their performance. Guidance and direction are provided by the instructor regarding what data best reflect their learning and growth. This selection process is the initial step required for portfolio assessment and described by Hill and Ruptic
as “The process of gathering evidence and documenting a child’s learning and growth.” (Hill & Ruptic, 1994, p. 8). Another definition of portfolio assessment echoes Hill and Tuptic’s point stating that “the purpose of good assessment is to inform instruction and, simultaneously, to provide…accurate and meaningful information regarding the student’s progress (Gillespie, Fore, Gillespie, & Leavell, 1996, p. 480). Therefore, the goal of assessment is shifted from merely looking at the final product of student work but rather to identifying individual learning, growth, and progress. Evidence of much growth and progress indicates high performance from the evaluation perceptive, and vice versa. Comparatively low performance demonstrates little evidence growth and progress.
Research Methods Instructional design Four times during the semester, the American researcher spoke “live” with the 17 Taiwanese students in the English class in Internet videoconferences (via SKYPE). The American had previously provided video presentations each time about American cultural topics such as American holidays, American vacations and national parks, entertainment in America, and American clothing. Each lesson consisted of advance preparation by the students, the recorded video presentation by the American teacher, the student "live" presentation back to the American teacher in class, and follow-up conversation between the native speaker and the student group in the videoconference about the topic. In each case, the vocabulary was kept close to the range of the students, but they were also challenged by adding new vocabulary specific to the cultural elements being discussed. The local teacher in Taiwan provided instruction and methods of critical thinking since the instructional design required the students to be actively involved in their learning, including analyzing the American’s presentations, deciding their own presentation topics, and then researching selected topics, preparing and delivering the presentation (see Table 1). In addition, scaffolding from team members and the instructor also played an essential role in guaranteeing a successful student presentation, including initial brainstorming as a team for ideas and contents development to be included in their presentations, actual compositing written texts for presentations, advising and suggestions from the instructor, and spiritual support and rapport with team members. Participants This required advanced English conversation class, offered during the 2009-2010 academic year, consisted of 17 sophomores (15 girls and 2 boys) majoring in English at a private four –year academic university in central Taiwan. Since all of them had successfully passed a basic English conversation course in their freshman year and the intermediate level of a standardized test (General English Proficiency Test), their English ability in oral communication was sufficient to create a theme-based presentation in English and interact with native speakers. Final Essay Each student in the class wrote a final reflective paper based on information collected during the course of the semester, and following a framework provided by the Taiwanese teacher. Reference data for preparing the paper included a statement of expectations for the conversation class, an observational journal, Power Point presentation materials, video recordings of the American’s presentations and in-class interaction videos. The paper was about 1,000 words written in English. The students received the following “prompt questions” in order to stimulate their critical thinking:
a. What were the differences and similarities that you observed or recognized? b. What are your reflections based on what you have organized?
c. What new knowledge and experience did you obtain? Please give specific examples. d. To what degree did your English ability improve in terms of the four skills? e. Did you broaden your worldview by taking this class? f. Have you understood America more deeply and thoroughly? g. Have you changed in your way of thinking? h. How has the experience of interacting with the native speaker affected your future plans or goals regarding your English study? The essays were evaluated by the American researcher and three American colleagues who teach communications-related subjects for cross-cultural recognition and reflection. The researchers used two assumptions in quantifying the Critical Text Analysis, as follows:
a. Refection can happen only if recognition exists. In other words, recognition is the perquisite for reflection.
b. Therefore, recognition is more common in an individual’s thinking process, and we presume it will occur more frequently. In other words, there will be examples where recognition can be located the text without the company of reflection. Based on these two assumptions, the researchers used the following formula, inspired by Ingulsrud, et al. (2002) to measure the degree of critical thinking about other cultures is:
A
R C
In this construct, A stands for cultural critical thinking in the EFL context. R and C stand respectively for the number of reflection and recognition incidents in the text. Ideally, there would be at least one reflection for every recognition if a student reflects on every recognition incident. In reality, the students sometimes performed more than one reflection for a single recognition. The highest score was defined as indicating the highest degree of cultural critical thinking in the EFL context. Data Analysis The American researcher and three American colleagues, all of whom teach communication or language related college courses, listened to the audio recordings of the pre- and post-tests and also read the final essays of the students to evaluate the students’ linguistic competence and cultural awareness. The points determined by rubric for each of the 17 students in each of three categories, essay recognitions, essay reflections, and pretest/posttest improvement, were analyzed by Excel spreadsheet for basic percentages and averages. Because of the small size of the class, the researchers did not attempt further statistical analysis for this paper.
Findings The American reviewers read and independently evaluated the 17 student essays. The data for recognitions ranged from one student listing only seven recognitions, determined by one reviewer, to another student making 50 recognitions, determined by another reviewer. For reflections, the data ranged from seven reflections, identified by three of the four reviewers for one student, to 29 reflections identified by the remaining reviewer for another student. The reviewers found an average of 21.5 recognitions in each paper and an average of 14.8 reflections per paper. Applying the A=R/C formula, the researchers determined that the students averaged .705 reflections per recognition, defined as a cultural critical thinking (A) value of 70.5%.
Because the researcher were primarily performing their initial tests of Critical Text Analysis for evaluation of EFL learning, they did not take the additional step for this paper of analyzing themes in the student essays. This is a limitation of the study, however it can easily be addressed in the future by an expanded CTA methodology performing a full content analysis of the student portfolios for themes and richness of information on the experiences of the students.
Discussion and Conclusions The critical text analysis portion of this study was inspired by Ingulsrud et al. (2002) but made significant changes in their methodology. That study used a construct in which cultural awareness (A) was defined as recognitions (C) plus reflections (R), A=R+C. The current study authors, however, believed that awareness was less important than critical thinking about the students’ own and other cultures and therefore used A=R/C. In this way, the study assessed the level of critical thinking, addressing a flaw that Ingulsrud et al. identified in their own study, that a student with numerous but superficial recognitions could still achieve a high score with no critical thinking. They stated that “…critical tools involving uncovering presuppositions and socio-political systems of power would have to be taught in order to expect critical reflection to occur with regularity” (p. 487). Advanced EFL skills require critical thinking about the cultural differences and similarities, in order to fully understand the target culture or cultures. In the EFL environment, critical thinking about cultural differences and similarities are not just important but vital to functioning in the cosmopolitan international culture postulated by Dörnyei (2005). As EFL has evolved from training to interact with native speakers to training to interact with others who also speak English as a second or foreign language, the need for critical thinking has burgeoned. It is now a multi-dimensional requirement in which, for example, the Taiwanese EFL learner must compare and contrast American and Taiwanese culture, how those cultures compare and contrast with other Asian cultures, and how all of those cultures interaction with European and African cultures, etc. The instructional design employed in the project can be used to stimulate critical thinking about the multiple cultures encompassed by EFL learning in the 21st Century. It is student-centered classrooms, authentic materials that intrigue and engage the students, and active learning that promotes critical thinking. This study has, again, validated these instructional techniques and has, furthermore, performed a pilot study of a methodology by which multimodal assessment can lead to rich understanding of the experiences of the students in order to even further center the learning experience on the students.
References Bourne, J., & Jewitt, C. (2003).
Orchestrating debate: a multimodal analysis of classroom interaction. Reading,
37(2), 64-72. Brislin, R. W., & Yoshida, T. (1994). Intercultural communication training: An introduction. Thousand Oaks, CA: Sage. Dörnyei, Z. (2005). The psychology of the language learner: Individual differences in second language acquisition. Mahwah, NJ: Lawrence Erlbaum. Gillespie, C. S., Fore, K. L., Gillespie, T. D., & Leavell, A. G. (1996). Portfolio assessment: Some questions, some answers, some recommendations. Journal of Adolescent and adult Literacy, 39 (4), 480-491. Hill, B. C., & Ruptic, C. (1994). Practical aspects of authentic assessment: putting the pieces together. Norwood,
MA: Christopher Gordon. Ingulsrud, J. E., Kai, K., Kadowaki, S., Kurobane, S. & Shiobara, M. (2002). The assessment of cross-cultural experience: Measuring awareness through critical text analysis. International Journal of Intercultural Relations 26, 473-491. Kauffmann, N. L., Martin, J. N., Weaver, H. D., and Weaver, J. (1992). Students abroad: Strangers at home. Yarmouth, ME: Intercultural Press, Inc. Kwon, O., Shih, Y. H., Renandya, W., Koike, I. (2000). Opening Address. Plenary Symposium: English Education in East Asia for the 21st Century: Seeking for Common Ground, Okinawa, Japan. Liu, G., & Chen, A. S. (2007). A taxonomy of Internet-based technologies integrated in language curricula. British Journal of Educational Technology, 38(5), 934–938. Norris, J. M. (2006). The why (and how) of student learning outcomes assessment in college FL education. Modern Language Journal, 90(4), 590-597. Payne, J. S., & Ross, B. M. (2005). Synchronous CMC, working memory, and L2 oral proficiency development. Language Learning and Technology, 19(3), 35-54. Sengupta, S. (2001).
Exchanging ideas with peers in network-based classrooms: An aid or a pain? Language
Learning & Technology, 5(1), 103-143. Seelye, H. N., Wasilewski, J. H. (1996). Between culture: Developing self-identity I a world of diversity. Lincolnwood, IL: National Textbook Company. Sullivan, J. H. (2006). The importance of program evaluation in collegiate foreign language programs. The Modern Language Journal, 90, 590-593. Wu, W. & Marek, M. (2009). The Impact of Teleconferencing with Native English Speakers on English Learning by Taiwanese Students. International Journal on E-Learning. 8(1), 107-126.
