Session F2A USE ASSESSMENT TO DEVELOP SERVICE ...

Session F2A USE ASSESSMENT TO DEVELOP SERVICE-LEARNING REFLECTION COURSE MATERIALS Edmund Tsang1 2 Abstract – The Reflection Rubric proposed by Barbara Olds will be used to re-examine the reflection course materials developed previously for three common types of servicelearning design project. The Reflection Rubric is based on the Reflective Judgment Model developed by Patrick M King and Karen Strohm Kitchener, and it can be used to assess and document the development of a student from an uncritical thinker to a Reflective Thinker -- someone with “fully differentiated abstract categories who can see the problematic nature of controversies.” The purpose of the reflection materials is for students to examine the societal aspects of engineering and technology by examining the larger issues behind the needs to which the service-learning projects are responding, and to explore the professional and ethical responsibilities of an engineer by focusing on doing the work of an engineer responsibly and well, whether in the workplace or in community service.

experience the positive impact of engineering and technology on community. The difference between a service-learning design project and a traditional design project in engineering is compared in Table I below. Thus, service-learning design projects compliment traditional design projects in the training of engineering undergraduates regarding engineering design, teamwork, and oral and written communication skills. TABLE I. Comparison Between Traditional Design Project and Service-Learning Design Project Traditional Design Project Sponsor

Index Terms – Service-Learning Reflection, Reflection Rubric, Reflective Judgment Model, Societal Aspects of Engineering and Technology. Interaction

INTRODUCTION Service-learning is a form of experiential education in which community service provides the context for student to learn and develop. In a service-learning design project, engineering students work with team member(s) with a different socio-economic background and little understanding of engineering and technology; are required to communicate effectively with a diverse audience; and Service-learning practitioners recognizes that “learning and development do not necessarily occur as a result of experience itself but as a result of a reflective component explicitly designed to foster learning and development” [1]. The primer on service-learning in higher education further states that “In service-learning, the goal of reflection is to promote learning about the larger social issues behind the needs in which [student] service is responding. This include the social, cultural, economic, and political contexts of the needs or issues being addressed,” and “Reflection should include opportunities for participants to receive feedback from those persons being served, as well as from peers and program leaders.” Unfortunately, the words, “reflect” and “reflection,” have a negative connotation among engineering and 1

Analysis & Synthesis

Industry, Engineering Organization, Faculty Engineers, Engineering Faculty and Students

Mostly En gineering Topics

Service-Learning Design Project Community Groups, K-12 Schools

People from diverse socio-economic groups; some with little knowledge of engineering and technology Engineering and Non-Engineering Topics

technology faculty because, as one engineering professor put it, they conjure up the image of “a faculty sitting by ‘Walden Pond’ and meditating for the summer.” If “reflect” is replaced by the word “examine,” then the learning outcomes of service-learning reflection (“learning the larger social issues behind the needs in which [student] service is responding” and gaining “a deeper understanding of the historical, sociological, cultural, economic, and political contexts of the needs or issues being addressed”) match Outcome (e) of Criterion 3 of ABET Engineering Criteria 2000 (“the broad education necessary to understanding the impact of engineering solutions in a global and societal context”) [2]. While community-based design projects can be found in many engineering programs, most focus primarily on

Edmund Tsang, College of Engineering and Applied Sciences, Western Michigan University , Kalamazoo, MI 49008 [email protected]

2

0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference F2A-15

Session F2A engineering design and the reflection component is either absent or occurs by happenstance rather than deliberately structured. This is probably because of the training of engineering and technology faculty, and of the lack of course materials for reflection in engineering and technology. Consequently, course materials on the “reflective component” of service-learning design project are needed for community-based design projects to engage the students to exa mine the societal aspects of engineering and technology. Reflection course materials for three types of servicelearning projects found commonly in engineering and technology have been proposed previously [3]. Using assessment as a lens to focus on the learning outcomes, the reflection course assignments will be re-examined in this paper. , The community partners of the service-learning design projects are K-12 schools, Habitat for Humanity, or grassroots environmental organizations,

REFLECTION RUBRIC AND REFLECTIVE JUDGMENT M ODEL The tool to re-examine the reflection course materials is the Reflection Rubric proposed by Barbara Olds [4]. The Reflection Rubric is given in Table 1 at the end of paper. The rubric can be used to assess and document the stages of student development described by the King and Kitchener “Reflective Judgment (RJ) Model” [5]. According to the RJ model, the lowest, Stage 1 describes an uncritical thinker while the highest, Stage 7, describes someone possessing Reflective Judgment (“based on the evaluation and integration of existing data and theory into a solution about the problem at hand, a solution that can be rationally defended as the most plausible or reasonable, taking into account the set of conditions under which the problem is being solved”). The RJ Model requires that the nature of the problems and the assignments presented to the students be “ill structured” in order for students to develop and grow. The Reflection Rubric can be used to assess four sets of critical thinking skills grouped under Blosser Taxonomy as Evaluative Thinking, Divergent Thinking, Convergent Thinking, and Cognitive Memory. In Evaluative Thinking, the Reflection Rubric documents a student growing from depending solely on authority or unexamined prior beliefs to evaluate the information presented (Pre-Reflective Thinking: RJ3 and less), to using information to support beliefs and recognizing the need to gather more information (QuasiReflective Thinking: RJ4 and RJ5), and finally to knowing how to address the self-identified limitations to achieving goals (Reflective Thinking: RJ6 or higher). In Divergent Thinking, the Reflection Rubric documents a student growing from being unable to make connections among relevant information (Pre-Reflective Thinking), to organizing available information into a framework for achieving goals (Quasi-Reflective Thinking),

and finally to organizing and prioritizing the available information appropriate for the task of self-assessing achievements of goals (Reflective Thinking). In Convergent Thinking, the Reflection Rubric documents a student growing from being unable to analyze or interpret information regarding how well goals were met (Pre-Reflective Thinking), to analyzing information from multiple perspectives to determine how well the goals were met (Quasi-Reflective Thinking), and finally to analyzing how to improve goal achievements (Reflective Thinking). In Cognitive Memory, the Reflection Rubric documents a student growing from relying from external validation to determine goal achievement (Pre-Reflective Thinking), to using a range of evaluated information to determine goal achievement (Quasi-Reflective Thinking), and finally to developing strategies for improving goal achievements (Reflective Thinking). Ideally, the reflection assignments for service-learning design projects should be created to allow the students to develop and grow these four sets of critical thinking skills.

REFLECTION ASSIGNMENTS Journal writing is a common form of assignment for the reflection component of service-learning [6]. Reflection assignments in which the community partners are K-12 schools, Habitat for Humanity, and grassroots environmental organizations have been proposed previously [3], because they are the most common service-learning projects in engineering and technology. Applying assessment based on the Reflective Rubric, these reflection assignments are being re-examined and revised. For K-12 service-learning design projects, the following reflection assignments are proposed: 1) Prior to assigning the service-learning design projects, ask students to write an essay on their opinions of the factors affecting K-12 schools and the needs that have arisen. At the completion of the service-learning design projects, ask student to write a follow-up essay and evaluate whether their original assumptions were justified or not justified, as a result of their interaction with K-12 teachers, administrators, and students. 2) Ask the students prior to assigning the servicelearning projects if they have any stereotypical impressions of K-12 teachers and administrators, and of their roles regarding to the need that the service-learning project is responding. Ask the students at the completion of the projects if these prior impressions were confirmed or debunked as a result of interacting with these groups during the service-learning projects? How? 3) Raymond B. Landis, in his book Studying Engineering: A Road Map to a Rewarding Career [7], identifies “Putting Something Back” as a part of


Session F2A personal and professional development to broaden the student’s education. Landis cites “Serving as an Ambassador” (returning to the student’s high school or other high schools to speak to teachers and students) as an example of “Putting something back.” Assign the students to write essays to discuss whether the service-learning project met Landis’ definition of “putting something back,” and what advice they would give to K-12 students and teachers if they were student ambassadors. And why. For service-learning design projects in which the community partner is a grassroots environmental, conservation, or preservation organization, Moffat and Rand’s [6] suggestions for reflection topics are modified to the following: 1) At the start of the service-learning project, ask the students to identify stereotypical images of an environmentalist as portrayed in mainstream media. Ask the students to ask the community partners if they have any prior stereotypical impressions of an engineer. 2) At the completion of the service-learning project, ask the student if the stereotypical images of environmentalist and engineer were confirmed or debunked by the interaction, and why. 3) Ask students to analyze the commonalities and differences, if any, of the way engineers and the general public view and approach environmental problems, and why the commonalities and differences exist. 4) At the completion of the service-learning project, ask the students to propose ways in which engineers and citizens can work together to solve environmental problems, describing the common values that these two groups can drawn upon as well as the obstacles that must be overcome. For service-learning projects in which the partner is Habitat for Humanity, the following topics for journal or essay writing are proposed: 1. Prior to assigning the service-learning project, ask students to discuss their impressions of whether there is a shortage of affordable housing in the United States, what the factors might be, and the profile of someone needing housing assistance. 2. At the completion of the service-learning project, ask students to write an essay to assess their original impressions regarding affordable housing and the profile of someone needing assistance in housing. How and why. 3. There are a number of essays listed on The Ayn Rand Institute website that attack community service in higher education [8]. One such essay postulates that “liberals and conservatives alike have the view that individuals have a moral duty to serve society…It is the opposing morality, that of ‘selfishness,’ that

enables man to achieve his own happiness.” Ask students to compare and contrast this view with the ABET definition of engineering, which states that the profession is “for the benefit of mankind.”

ASSESSING STUDENT PERFORMANCE USING THE REFLECTIVE RUBRIC Students develop Evaluative Thinking skills when they are asked to compare and contrast, prior to and upon completing the service-learning projects, their understanding of the community need as well as their impressions of community partners. The Reflective Rubric can be used to assess and document the development of students from the PreReflective/Pre-Evaluative Thinking stage (reply on unexamined prior impressions of their community partners, and the interactions during the service-learning project exert no influence on the students’ prior impressions -- RJ3 and less); to the Quasi-Reflective/Quasi-Evaluative Thinking stage (superficially present information to support their prior impressions but recognizing their prior impressions of the community partners may be incomplete as a result of the service-learning project -- RJ4; present information to support their prior impressions but also indicating the needs to gather more information to support their impressions of the community partners -RJ5); and the Reflective/Evaluative Thinking stage (use information to support their prior impressions and able to correct the prior incorrect impressions as a result of the service-learning project -- RJ6 and higher). The assignment in which students are asked to justify the service-learning project based on Landis’ “Putting Something Back” criterion and to give advice to K-12 students can be used to grow a student’s Divergent Thinking skill. The Reflective Rubric can be used to assess and document the development of students from the PreReflective/Pre-Divergent Thinking stage (unable to make connection between service-learning project and community service, and give advices to K-12 students that are irrelevant – RJ3 and less); to Quasi-Reflective/Quasi-Divergent Thinking (make connection between service-learning project and community service and give advices to K-12 students based on own achievements – RJ4; make connection between service-learning and community service, and identify the commonalities and differences between K-12 students and self to give a few relevant advices– RJ5); to Reflective/Divergent Thinking (make connection between service-learning project and community service and give many relevant advices to K-12 students – RJ6 and higher) The assignment in which students are asked to examine the views expressed on The Ayn Rand Institutes and ABET’s definition of engineering can be used to grow a student’s Convergent Thinking skill. The Reflective Rubric can be used to assess and document student development from Pre-Reflective/Pre-Convergent Thinking (restates the


Session F2A views of The Ayn Rand Institute and ABET but unable to compare and contrast – RJ3 and less); to QuasiReflective/Quasi-Convergent Thinking (restates the two views and acknowledges incongruity – RJ4; uses carefully evaluated information to compare and contrast the two views – RJ5); to Reflective/Convergent Thinking (compare and contrast the two views, and present credible arguments to win over the other side – RJ6 and higher) The assignment in which students are asked to compare and contrast the way in which the public and engineers view environmental problems, and to propose solutions can be used to develop a student’s Cognitive Memory skill. The Reflective Rubric can be used to assess and document student development from Pre-Reflective Thinking/PreCognitive Memory (unable to make connection between the way the public and engineers view environment problems – RJ3 and less); to Quasi-Reflective/Quasi-Divergent Thinking (acknowledge the possibility of commonality between the public and engineer on environmental problems – RJ4; recognize the differences and commonalities of the public and engineer on environmental problems – RJ5); to Reflective Thinking/Cognitive Memory stage(identify and use the commonality between the public and engineer to propose solution to environmental problem – RJ6 and higher).

CONCLUSION Service-learning design projects compliment traditional engineering design projects in the training of a more complete engineering undergraduate, because engineers need more than technical skills to meet the challenges of the 21st Century. Solutions to many societal problems facing engineers require extensive interaction and communication with people of diverse social, cultural, and economic backgrounds. They must have a broad education to appreciate the impact of engineering and technology on

societal and global context. Reflection assignments in service-learning design project prepare engineering students to meet these challenges. Using assessment as a lens to focus on the learning outcomes, previously proposed reflection assignments are re-examined, revised, and presented in this paper. These assignments, together with the Reflective Rubric proposed by B. Olds, can be used to grow and assess student performance in understanding the impact of engineering and technology on society.

REFERENCES 1. Jacoby, B., “Service-Learning in Today’s Higher Education,” in Service-Learning in Higher Education: Concepts and Practice, ed. B. Jacoby and Associates, Jo ssey-Bass Publishers, San Francisco, CA, 1996, page 5. 2. Accreditation Board for Engineering and Technology, www.abet.org . 3. Tsang, E., and Pritchard, M.S., “Service Learning: A Positive Approach to Teaching Engineering Ethics and Social Impact of Technology,” Proceedings of 2000 American Society for Engineering Education Annual Conference, St. Louis, MO, June 18-21, 2000, Session 3630. 4. Olds, B., private communication. 5. “King, P.M., and Kitchener, K.S., editors, Developing Reflective Judgment: Understanding and Promoting Intellectual Grown and Critical Thinking in Adolescents and Adults, Jossey-Bass Publishers, San Francisco, CA; 1994, page 8. 6. Moffat, J., and Decker, R., “Service-Learning Reflection for Engineering: A Faculty Guide,” in Design That Matters: Concepts and Models for Service-Learning in Engineering, ed. Tsang, E., American Association for Higher Education, 2000, page 31-39. 7. Landis, Raymond B., Studying Engineering: A Road Map to a Rewarding Career. Discovery Press, 2000, page 232. 8. The Ayn Rand Institute, http://www.aynrand.org/medialink


Session F2A TABLE II. Reflection Rubric of Barbara Olds (Based on Blosser Taxonomy and Reflective Judgment Model RJ LEVEL RJ 3 and less (1)

RJ 4 (2)

RJ 5 (3)

RJ 6 or higher (4)

Score Unable to evaluate information presented; relies primarily on unexamined prior beliefs.

Presents information to support previously held beliefs; superficial understanding of information; acknowledges need to gather more information.

Uses information to establish wellsupported argument for achieving goals; indicates need to gather more information to further support assertions.

Divergent thinking

Does not make connections among relevant information.

Presents holistic selfassessment; limited breakdown and focus on achievement of individual goals.

Organizes available information into viable framework for exploring complexities of achieving goals.

Convergent thinking

Presents information but does not attempt to interpret or analyze how well goals were met.

Provides limited interpretation or analysis of how well goals were met.

Presents interpretation and analysis from multiple perspectives of how well goals were met.

Cognitive Memory

Asserts that goals were met; relies on external authority (moderators) for validations

Uses limited information but acknowledges at least the possibility of uncertainty

Uses range of carefully evaluated relevant information during selfassessment

EVALUATIVE

BLOSSER TAXONOMY

THINKING

Uses information to establish wellsupported argument for achieving goals; suggests viable strategies for addressing selfidentified limitations. Organizes and prioritizes available information appropriate for the task of selfassessing achievement of goals. Presents interpretation and analysis from multiple perspectives of how well goals were met; also includes analysis of how to continue to attempt to achieve goals. Uses range of carefully evaluated, relevant information during self assessment; suggests viable strategies for obtaining new information to address limitations
