The Public Affairs Scale: Measuring the Public Good Mission ... - naspaa [PDF]

The Public Affairs Scale: Measuring the Public Good Mission of Higher Education Chantal Levesque-Bristol Purdue University Jeffrey Cornelius-White Missouri State University Abstract This article describes the development and construction of the Public Affairs Scale, specifically designed to assess student development in the three areas of the public affairs mission at Missouri State University. This development process involved shared governance, literature review, and other tools used to assess components related to public affairs. The article also explains the public affairs mission at Missouri State University, including definitions of themes and goals and select markers of that mission’s manifestation during the past 15 years. It provides a brief review of assessments used in universities to measure similar constructs, such as engagement and the public good. The Public Affairs Scale is shown to have feasibility (40 items), adequate to strong internal consistency, and construct validity. It appears sufficiently sensitive to capture differences between students and within students over time. The Public Affairs Scale has been used to assess the impact of a first-year service-learning initiative. Readers are encouraged to use it for further program evaluation and research. In the early 20th century, Dewey (1916) asserted that the core mission of the university is civic engagement. This assertion has since been supported by recent reports from the Kellogg Commission on the Future of State and Land-Grant Universities, the Kettering Foundation, Public Policy Institutes, Campus Compact, Carnegie Foundation, and other groups (London, 2002). Indeed, the last decade in particular has shown that the democratic mission of education is now commonly understood for academics, with terms like engagement and service learning now in common parlance. For example, Chambers and Gopaul’s (2008) survey of leading scholars in higher education, education association leaders, university presidents, and others found that most JPAE 18(4), 695–716

Journal of Public Affairs Education

695

C. Levesque-Bristol & J. Cornelius-White of the respondents stated that educating students for democracy and citizenship was the core mission of higher education (p. 82). On June 15, 1995, Missouri Governor Mel Carnahan signed into law Senate Bill 340, which gave Missouri State University a statewide public affairs mission, distinguishing it from other universities in the state and beyond. This was done at the urging of then university president John H. Keiser and the Missouri State board of governors. This mission defines a primary way in which a Missouri State education is different from that of other universities and one way by which it educates students to imagine the future. The university situates its mission at the forefront of the sea change toward a more accepted articulation of the broader mission of higher education for the public good. This article describes the specific conceptualization at Missouri State University of a public affairs mission in higher education, highlights selected markers of that mission’s manifestation, and gives a brief review of other universities’ attempts to measure public affairs engagement. This article also summarizes the process the authors used in constructing the Public Affairs Scale. It provides the results of the scale validation, including a brief example of its utilization to evaluate the effectiveness of a first-year service-learning initiative. The Public Affairs Scale grew out of one particular university climate. However, it will undoubtedly prove useful for other universities when considering, developing, and/or testing the role that other universities’ programs serve in meeting the public good. It will be particularly useful to programs focused on community engagement, cultural competence, and ethical leadership.

The Specific Public Affairs Mission of Missouri State University At the beginning of the Public Affairs Scale construction and validation project, Missouri State University was in the midst of redefining and focusing more intently on its public affairs mission. These changes were signaled in many ways, including the establishment of an associate provost for Student Development and Public Affairs and preparations for a new long-range strategic planning process. In addition, a clearer articulation of the mission had developed, culminating in an action from the Faculty Senate during spring 2009 in which goals were adopted in each of three distinct areas: (a) Ethical Leadership, (b) Cultural Competence, and (c) Community Engagement. As expressed by Darabi (2010): This development of specific goals related to the public affairs mission signaled the significance of the mission and the importance of its integration into the curriculum. Community engagement, cultural competence, and ethical leadership are evident in the curriculum of all the colleges and in every department. (p. 1) 696


The Public Affairs Scale However, even though the Public Affairs Mission had been defined as the facilitation and development of community engagement, cultural competence, and ethical leadership, precise definitions of the three themes of the Public Affairs Mission were under discussion and had not yet been accepted by the faculty. Furthermore, the university needed an empirical way to conceptualize and assess the realization of the Public Affairs Mission. For this reason, the authors developed the Public Affairs Scale. To develop the items included in the Public Affairs Scale, the authors examined information from various organizations and sources described in the following review. We also consulted with the Missouri State Faculty Senate and three Provost Public Affairs committees, the Citizenship and ServiceLearning office, Residence Life, the provost’s working definitions at the time, the associate provost for Student Development and Public Affairs, and the Public Affairs Reports from the Missouri State colleges (Cornelius-White & Levesque, 2008). The working definitions we constructed from these sources as well as the currently adopted Public Affairs Mission Pillars Definitions and Goals are presented next. There is general consistency between these three sets of definitions, which demonstrates the utility of the concepts that served as the foundation for the elaboration of the Public Affairs Scale items. Working Definitions of the Three Pillars of the Public Affairs Mission Community Engagement is the process of working collaboratively with and through groups of people affiliated by geography, special interest, or similar situations to address issues influencing the quality of life (Centers for Disease Control and Prevention, 1997, p. 4). Individuals skilled at community engagement are effective participants in the world around them—whether they are working for sustainability, better government, human rights, or other purposes. Cultural Competence means obtaining cultural information and then applying that knowledge (Transcultural Nursing, 2010, p. 1). Individuals who are culturally competent are culturally self-aware, conscious of the dynamics inherent when cultures interact, and able to use cultural knowledge effectively in communication and behavior (King, Sims, & Osher, 2010). Culturally competent individuals can interact effectively with people of diverse origins, perspectives, and abilities. Ethical Leadership is leadership informed by principles. Ethical leaders know their core values and have the skills to live them in all areas of life (Center for Ethical Leadership, 2007). Communication, collaboration, and consensus building are important tools of ethical leaders. Ethical leaders bring about change, but they know that doing it the right way is as important as “getting it done.”


697

C. Levesque-Bristol & J. Cornelius-White Accepted Definitions and Goals of the Public Affairs Mission The preceding definitions were used as the conceptual foundation for the creation of items for the original administration of the Public Affairs Scale. The following definitions are similar, but represent the current definitions and goals of the themes accepted by faculty at Missouri State University (Public Affairs Mission, 2010). Community Engagement means recognizing needs in the communities within which an individual belongs, and then contributing knowledge and working with the community to meet those needs. Community engagement requires people to extend beyond themselves for the betterment of the community. It is a process that fosters greater awareness and personal growth. Cultural Competence begins with cultural self-awareness and expands to knowledge, respect, and skills for engaging with those of other cultures. Culturally competent individuals respect multiple perspectives and are able to successfully negotiate cross-cultural differences. Ethical Leadership means striving for excellence and integrity as an individual continually develops ethical and moral reasoning while contributing to the common good. Ethical leaders have the courage to live by their principles in all parts of their personal and professional lives.

Brief Review of Public Affairs Engagement Assessments In this section, we present a brief review of the literature consulted from various organizations and sources. As described here, although many references to community engagement, cultural competence, and ethical leadership were found and tools associated with some of the pillars could be identified, measures and tools assessing all three pillars of the Public Affairs Mission from the students’ perspective did not exist. For example, Westdijk, Koliba, and Hamshaw (2010) recently provided a review in the Journal of Higher Education Outreach and Engagement on assessment of community engagement within universities. They found that most methods for assessing engagement in higher education have focused on the institutional level. In line with this finding, the Missouri State Center for Citizenship and Service Learning (CASL) report and the College Level Synthesis review, summarized later, provide institution-level data. In their review, Westdijk et al. (2010) gave examples of institution-level assessments from North Carolina State University, University of Minnesota, Penn State, and Ohio State. Similarly, Furco (2001), Gelmon, Seifer, Kauper-Brown, and Mikkelsen (2005), Holland (1997), and Wortis, Goldstein, Varagas, and Grumback (2007) also provided published assessment models in this vein. In addition, those authors emphasized the importance of assessing broad institutional practices; and like Westdijk et al. (2010), they also recommended assessment of “specific community engagement 698


The Public Affairs Scale practices of individual” faculty members and, by implication, students (p. 5). Westdijk et al.’s (2010) survey, using mixed methods to assess faculty involvement, perspectives, and motives with regard to engagement, found that three out of the four “important” or “very important” motivators endorsed by faculty for their engagement initiatives concerned student learning outcomes related to transfer of learning from knowledge to application, student awareness, and participation in the community (the fourth was meeting genuine community needs). Chesney, Samuel, and Fuller (2009) provided a thorough descriptive evaluation of Tennessee State University’s service-learning and engagement initiatives. They emphasized the reciprocity between the university and community, highlighted the role of the Tennessee Campus Compact, articulated details on grant and program initiatives as well as scholarship of engagement projects, and described criteria for excellence in their university’s Cooperative Extension entity. TSU’s ongoing evaluation of their mission is based on five questions. The first two of these questions ask whether the TSU student is a “better and more effective leader who is socially responsible, cultivates respect for diversity, and encourages compassion for all” and whether the student is a “more responsible and engaged member of the community” (p. 85). Measurement of student learning outcomes with regard to socially responsible, diversity-respecting, and compassionate leadership and engagement bears obvious conceptual similarity to Missouri State University’s emphasis on ethical leadership, cultural competence, and community engagement. While the authors had difficulty finding student assessments of broad engagement with public affairs, especially as defined by Missouri State University’s three pillars, we could find individual measures of specific elements. As described earlier, the assessment of the construct of community engagement has been most associated with institutional assessment for the public good. However, our conceptualization of the Public Affairs Mission at Missouri State University encompasses more than only markers of community engagement. On the other hand, self-assessments of specific elements of ethical leadership and cultural competence were found online through the Center for Ethical Leadership, the American Speech-Language-Hearing Association, and the Alameda County Public Health Department websites, respectively. The Public Affairs Scale was designed to fill this dearth of resources in broad learning outcomes related to public engagement within universities associated with all three pillars of community engagement, cultural competence, and ethical leadership.


699

C. Levesque-Bristol & J. Cornelius-White We also consulted a number of additional sources, including the Association of American Colleges and Universities (AAC&U), the Carnegie Foundation for the Advancement of Teaching, the Centers for Disease Control, Transcultural Nursing, and the U.S. Department of Health and Human Services: Health Resources and Services Administration. These sources provided additional information from organizational philosophies and practices related to the themes of the Public Affairs Mission. In particular, the AAC&U has been engaged in many initiatives related to the identification of essential learning outcomes important for all students that should be fostered and developed across the entire educational experience. The LEAP Essential Learning Outcomes campus toolkit provides a useful conceptualization of elements related to the Public Affairs Mission. As part of the National Survey of Student Engagement (NSSE), five “Benchmarks of Effective Educational Practice” are identified and assessed. The one most closely related to the pillars of the Public Affairs Mission is “Enriching Educational Experience.” It comprises items that are related to our conceptualization of community engagement and cultural competence as well as items that are unrelated to any of the pillars. In addition, the authors consulted Cross, Bazron, Dennis, and Isaacs (1989), which provided a thorough contribution to the concepts of cultural competence. After summarizing the campus-wide indicators of the Public Affairs Mission in the following section, we explain the empirical construction process and analytical strategy followed to develop the Public Affairs Scale to measure all three pillars. Campus-Wide Indicators of the Public Affairs Mission Missouri State University has served as a leader in the American Democracy Project and received recognition from the Princeton Review, the Corporation for National and Community Service, and the Templeton Foundation for its community engagement. The university hosts a yearly Public Affairs Conference with internationally renowned speakers, a yearly Public Affairs Week organized by students, and a Public Affairs Academy for outstanding high school students. Missouri State University organizes yearly common reader programs, reading guides for students, and yearlong activities centered on selected themes (e.g., 2008–2009, Sustainability; 2009–2010, Evolving Economic Realities; 2010–2011, Leading in a Global Society). For about 10 years, ending in 2006, Missouri State University supported the printed Journal of Public Affairs. It typically consisted of one issue per year with 6–12 articles, and according to the masthead was an “interdisciplinary journal that addresses public affairs issues in the broadest sense in academe, the professional world, and personal 700


The Public Affairs Scale experience.” In 2012, Missouri State University re-launched its journal, previously Public Affairs, as the eJournal of Public Affairs. Additionally, Missouri Campus Compact—a collaboration between 35 organizations, colleges, and universities, dedicated to strengthen civic engagement and service learning through partnership with the community—is located at Missouri State University. Missouri Campus Compact reports to the associate provost for Student Development and Public Affairs and will soon be launching a peer-reviewed journal dedicated to public scholarship, the Journal of Public Scholarship in Higher Education (JPSHE). Missouri Campus Compact is part of the larger Campus Compact, which is a coalition of more than 1,100 colleges and universities with more than 6 million students. The Center for Citizenship and Service Learning (CASL) represents one of the most mature and successful aspects of the public affairs mission. While most clearly associated with the community engagement theme, CASL fosters extensive development of students, faculty, and community partners with regard to cultural competence and ethical leadership (Levesque & Strong, 2010). CASL aims to “develop skills, sensitivities and commitments necessary for effective citizenship in a democracy” using an “experiential education combining academic achievement and community service in a seamless weave utilizing reflection exercises” (Citizenship and Service-Learning, 2010). The 2009–2010 annual report shows that over 2,000 students served for over 52,000 hours, which represents $984,344 worth of service that students provided to the community during the 2009–2010 year alone. The university has involved nearly 700 community partners in service learning. Participation in Service-Learning has risen every year since its inception. CASL aims to develop and maintain long-term, positive relationships with community partners. It hosts luncheons and community fairs, and employs several other initiatives that build these bridges. It supports research and development of service-learning quality through its research grants, faculty fellowships, and travel stipends and external grants. National Campus Compact named CASL’s associate director one of 10 Engaged Scholars in May 2008 to participate in an 18-month task force to promote collaborative dialogue, planning, and action on campuses nationally. Aside from service rendered through service learning, university students now volunteer more than 100,000 hours a year. Likewise, the university has an active and rapidly growing international center, which supports several initiatives related to cultural competence and global citizenship, including study away. The development of the Public Affairs Scale serves as a specific measurement tool addressing the student level of analysis. It is designed


701

C. Levesque-Bristol & J. Cornelius-White to assess the goals and learning objectives of the campus-wide indicators and programs related to the three pillars of the Public Affairs Mission presented earlier. The Public Affairs Scale also is designed to assess the improvement change in Community Engagement, Cultural Competence, and Ethical Leadership at the program and university levels.

Methods Procedures and Measures We secured human subjects approval from the Internal Review Board to conduct this study. Following examination of the materials described earlier and compilation of the feedback obtained from conversations with students, staff, administrators, and faculty, a series of 81 items designed to assess the three elements of the Public Affairs Mission at Missouri State University were created. A total of 30 items were created to assess Community Engagement, 22 items to measure Cultural Competence, and 29 items to measure Ethical Leadership. The 83 items of the Socially Responsible Leadership Scale–Revised were also included to further assess the construct validity of the Public Affairs Scale. The Socially Responsible Leadership Scale–Revised (Tyree, 1998) is a previously validated instrument derived from the Social Change Model (Higher Education Research Institute, 1996), which measures some similar constructs as the Public Affairs Scale. It has been used at Missouri State University in Student Activities to educate and assess student leadership initiatives. We administered this survey as a means to assess construct validity of the Public Affairs Scale. The Socially Responsible Leadership Scale–Revised is composed of eight constructs, which represent critical values. These are Change (community/society), Citizenship (community/society), Collaboration (group), Commitment (individual), Common Purpose (group), Congruence (individual), Consciousness of Self (individual), and Controversy with Civility (group). These items were then administered to a sample of 255 undergraduate and graduate students at Missouri State University during the 2008–2009 academic year through Survey Monkey. Participants Participants were obtained from general e-mail advertisements, especially targeting the introduction to psychology subject research pool, and from various undergraduate and graduate classes. Approximately 800 students received the invitation to participate, yielding a response rate of approximately 32%. The sample comprised 70.6% women and 17.6% men. Participants’ ages ranged from 18 to 59 with a mean age of 23. Of the participants, 702


The Public Affairs Scale 80% reported being Caucasians, 1.6% reported being African Americans, less than 1% reported being Asian Americans. Of the students participating, 69.4% were undergraduate students, and the rest were graduate students. The undergraduate students were mostly freshmen (35.7%). In addition, the sample comprised about 10% each of sophomores, juniors, and seniors. The students came from all colleges, and the College of Health and Human Services (28.6%) and the College of Education (23.1%) had the most participants. Analytical Strategy The 81 original items were subjected to an Exploratory Factor Analysis in order to uncover stable underlying factors. Since the original items were created to fit one of the three pillars of the Public Affairs Mission at Missouri State University—Community Engagement, Cultural Competence, and Ethical Leadership—we expected three stable factors to emerge from the factor analysis. The initial data analysis revealed the presence of three possible factors as predicted. We then proceeded to clean up the factor structure of the Public Affairs Scale by deleting items that did not fit the Public Affairs Mission conceptual model. Reducing the number of items to a limited number per defined factor is one of the goals of factor analysis. Items were deleted from the scale for one of the following reasons: (a) items did not significantly load (belong) on their predicted factor, (b) items loaded on the wrong factor, or (c) items loaded on two or three of the factors to a relatively equal degree (cross-loading). Additionally, we kept some items that maintained conceptual breadth within the three areas, but neither significantly improved or reduced the psychometric properties of the scale, leaving 40 of the original 81 items. The final best factorial solution from this data set for the Public Affairs Scale is described next.

Results Items Included in the Final Public Affairs Scale The final scale presented in this paper is composed of 40 items: 15 items for the Community Engagement subscale, 15 items for the Ethical Leadership subscale, and 10 items for the Cultural Competence subscale. The items along with their respective factor loadings are presented in Table 1. Figure 1 includes a user-friendly version of the 40-item scale. This structure explained 42% of the variance in the pattern of relationship among the items. Seven of the included items had significant cross-loadings, but otherwise the structure was fairly clean. The alpha reliabilities of the three subscales were high to adequate: Community Engagement, a = .90; Cultural Competence, a = .75; and Ethical Leadership, a = .92.


703

C. Levesque-Bristol & J. Cornelius-White Table 1. The Items and Three-Factor Structure of the Public Affairs Scale

Factors Community Engagement Reliability (a = .90) 21. Volunteering will help me succeed in my own profession.

.787

19. I plan to do some volunteer work next year.

.774

20. Volunteering makes me feel like I am contributing to the community.

.768

23. I do things for a cause bigger than myself.

.684

26. I feel an obligation to contribute to the community.

.660

17. Volunteering allows me to gain a new perspective on things.

.657

10. I am confident that I will participate in community service activities in the future.

.643

25. Community service is necessary to making our communities better.

.633

8. I know I can make a difference in my community.

.633

27. I feel that I can make a difference in the world.

.633

11. I have worked collaboratively in a student organization to influence my community’s quality of life.

.625

22. Volunteering lets me learn through direct “hands on” experience.

.600

16. People I know share an interest in community service.

.543

3. I participate in the political process.

.353

1. I have had one or more service-learning classes.

.301

Ethical Leadership Reliability (a = .92)

.319

60. When I am in groups, I am thoughtful of other people’s feelings.

.758

54. I am dependable and reliable.

.704

64. I try to make certain that my actions never intentionally harm another person.

.702

57. I am aware of what kind of person I am.

.663

67. When working in groups, I try to assure everyone’s voice is heard before a decision is reached.

.659

65. When making a decision, I weigh the consequences of each alternative.

.634

704


Cultural Competence Reliability (a = .75)

The Public Affairs Scale Table 1. The Items and Three-Factor Structure of the Public Affairs Scale (continued)

Factors

59. In nearly everything I do, I am striving to improve myself and become a better person.

Community Engagement Reliability (a = .90)

Ethical Leadership Reliability (a = .92)

.304

.627

81. I am in part responsible for the quality and quantity of knowledge I obtain from the courses I take.

.621

53. I understand the importance of being true to my word.

.615

68. When I am a leader, I am sure to recognize the accomplishments of others more than myself.

.597

66. When I listen, I regularly try to take someone else’s perspective.

.551

62. When I make mistakes, I am able to admit them.

.540

56. I stand by my decisions even when others protest.

.528

63. I usually take an active interest in someone else’s concerns.

.482

72. I try to build a consensus where everyone finds a decision acceptable before an important change is made.

.474

Cultural Competence Reliability (a = .75)

31. I am able to communicate effectively with people from different cultures.

.535

33. I understand the challenges faced by people from different cultures.

.505

34. I have been involved in organizations providing services to people from different cultural backgrounds.

.387

.497

47. In the future, I will travel to other countries to better understand culture and diversity.

.487

44. I can easily relate to people that are different from me. 51. I have been involved in organizations providing services to diverse populations. 45. Traveling allows me to understand different cultures.

.305

.338 .396

.473 .453

.374

.434

46. I have participated in a study abroad or exchange program.

.355

43. I could not easily get accustomed to living in another country (reverse scored).

.310

36. I avoid imposing values that may conflict with cultural groups other than my own.

.292


705

C. Levesque-Bristol & J. Cornelius-White Figure 1. The Public Affairs Scale

1. 2. 3. 4. 5.

6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18.

19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37.

38. 39. 40.

I have had one or more service-learning classes. [1] I participate in the political process. [3] I know I can make a difference in my community. [8] I am confident that I will participate in community service activities in the future. [10] I have worked collaboratively in a student organization to influence my community’s quality of life. [11] People I know share an interest in community service. [16] Volunteering allows me to gain a new perspective on things. [17] I plan to do some volunteer work next year. [19] Volunteering makes me feel like I am contributing to the community. [20] Volunteering will help me succeed in my own profession. [21] Volunteering lets me learn through direct “hands on” experience. [22] I do things for a cause bigger than myself. [23] Community service is necessary to making our communities better. [25] I feel an obligation to contribute to the community. [26] I feel that I can make a difference in the world. [27] I am able to communicate effectively with people from different cultures. [31] I understand the challenges faced by people from different cultures. [33] I have been involved in organizations providing services to people from different cultural backgrounds. [34] I avoid imposing values that may conflict with cultural groups other than my own. [36] I could not easily get accustomed to living in another country. [43] I can easily relate to people that are different from me. [44] Traveling allows me to understand different cultures. [45] I have participated in a study abroad or exchange program. [46] In the future, I will travel to other countries to better understand culture and diversity. [47] I have been involved in organizations providing services to diverse populations. [51] I understand the importance of being true to my word. [53] I am dependable and reliable. [54] I stand by my decisions even when other protest. [56] I am aware of what kind of person I am. [57] In nearly everything I do, I am striving to improve myself and become a better person. [59] When I am in groups, I am thoughtful of other people’s feelings. [60] When I make mistakes, I am able to admit them. [62] I usually take an active interest in someone else’s concerns. [63] I try to make certain that my actions never intentionally harm another person. [64] When making a decision, I weigh the consequences of each alternative. [65] When I listen, I regularly try to take someone else’s perspective. [66] When working in groups, I try to assure everyone’s voice is heard before a decision is reached. [67] When I am a leader, I am sure to recognize the accomplishments. [68] I try to build a consensus where everyone finds a decision acceptable before an important change is made. [72] I am in part responsible for the quality and quantity of knowledge I obtain from the courses I take. [81]

Note. Item 20 is reverse scored. The original item number is presented in brackets.

706


The Public Affairs Scale Overall Ratings Table 2 presents the means on the three pillars of the Public Affairs Mission. These means represent the degree to which students report endorsing or participating in the elements of the mission. Overall, Cultural Competence was the element of the Public Affairs Mission students endorsed the least. The most strongly endorsed element of the Public Affairs Mission was Ethical Leadership. All three elements of the Public Affairs Scale showed similar degrees of variability (standard deviation) and range of values. Table 2. Mean Ratings and Standard Deviations for the Three Elements of the Public Affairs Mission

Mean

Std. Dev.

Skewness

Minimum

Maximum

Community Engagement

3.93

.58

–.61

1.00

5.00

Ethical Leadership

4.19

.47

–1.24

1.00

5.00

Cultural Competence

3.55

.53

–.45

1.40

5.00

Engagement in Public Affairs and Class Rank Class rank was not strongly correlated with the endorsement of the pillars of the Public Affairs Mission. However, statistically significant correlations were found between class rank and Community Engagement (r = .14, p < .05) as well as between class rank and Cultural Competence (r = .17, p < .01). In general, the direction of the association was such that as class rank increased, so did the endorsement of the pillars of the Public Affairs Mission. This is expected and desired, since we intend for students to better understand and participate in the themes of the Public Affairs Mission as they progress with their studies. However, the correlations were not strong, which suggests that greater emphasis would need to be placed on the Public Affairs Mission at Missouri State University for measurable maturation to be more apparent at the university level. The means by class rank are presented in Table 3.


707

C. Levesque-Bristol & J. Cornelius-White Table 3. Public Affairs Elements Means by Class Rank Class Rank

Cultural Competence


Ethical Leadership

Freshmen

3.45

3.80

4.16

Sophomores Juniors Seniors Graduate

3.53 3.90 3.53 4.11 3.56 4.03 3.69 3.99 Difference between Difference between Freshmen and Graduates Freshmen and Graduates p < .01 p < .05

4.12 4.17 4.13 4.28 Difference between Freshmen and Graduates p = .05

Sufficient Sensitivity Using this scale, the examination of the means and standard deviations shows that there is sufficient room for improvement in the students’ level of endorsement and participation in the university’s mission. Nevertheless, the mean endorsement for respondents on the Ethical Leadership subscale is relatively high, suggesting a possible ceiling effect and less room for change on this element. This may simply represent the students’ belief that they behave ethically most of the time. When students are exposed to programs designed to them think more critically about the meaning and implications of ethical leadership, an initial drop in level of endorsement might be expected. Significant increases in endorsement level would be documented as students develop a greater and more complex understanding of ethical leadership through continued program exposure. Construct Validity Overall ratings for the eight constructs of the Socially Responsible Leadership Scale-Revised are presented in Table 4. The highest means were found for Commitment and Congruence. Table 4. Mean Ratings of the Eight Subscales of the Socially Responsible Leadership Scale

Change Citizenship Collaboration Commitment Common Purpose Congruence Consciousness of Self Controversy

708

Mean

Std. Dev.

Skew

Minimum

Maximum

3.74 3.90 3.98 4.27 3.95 4.10 3.85 3.88

.55 .66 .54 .59 .51 .58 .60 .48

–.05 –.99 –1.21 –1.92 –1.41 –1.46 –.78 –.34

2.00 1.00 1.00 1.00 1.00 1.00 1.00 2.00

5.00 5.00 5.00 5.00 5.00 5.00 5.00 5.00


.547**

.570**

Ethical Leadership

Cultural Competence

.386**

.487**

.624**

.706**

Citizenship

.490**

.688**

.493**

Collaboration

Note. ** Correlation is significant at the 0.01 level (two-tailed).

Public Affairs Scale


Change

.309**

.666**

.374**

Commitment

.398**

.678**

.478**

.389**

.681**

.418**

Common CongruPurpose ence

.404**

.544**

.401**

.529**

.650**

.380**

Conscious- Controversy ness of Self with Civility

Socially Responsible Leadership Scale–Revised

The Public Affairs Scale

The associations between the eight constructs of the Socially Responsible Leadership Scale–Revised and the three elements of the Public Affairs Scale are presented in Table 5. All of the correlations were significant at the p < .001 level, Table 5. Correlations Between the Three Pillars of the Public Affairs and the Constructs of the Socially Responsible Leadership Scale


709

C. Levesque-Bristol & J. Cornelius-White showing a broad validation of the overall Public Affairs Conceptualization and Social Change Model. Likewise, correlations were highest between the Ethical Leadership Scale and all scales of the Socially Responsible Leadership ScaleRevised as compared to the other Public Affairs subscales, with the exception of the correlation between Community Engagement and Citizenship (r = .706) and the correlation between Cultural Competence and Change (r = .570). The second-highest correlation between Cultural Competence and a construct of the Socially Responsible Leadership Scale was with Controversy with Civility (r = .529). The correlations were highest between scales that are conceptually most similar, thus providing empirical validation of the constructs without being too high to signify that the constructs were simply replicated in another scale. These findings offer strong support for the validity of the Public Affairs Scale. Using the Public Affairs Scale to Evaluate a First-Year Service-Learning Program Levesque and Strong (2010) conducted a study using the Public Affairs Scale to assess a first-year service-learning program regarding its effectiveness at meeting learning objectives related to civic engagement and public affairs. This study offered further evidence of the validity of the Public Affairs Scale. The study involved nine sections of an Introduction to University Life course: four of these were integrated service-learning sections, and five were control sections (from a total of 107 sections offered at the university). There were 203 students (87 in the service-learning sections and 116 in the non-servicelearning sections) who completed the time 1 assessment at the beginning of the semester, and 173 students who completed the time 2 assessment at the end of the semester. Among these students, 135 completed both assessments. In addition to the Public Affairs Scale, participants completed a measure of selfefficacy and cognitive communication competence. Data analyses involved mean comparisons between service-learning and control sections at time 2 and change over time. Mean comparisons at time 2 showed that students in the service-learning sections reported significantly higher levels of community engagement, cultural competence, and ethical leadership (p < .01, .05, and .01 respectively). Students in the service-learning sections at time 2 also showed higher levels of self-efficacy (p < .05) but similar levels of cognitive communication competence. Analyses of change over time also showed significant increases for the service-learning group with regard to community engagement and cultural competence (p < .001 and .05, respectively). Ethical leadership and self-efficacy did increase marginally over time, although this improvement was not statistically significant. While showing significantly less improvement than the service-learning group, the control group did show significant improvement on the community engagement dimension of the Public Affairs Scale from time 1 to time 2. 710


The Public Affairs Scale This study shows how the Public Affairs Scale has been and can be used to evaluate academic programs with a public affairs focus and the attainment of student learning objectives related to the three pillars of the Public Affairs Mission: Community Engagement, Cultural Competence, and Ethical Leadership. In this case, the scale was used to evaluate a first-year service-learning program. It was sensitive to change, more so than the other measures used, and showed changes in the expected directions. The students in the service-learning sections in comparison to students in the control sections did show greater endorsement of the Public Affairs themes at time 2 as well as greater change over time.

Discussion Limitations While sample size is sufficient for the factor analytic method employed, the study is nevertheless limited by its sample. The sample may not be completely representative, given its size in relation to the full student body population (about 22,000 overall, and 16,000 on the main campus). The survey was sent to approximately 800 students, including all the students who were part of the Introduction to Psychology subject pool, for a response rate of about 32%. Consequently, the sample was heavily weighted with freshmen. Due to the large sample of freshmen, the results may be more applicable to younger students even though the psychometric properties did appear similar for all class ranks, including graduate students. Additionally, the sample was mostly Caucasian (80%) or unidentified with regard to race (15%), providing a potential limitation for more diverse samples. It is also important to note that Missouri State has a strong mission in Public Affairs that is promoted throughout academic departments and nonacademic student organizations. Because of the many initiatives focused on the elements of the Public Affairs Mission, Missouri State may not be the typical institution of higher education with regard to an emphasis in civic engagement. Future studies should seek to use the Public Affairs Scale in institutions of higher education that do not have a specifically stated Public Affairs mission and consequently extend the results to more general institutions of higher education. However, even in an institution with a strong focus in public affairs, variation was found in responses to the elements of the Public Affairs Scale. The study examines students’ endorsement of dispositions and experiences, but does not address other aspects of program evaluation necessary to assess something as broad as a university’s mission (Westdijk et al., 2010). To help provide this broader context, the background information reviewed other markers of the Public Affairs Mission manifestation at Missouri State University and how other universities have approached mission evaluation and accountability. Similarly, the Public Affairs Scale is a self-report measure


711

C. Levesque-Bristol & J. Cornelius-White and is subject to typical self-rating biases. Nevertheless, the scale provides a much-needed and more direct measure of the success of the mission in its most important constituent group, students. Implications of the Results as a Program Evaluation This validation project serves as a pilot program evaluation of the university’s fulfillment of the Public Affairs Mission with students comprising a relatively small sample. The results show that on average, students endorsed items in the desired direction but less so on the Cultural Competence subscale than the Community Engagement or Ethical Leadership subscales. Students did not consistently respond to the items intended to measure cultural competence as compared to the other themes. This result may signify that students do not necessarily get exposure or understand how to engage with cultural competence in a consistent manner. These findings are consistent with an analysis of the diversity climate data for the university (Levesque & Cornelius-White, 2009), which showed that different community constituencies (e.g., minority respondents, Caucasian respondents, respondents with disabilities, etc.) tend to see the university climate in different ways and that some groups do not appear to understand or strongly endorse the need for recruitment and retention of diverse individuals. Nonetheless, students at all levels (i.e., freshmen, sophomores, juniors, seniors, and graduate) generally appear engaged with the mission. Future of Programmatic Evaluation The main benefit of the Public Affairs Scale is that it provides a collaboratively constructed and empirically validated method of exploring civic engagement missions at Missouri State and at other universities. While further revisions may occur in using the same collaborative and empirical processes at Missouri State University or elsewhere, in its current form, the Public Affairs Scale does represent a reliable and valid instrument. It is user-friendly and of reasonable length (40 items fitting easily onto one page), shows evidence of strong to adequate internal consistency, and has face validity as well as convergent validity. It appears sufficiently sensitive to capture differences between students and within students over time. If used systematically, it can assess the extent to which students are engaged with the mission in general and over time. In this way, the scale can be used to evaluate how both students and the fulfillment of the mission may mature, for such purposes as strategic planning and the public scorecard. For example, the Public Affairs Scale is currently used in several research projects assessing the impact of the newly developed Freshmen Seminar, which now offers sections integrating the Seven Revolutions (CSIS, 2011) content with integrated service learning. The Public Affairs Scale will also be administered to every entering freshman and to graduating seniors beginning in Fall 2012 to track and assess student development related 712


The Public Affairs Scale to the three pillars of the Public Affairs Mission. At the unit level (i.e., college, department, program), the assessment can be used to describe how the mission is being met within more homogenous groups and to serve as an evaluation of the efficacy of specific interventions or initiatives. At Missouri State University, all programs are encouraged to develop measurable public affairs learning outcomes. Some programs are considering the use of the Public Affairs Scale as an indirect measure of the realization and attainment of the learning objectives of the Public Affairs Mission by majors.

Conclusion As outlined in the introduction, the development of the conceptualization of the Public Affairs Mission took many years. Of importance is the fact that many public affairs activities and initiatives were occurring on campus without a clear framework to foster an intentional realization of the mission. The conceptualization of the Public Affairs Mission—including the three pillars of Community Engagement, Cultural Competence, and Ethical Leadership— provided a clear framework for Missouri State University. As other institutions of higher education are attempting to meet civic engagement and public affairs goals, we would recommend the establishment of a clear and formal structure that would be agreed upon through discussions and reflections on the nature of civic engagement at each particular institution. The creation of the Public Affairs Scale provided a valid and reliable way to empirically examine and measure progress toward the realization of the more clearly defined mission. It allowed us to quantify the baseline civic engagement and public affairs understanding of students participating in public affairs initiatives and programs. It will also enable us to do the same for the entire student population. Over time, it will allow us to measure change and progress toward the realization of the public good mission of higher education. Through this process, we have already noted that some programs designed to help meet the Public Affairs Mission work better with some students than others. By using a validated scale like the Public Affairs Scale, we are now able to begin identifying the types of public affairs or civic engagement programs that are most impactful for specific groups of students and consequently target our interventions in a much more meaningful and systematic way. While the mission at Missouri State is unique, it is in many respects more similar to than different from what many view as the broad mission of higher education. Whether we go back to Dewey 100 years ago or to the writings and vision of the Carnegie Foundation for the Advancement of Teaching (2006), the civic mission to create a more democratic society is apparent. Consequently, the Public Affairs Scale with the three subscales of Community Engagement, Cultural Competence, and Ethical Leadership could be used by other


713

C. Levesque-Bristol & J. Cornelius-White institutions of higher education to assess the realization of the broad mission of higher education. For example, other institutions could use the Public Affairs Scale to assess overall baseline realization of the broad public good mission as well as improvement over time. The Public Affairs Scale could also be used to assess the effectiveness of service-learning or similar programs, especially by using information gathered from specific subscales (e.g., Community Engagement). In conclusion, perhaps the largest benefit of the Public Affairs Scale is in its unique student-level assessment contribution. Other scales measure relevant institutional goals, but none measures all three pillars of the Missouri State public affairs conception. Likewise, this individual-level scale can be used within a comprehensive, continuous improvement process in which planning, evaluation, and follow-through are linked. Within a shared or emphasized focus on the cultural, ethical, leadership, and community interaction aspects of outreach and engagement, the Public Affairs Scale provides an additional resource among a breadth of potential program evaluation and research tools. We encourage faculty, staff, administrators, and students to use the Public Affairs Scale to assess their outreach and engagement activities and use the results for program modifications and improvements.

References

Carnegie Foundation for the Advancement of Teaching. (2006). Higher education: Civic mission and civic effects. Retrieved from www.carnegiefoundation.org/sites/default/files/publications/elibrary_ pdf_633.pdf Center for Ethical Leadership. (2007). The 4-V model of ethical leadership. Retrieved from http:// ethicalleadership.org/about-us/philosophies-definitions/ethical-leadership Center for Strategic & International Studies (CSIS). (2011). Seven Revolutions: Part of the Global Strategy Institute. Retrieved from http://csis.org/program/seven-revolutions Centers for Disease Control and Prevention. (1997). Principles of community engagement. Atlanta, GA: CDC/ATSDR Committee on Community Engagement. Retrieved from www.cdc.gov/ phppo/pce/part1.htm Citizenship and Service-Learning. (2010). Missouri State University. Retrieved from www. missouristate.edu/casl/ Chambers, T., & Gopaul, B. (2008). Decoding the public good of higher education. Journal of Higher Education Outreach and Engagement, 12 (4), 59–91. Chesney, C. E., Samuel, J., & Fuller, D. S. (2009). Outreach and engagement at Tennessee State University. Journal of Higher Education Outreach and Engagement, 13(3), 67–88. Cornelius-White, J. H. D., & Levesque, C. (2008). Initial synthesis of public affairs operationalization and development: Consideration of college Public Affairs Inventories. Unpublished manuscript, Missouri State University.

714


The Public Affairs Scale Cross, T., Bazron, B., Dennis, K., & Isaacs, M. (1989). Toward a culturally competent system of care, volume 1. Washington, DC: Georgetown University. Darabi, R. (2010, May). Public affairs essay. Missouri State Magazine, 5, 1. Retrieved from http:// magazine.missouristate.edu/2010/02/01/public-affairs-essay/ Dewey, J. (1916). Democracy and education: An introduction to the philosophy of education. New York: Free Press. Furco, A. (2001). Advancing service-learning at research universities. In M. Canada and B. W. Speck (Eds.), Developing and implementing service learning programs (New Directions for Higher Education, 114, 67–78). San Francisco: Jossey-Bass. Gelmon, S. B., Seifer, S. D., Kauper-Brown, J., & Mikkelsen, M. (2005) Building capacity for community engagement: Institutional self-assessment. Seattle, WA: Community-Campus Partnerships for Health. Retrieved from http://depts.washington.edu/ccph/pdf_files/selfassessment-copyright.pdf Higher Education Research Institute. (1996). A social change model of leadership development: Guidebook Version III. Los Angeles: Regents of the University of California. Holland, B. (1997). Analyzing institutional commitment to service: A model of key organizational factors. Michigan Journal of Community Service Learning, 4, 30–41. King, M. A., Sims, A., & Osher, D. (2010). How is cultural competence integrated in education? Center for Effective Collaboration and Practice. Retrieved from http://cecp.air.org/cultural/ Q_integrated.htm#def Levesque, C., & Cornelius-White, J. H. D. (2009). Further analysis of the 2007–2008 President’s Commission Diversity Climate Assessment. Unpublished manuscript, Missouri State University. Levesque, C., & Strong, E. C. (2010, September). Service-learning: The first-year experience. Presentation at the Annual Regional Campus Compact Conference, Tulsa, OK. London, S. (2002). The civic mission of higher education: From outreach to engagement. A report from the workshop on higher education and public life, Kettering Foundation. Retrieved from www.scottlondon.com/reports/seminar2001.pdf Public Affairs Mission. (2010). Missouri State University. Retrieved from http://publicaffairs. missouristate.edu/mission.htm Transcultural Nursing. (2010). Cultural competence: Basic concepts and case studies. Retrieved from www.culturediversity.org/cultcomp.htm Tyree, T. M. (1998). Designing an instrument to measure socially responsible leadership using the social change model of leadership development. Unpublished doctoral dissertation, University of Maryland, College Park, MD. Westdijk, K., Koliba, C., & Hamshaw, K. (2010). Collecting data to inform decision making and action: The University of Vermont’s Faculty Engagement Tool. Journal of Higher Education Outreach and Engagement, 14(2), 5–34. Wortis, N., Goldstein, Varagas, R. A., & Grumback, K. (2007). Engaging a university in self-assessment and strategic planning to build partnership capacity: The UCSF experience. Journal of Higher Education Outreach and Engagement, 11(2), 121–138.


715

C. Levesque-Bristol & J. Cornelius-White Chantal Levesque-Bristol, PhD, is professor of Educational Studies and director of the Center for Instructional Excellence (CIE) at Purdue University. She is the former director of the Faculty Center for Teaching and Learning at Missouri State University. Using the theoretical framework of self-determination theory, she conducts basic research in motivation, and applied research in education, learning, student retention, and service learning, and has been active in several teaching and learning programs. As director of the CIE, she provides support for the instructional community and resources to faculty interested in learning pedagogies and the Scholarship of Teaching and Learning. CIE also oversees service learning at Purdue University. She has received several teaching awards and research awards. She is a grantee of the National Institute of Mental Health. She has lectured at several institutions of higher education on motivation and learning principles. Jeffrey H. D. Cornelius-White, PsyD, LPC, is professor of Counseling at Missouri State University and doctoral faculty in Educational Leadership and Policy Analysis at the University of Missouri–Columbia. He is the former editor of The Person-Centered Journal (2002–2011), and chair of the World Association for Person-Centered and Experiential Psychotherapy and Counseling (2008– 2010). He has published approximately 75 works, including Learner-Centered Instruction (Sage, 2010) and Carl Rogers: The China Diary (PCCS Books, 2012). Jef is currently working to complete coediting the Interdisciplinary Handbook of the Person-Centered Approach (Springer, 2012). His research has been mostly concerned with humanistic and cross-cultural education and counseling.

716
