NSSE Validation RUNNING HEAD: NSSE Validation ... - CiteSeerX

NSSE Validation RUNNING HEAD: NSSE Validation

Validation of the NSSE Benchmarks and Deep Approaches to Learning Against Liberal Arts Outcomes

Ernest T. Pascarella and Tricia A. Seifert The University of Iowa Charles Blaich Center of Inquiry in the Liberal Arts at Wabash College

Please direct questions regarding this paper to: Ernest T. Pascarella, The University of Iowa, College of Education, N491 Lindquist Center, Iowa City, IA 52242, [email protected]

This research was supported by a generous grant from the Center of Inquiry in the Liberal Arts at Wabash College to the Center for Research on Undergraduate Education at The University of Iowa.

Center of Inquiry in the Liberal Arts at Wabash College • www.liberalarts.wabash.edu

NSSE Validation

2

ABSTRACT This study examines the validity of the National Study of Student Engagement (NSSE) measures of good practices in predicting outcomes associated with a liberal arts education. Using a multi-institutional sample and a pretest-posttest longitudinal design, we find that net of student background characteristics, the type of institution attended, and other college experiences, one or more of the NSSE measures of good practices in undergraduate education consistently predicted first year development in students’ effective reasoning and problem solving, well-being, inclination to inquire and lifelong learning, intercultural effectiveness, leadership, moral character, and integration of learning. We find that, with institutions as the unit of analysis, the NSSE measures of good practices had a number of substantial positive associations with liberal arts outcomes that persist even in the presence of controls for a precollege measure of each outcome. The findings lend support for the assumption that the widely used NSSE scales are, in fact, measuring practices that positively influence undergraduate cognitive and personal development across a broad range of outcomes.


NSSE Validation

3

Since its inception, the National Survey of Student Engagement (NSSE) has clearly become one of the most broad-based annual surveys of undergraduates in the country. According to the NSSE 2007 Annual Report (“Experiences That Matter: Enhancing Student Learning and Success,” 2008), the NSSE survey has been completed by nearly 1.5 million students at nearly 1,200 different colleges and universities in the last decade. In 2008 alone, 774 different colleges and universities are participating in the annual spring administration of the 15-20 minute survey. The NSSE is specifically designed to measure the extent to which college students are engaged in empirically-vetted good practices in undergraduate education. Indeed, one of the major assumptions of the NSSE is that in measuring such good practices, one is essentially measuring experiences which yield desired student cognitive and personal development during college. Thus, other things being equal, the greater one’s engagement in, or exposure to these good practices the more developmentally influential one’s undergraduate education—or so the logic goes. In this paper, we analyze longitudinal data from the Wabash National Study of Liberal Arts Education to validate NSSE measures of good practices in undergraduate education against a range of liberal arts outcomes. We find that net of student background characteristics, the type of institution attended, and other college experiences, one or more of the NSSE measures of good practices in undergraduate education consistently predicted first year development in students’ effective reasoning and problem solving, well-being, inclination to inquire and lifelong learning, intercultural effectiveness, leadership, and moral character. We find that, with institutions as the unit of analysis, the NSSE measures of good practices had a number of substantial positive associations with liberal arts outcomes that persist even in the presence of controls for a precollege measure of each outcome. With more extensive controls in place at the


NSSE Validation

4

individual student level of analysis, each of the liberal arts outcomes considered was significantly, if modestly, influenced by at least one NSSE good practice scale. The findings lend a modicum of support for the assumption that the widely used NSSE scales are, in fact, measuring practices that positively influence undergraduate cognitive and personal development across a broad range of outcomes. Good Practices in Undergraduate Education In the late 1980s and early 1990s, Chickering and Gamson (1987, 1991) synthesized much of the evidence on the impact of college on students and translated it into seven principles for good practice in undergraduate education. These seven principles are: a) student faculty contact; b) cooperation among students; c) active learning; d) prompt feedback to students; e) time on task; f) high academic expectations; and g) respect for diverse students and diverse ways of knowing. From an empirical standpoint, these seven dimensions of good practice stand solid. Even in the presence of controls for salient confounding influences, various measures of individual good practice dimensions have been found to be significantly and positively linked to desired aspects of cognitive and noncognitive growth during college and to career and personal benefits after college (Astin, 1993; Chickering & Reisser, 1993; Kuh, Schuh, Whitt, & Associates, 1991; Kuh, Kinzie, Schuh, Whitt, & Associates, 2005; Pascarella & Terenzini, 1991, 2005; Seifert, Goodman, Lindsay, Jorgensen, Wolniak, Pascarella, & Blaich, 2008). Examples of individual studies supporting the predictive validity of specific good practices in undergraduate education would include the following: a) student-faculty interaction (Anaya, 1999; Avalos, 1996; Kuh & Hu, 2001; Terenzini, Springer, Yaeger, Pascarella, & Nora, 1994); b) cooperation among students/cooperative learning (Cabrera, Crissman, Bernal, Nora, Terenzini, & Pascarella, 2002; Johnson, Johnson, & Smith, 1998a, 1998b; Qin, Johnson, & Johnson, 1995); c) active


NSSE Validation

5

learning (Hake, 1998; Kuh, Pace, & Vesper, 1997; Lang, 1996; Murry & Lang, 1997); d) prompt feedback to students (d’Apollonia & Abrami, 1997; Feldman, 1997); e) academic effort/time on task (Astin, 1993; Hagedorn, Siadat, Nora, & Pascarella, 1997; Johnstone, Ashbaugh, & Warfield, 2002); f) high academic expectations (Arnold, Kuh, Vesper, & Schuh, 1993; Astin, 1993; Cruce, Wolniak, Seifert, & Pascarella, 2006; Whitmire & Lawrence, 1996); and g) diversity experiences (Gurin, Dey, Hurtado, & Gurin, 2002; Pascarella, Palmer, Moye, & Pierson, 2001; Terenzini, et al., 1994; Umbach & Kuh, 2006). Measuring Vetted Good Practices with the National Survey of Student Engagement The National Survey of Student Engagement (or NSSE) is a 15-20 minute self-report instrument specifically designed to measure the various good practices described above (Kuh, 2001). At the heart of NSSE are a series of items that ask students to indicate how much they are involved in specific academic and nonacademic activities and programs, the nature of their interactions with faculty and other students, the degree of intellectual challenge in their academic work, their involvement in diversity-related experiences, their perceptions of the supportiveness of the campus environment, and the like. Based on these student reports, NSSE provides participating colleges and universities with a comprehensive assessment of the extent to which their students are engaged in activities, or exposed to practices shown by the existing body of research to enhance the impact of an undergraduate education. Over time, NSSE has developed various scales or indexes underlying the individual items in the survey instrument. The most prominent and frequently reported are the five NSSE Benchmarks of Effective Educational Practice (hereafter referred to as benchmarks). These are: Level of Academic Challenge, Active and Collaborative Learning, Student-Faculty Interaction, Enriching Educational Experiences, and Supportive Campus Environment (NSSE, 2006). Level


NSSE Validation

6

of Academic Challenge is an eleven-item scale that measures time spent preparing for class, amount of reading and writing, deep learning, and institutional expectations for academic performance. Active and Collaborative Learning is a seven-item scale that measures extent of class participation, working collaboratively with other students inside and outside of class, tutoring and involvement with a community-based project. The Student-Faculty Interaction scale consists of six items and measures extent of interaction with faculty members and advisors, discussing ideas from classes with faculty members outside of class, getting prompt feedback on academic performance, and working with faculty on a research project. Enriching Educational Experiences is a scale with twelve items that measures extent of interaction with students of different racial or ethnic backgrounds or with different values or political opinions, using information technology, and participating in activities such as internships, community service, study abroad, and co-curricular activities. Finally, Supportive Campus Environment is a six-item scale measuring the extent to which students perceive the campus helps them succeed academically and socially, assists them in coping with nonacademic responsibilities, and promotes supportive relations among students and their peers, faculty members, and administrative personnel and offices. [The above descriptions of the five benchmark scales were taken from the College Student Report, 2006 Codebook developed by NSSE (2006).] Table 1 provides the specific items constituting each of the NSSE benchmark scales and the scale reliabilities from the present study’s sample. Table 1 about here Though more recent and less prominent than the benchmarks, three additional scales developed by NSSE seek to measure “deep approaches to learning” (hereafter the deep learning scales) (Nelson Laird, Shoup, & Kuh, 2006; Nelson Laird, Shoup, Kuh, & Schwartz, 2008). The


NSSE Validation

7

three scales, some of which use items from the benchmarks are: Higher-Order Learning, Integrative Learning, and Reflective Learning. According to Nelson Laird et al. (2008), the fouritem Higher-Order Learning Scale “focuses on the amount students believe that their courses emphasize advanced thinking skills such as analyzing the basic elements of an idea, experience, or theory and synthesizing ideas, information, or experiences into new, more complex interpretations” (p. 477). The Integrative Learning Scale consists of five items and measures “the amount students participate in activities that require integrating ideas from various sources, including diverse perspectives in their academic work, and discussing ideas with others outside of class” (p.477). Finally, the three-item Reflective Learning Scale asks “how often students examined the strengths and weaknesses of their own views and learned something that changed their understanding” (p. 477). Nelson Laird and his colleagues have also used the items from the three scales to create an overall deep learning scale. The deep learning items also yield a total score which combines all 12 items. We present the specific items constituting each of the three deep learning subscales and the overall scale along with the scale reliabilities from the present sample in Table 2. Table 2 about here Despite its relatively broad-based national use, it seems reasonable to ask if good practices in undergraduate education as measured by the rather brief NSSE instrument actually do predict desired educational outcomes. With some narrowly focused exceptions (Carini, Kuh, & Klein, 2006; LaNasa, Olson, & Alleman, 2007), however, nearly all the predictive validity evidence in this regard is based on studies that link the various NSSE measures of good practices to student self-reported gains in intellectual and personal development that are assessed by a set of 16 items near the end of the NSSE instrument itself (e.g., Hayek, Carini, O’Day, & Kuh,


NSSE Validation

8

2002; Kuh & Gonyea, 2003; Pike, 2006; Pike & Kuh, 2005; Pike, Kuh, & Gonyea, 2007; Zhao & Kuh, 2004; Nelson Laird, et al, 2008; Umbach & Kuh, 2003; Umbach & Wawrzynski, 2004). Although such self-reported gains can be formed into psychometrically reliable scales, there are serious problems with the internal validity of any findings in which self-reported gains are employed as an outcome or criterion measure (Pascarella, 2001). The key problem with using the NSSE self-reported gains as a criterion is that they are assessed cross-sectionally as one completes the NSSE instrument during college. Consequently, no precollege assessment of selfreported gains is available to take into account a student’s receptiveness to educational experiences as reflected by his or her response propensity on these types of items. Two students having the same educational experience could report substantially different gains because they enter college differentially open or receptive to the effects of postsecondary education. Absent a precollege measure of the students’ response propensities on self-reported gains items (e.g., selfreported gains during high school), it is nearly impossible to take this differential receptiveness to educational experiences into account. Thus, using the NSSE self-reported gains in college as a validity measure for good practices runs a high risk of confounding the effects of exposure to good practices with the particular individual characteristics of the students an institution attracts and admits (Astin & Lee, 2003; Pascarella, 2001). The result is that, at present, we may have only a very limited body of internally valid evidence with respect to the actual predictive validity of the NSSE. This is a serious issue if participating institutions are asked to view the NSSE scales as a proxy for practices in undergraduate education that facilitate student growth in broadbased educational outcomes. Our study addressed the relative paucity of predictive validity for the NSSE by estimating the extent to which the NSSE benchmark and deep learning scales


NSSE Validation

9

predict net change during the first year of college across six dimensions of objectively measured liberal arts outcomes. Methods Samples Institutional Sample. The sample in the study consisted of incoming first-year students at 19 four-year and two-year colleges and universities located in 11 different states from 4 general regions of the United States: Northeast, Southeast, Midwest, and Pacific Coast. Institutions were selected from more than 60 colleges and universities responding to a national invitation to participate in the Wabash National Study of Liberal Arts Education (WNSLAE). Funded by the Center of Inquiry in the Liberal Arts at Wabash College, the WNSLAE is a large, longitudinal investigation of the effects of liberal arts colleges and liberal arts experiences on the cognitive and personal outcomes theoretically associated with a liberal arts education. The institutions were selected to represent differences in college and universities nationwide on a variety of characteristics including institutional type and control, size, location, and patterns of student residence. However, because the study was primarily concerned with the impacts of liberal arts colleges and liberal arts experiences, liberal arts colleges were purposefully over-represented. Our selection technique produced a sample with a wide range of academic selectivity, from some of the most selective institutions in the country to some that were essentially open admissions. There was also substantial variability in undergraduate enrollment, from institutions with entering classes between 3,000 and 6,000, to institutions with entering classes between 250 and 500. According to the 2007 Carnegie Classification of Institutions, 3 of the participating institutions were considered research universities, 3 were regional universities that did not grant the doctorate, 2 were two-year community colleges, and 11 were liberal arts colleges.


NSSE Validation

10

Student Sample. The individuals in the sample were first-year, full-time undergraduate students participating in the WNSLAE at each of the 19 institutions in the study. The initial sample was selected in either of two ways. First, for larger institutions, it was selected randomly from the incoming first-year class at each institution. The only exception to this was at the largest participating institution in the study, where the sample was selected randomly from the incoming class in the College of Arts and Sciences. Second, for a number of the smallest institutions in the study—all liberal arts colleges—the sample was the entire incoming first-year class. The students in the sample were invited to participate in a national longitudinal study examining how a college education affects students, with the goal of improving the undergraduate experience. They were informed that they would receive a monetary stipend for their participation in each data collection, and were also assured in writing that any information they provided would be kept in the strictest confidence and never become part of their institutional records. Data Collection Initial Data Collection. The initial data collection was conducted in the early fall of 2006 with 4,501 students from the 19 institutions. This first data collection lasted between 90-100 minutes and students were paid a stipend of $50 each for their participation. The data collected included a WNSLAE precollege survey that sought information on student demographic characteristics, family background, high school experiences, political orientation, educational degree plans, and the like. Students also completed a series of instruments that measured dimensions of cognitive and personal development theoretically associated with a liberal arts education. These are described in greater detail in the subsequent section on “WNSLAE Outcomes/Dependent Measures.”


NSSE Validation

11

Follow-up Data Collection. The follow-up data collection was conducted in spring 2007. This data collection took about two hours and participating students were paid an additional stipend of $50 each. Two types of data were collected. The first was based on questionnaire instruments that collected extensive information on students’ experience of college. Two complementary instruments were used: the National Survey of Student Engagement (NSSE), previously described, and the WNSLAE Student Experiences Survey (WSES). However, for the purposes of this study, we focus on information provided by the NSSE. The second type of data collected consisted of follow-up (or posttest) measures of the instruments measuring dimensions of cognitive and personal development that were first completed in the initial data collection. All students completed the NSSE and WSES prior to completing the follow-up instruments assessing cognitive and personal development. Both the initial and follow-up data collections were administered and conducted by ACT (formerly the American College Testing Program). Of the original sample of 4,501 students who participated in the fall 2006 testing, 3,081 participated in the spring 2007 follow-up data collection, for a response rate of 68.5%. These 3,081 students represented 16.2% of the total population of incoming first-year students at the 19 participating institutions. To provide at least some adjustment for potential response bias by sex, race, academic ability, and institution in the sample of students, a weighting algorithm was developed. Using information provided by each institution on sex, race, and ACT score (or appropriate SAT equivalent or COMPASS score equivalent for community college students), follow-up participants were weighted up to each institution’s first-year undergraduate population by sex (male or female), race (Caucasian, African American/Black, Hispanic/Latino, Asian/Pacific Islander, or other), and ACT (or equivalent score) quartile. While applying weights


NSSE Validation

12

in this manner has the effect of making the overall sample more similar to the population from which it was drawn, it cannot adjust for nonresponse bias. Conceptual Framework for Liberal Arts Outcomes Since the WNSLAE was fundamentally concerned with understanding the conditions and experiences that constituted an influential liberal arts education, its first task was to conceptually define the desired cognitive and personal outcomes of such an education. Synthesizing much of the literature on liberal arts education, and building on the work of Jones and McEwen (2000), King, Kendall Brown, Lindsay, and VanHecke (2007) developed a comprehensive model of liberal arts outcomes that embraced seven general dimensions: effective reasoning and problem solving, well-being, inclination to inquire and lifelong learning, intercultural effectiveness, leadership, moral character, and integration of learning. Although such outcome dimensions appear central to the undergraduate mission of a large cross-section of American colleges and universities (see, for example, the outcome taxonomy employed by Pascarella & Terenzini, 1991, 2005, in organizing college impact outcomes), the distinctiveness of the liberal arts outcomes lies in the integrated connections that exist between outcomes and their holistic nature, which spans cognitive, interpersonal, and intrapersonal domains. Consequently, the WNSLAE was largely guided by this conceptual framework of liberal arts outcomes in selecting specific outcome measures. Indeed, with the single exception of integration of learning, the WNSLAE study was able to identify specific outcome or dependent measures representing six of the seven liberal arts outcomes specified by the King et al. conceptual model. WNSLAE Outcome/Dependent Measures Effective Reasoning and Problem Solving. To tap this outcome, we used the critical thinking module from the Collegiate Assessment of Academic Proficiency (CAAP) developed


NSSE Validation

13

by the American College Testing Program (ACT). The critical thinking test is a 40-minute, 32item instrument designed to measure a student’s ability to clarify, analyze, evaluate, and extend arguments. The test consists of four passages in a variety of formats (e.g., case studies, debates, dialogues, experimental results, statistical arguments, editorials). Each passage contains a series of arguments that support a general conclusion and a set of multiple-choice test items. The internal consistency reliabilities for the CAAP critical thinking test range between .81 and .82 (ACT, 1991). It correlates .75 with the Watson-Glaser Critical Thinking Appraisal (Pascarella, Bohr, Nora, & Terenzini, 1995). Well Being. We operationalized this dimension of liberal arts outcomes with several individual measures. The first was the Ryff Scales of Psychological Well-Being (SPWB) (Ryff, 1989; Ryff & Keys, 1995). The SPWB is a 54-item, theoretically-grounded instrument that specifically focuses on measuring six dimensions of psychological well-being: positive evaluations of oneself (Self-Acceptance), sense of continued growth and development as a person (Personal Growth), belief in a purposeful and meaningful life (Purpose in Life), quality relations with others (Positive Relations with Others), capacity to effectively manage one’s life and surrounding world (Environmental Mastery), and sense of self-determination (Autonomy) (Ryff & Keyes, 1995; Ryff, 1989; Keyes, Shmotkin, & Ryff, 2002). Due to recent concerns about the construct validity and interpretation of the six subscales (Springer & Hauser, 2006; Springer, Hauser, & Freese, 2006), we used the total score in this study. The total score for the SPWB had a reliability of .88. The SPWB tends to have significant, positive associations with frequently used measures of happiness and satisfaction and negative associations with depression (Ryff & Keyes, 1995).


NSSE Validation

14

Inclination to Inquire and Lifelong Learning. This outcome was operationally represented with two scales. The primary measure was the 18-item Need for Cognition Scale (NCS). Need for cognition refers to an individual’s “tendency to engage in and enjoy effortful cognitive activity” (Cacioppo, Petty, Feinstein, & Jarvis, 1996, p. 197). Those who have a high need for cognition “tend to seek, acquire, think about, reflect back on information to make sense of stimuli, relationships, and events in their world” (p. 198). In contrast, those with low need for cognition are more likely to rely on others, such as celebrities and experts, cognitive heuristics, or social comparison processes to provide or make sense of their world. The reliability of the NCS ranges from .83 to .91 in samples of undergraduate students (Cacioppo et al., 1996). With samples of undergraduates, the NCS has been positively associated with the tendency to generate complex attributions for human behavior, high levels of verbal ability, engagement in evaluative responding, one’s desire to maximize information gained rather than maintain one’s perceived reality (Cacioppo et al., 1996) and college grades (Elias & Loomis, 2002). The NCS is negatively linked with authoritarianism, need for closure, personal need for structure, the tendency to respond to information reception tasks with anxiety, and chronic concern regarding self-presentation (Cacioppo et al., 1996). The second measure designed to tap continuing motivation for lifelong learning was a 6item measure entitled the Positive Attitude Toward Literacy Scale (PATL). The PATL assesses students’ enjoyment of such literacy activities as reading poetry and literature, reading scientific and historical material, and expressing ideas in writing, and has an internal consistency reliability of .71. The PATL score at entrance to college correlated .36 with three-year cumulative scores during college on a measure of library use, .48 with the cumulative number of unassigned books


NSSE Validation

15

read during three years of college, and .26 with a measure of reading comprehension administered after three years of college (Bray, Pascarella, & Pierson, 2004). Intercultural Effectiveness. This outcome dimension was measured with two scales. The primary measure was the 15-item, short form of the Miville-Guzman Universality-Diversity Scale (M-GUDS). The M-GUDS measures an individual’s universal-diverse orientation, which is defined as “an attitude of awareness and acceptance of both similarities and differences that exist among people” (Miville, Gelso, Pannu, Liu, Touradji, Holloway, & Fuertes 1999; Fuertes, Miville, Mohr, Sedlacek, & Gretchen, 2000). The instrument has a total scale score and three subscale scores: Diversity of Contact (interest and commitment to participating in diverse, intentionally focused social and cultural activities), Relativistic Appreciation (appreciation of both similarities and differences in people and the impact of these in one’s self-understanding and personal growth), and Comfort with Differences (the degree of comfort with diverse individuals). The internal consistency reliability for the total M-GUDS score in the present study was .85, while reliabilities for the three subscales ranged from .77 to .78. The precollege total MGUDS score correlated .47 with a measure of students’ experiences and interactions with diverse others and diverse ideas during the first year of college. In the present study, we used the MGUDS total score. The second instrument used to assess student growth in intercultural effectiveness was the seven-item Openness to Diversity/Challenge (ODC) scale. This scale measures one’s openness to cultural and racial diversity as well as the extent to which one enjoys being challenged by different perspectives, values, and ideas (Pascarella, Edison, Nora, Hagedorn, & Terenzini, 1996). The ODC has internal consistence reliabilities in the present study ranging from .83 to .87, depending on the data collection wave. In previous research, precollege ODC


NSSE Validation

16

scores have significantly predicted the likelihood of participating in a racial/cultural workshop during the first year of college (Whitt, Edison, Pascarella, Terenzini, & Nora, 2001). In the present study, precollege ODC scores correlated .37 with a measure of students’ experiences and interactions with diverse others and diverse ideas during the first year of college. Leadership. This outcome dimension was assessed with the 68-item, revised version II of the Socially Responsible Leadership Scale (SRLS). The SRLS measures the eight dimensions of Astin’s Social Change Model of leadership development (Astin, A., Astin, H., Boatsman, Bonous-Hammarth, Chambers, Goldberg, et al., 1996). According to this model, leadership is a collaborative group process directed toward promoting positive social change in an organization or community (Tyree, 1998). A person who demonstrates strong socially responsible leadership capabilities is self-aware, acts in accordance with personal values and beliefs, invests time and energy in activities that he or she believes are important, works with diverse others to accomplish common goals, has a sense of civic and social responsibility, and desires to make the world a better place. The SRLS was developed specifically to measure leadership in college students. The instrument has eight scales corresponding to the eight dimensions of leadership specified in the Astin model (Astin et al., 1996; Dugan, 2006). The eight scales are: a) Consciousness of Self - being aware of the values, emotions, attitudes, and beliefs that motivate one to take action; b) Congruence - thinking, feeling, and behaving with consistency, genuineness, authenticity, and honesty toward others; c) Commitment - intensity and duration in relation to a person, idea, or activity—the energy and passion that propels one to act; d) Collaboration - working with others in a common effort; e) Common Purpose - working with others within a shared set of aims and values; f) Controversy with Civility - recognizing two fundamental realities of any group effort, that 1) differences of viewpoint are inevitable and valuable, and 2) such differences must be


NSSE Validation

17

aired openly and with respect and courtesy; g) Citizenship - believing in a process whereby a person or group is responsibly connected to the environment and the community; and h) Change adapting to continuously evolving environments and situations, while maintaining the primary functions of the group. The internal consistency reliabilities for the eight subscales of the SRLS in the present study ranged from .77 to .88. Previous research has shown the various scales of the SRLS discriminate between involved and uninvolved undergraduate students in community service, student organizational membership, formal leadership programs, and positional leadership roles (Dugan, 2006). Additional research by Rubin (2000) has demonstrated that undergraduates identified as “emerging student leaders” tend to score significantly higher on the SRLS congruency, collaboration, common purpose, citizenship, and change scales than a control group of students not identified as “emerging student leaders.” Because the SRLS does not have a total score, we used the eight SRLS subscales. Moral Character. We assessed the outcome dimension of “Moral Character” with the Defining Issues Test 2 (DIT2). The DIT2 is a revised version of James Rest’s original DIT from 1979 that measures one component of moral development, known as moral judgment or reasoning (Rest, Narvaez, Thoma, & Bebeau, 1999). The DIT2 presents several dilemmas about social problems, such as should a starving man steal food for his family from someone who is hoarding resources? After each, a series of 12 items representing different issues that might be raised by the problem are presented. For example, in the scenario described above, the items include such questions as: “Would stealing bring about more total good for everybody concerned or wouldn’t it? Shouldn’t the community’s laws be upheld?” In response to the scenario and questions, respondents are asked to do three things:


NSSE Validation

18

1. make an action choice (i.e., yes, one should steal or no, one should not steal); 2. rate the series of 12 items in terms of their importance in making a decision about the scenario; and 3. rank the top four most important items. The DIT2 produces two relevant scores. The first is the P-score, which represents the degree to which an individual uses higher order (principled/post-conventional) moral reasoning in resolving moral issues presented in each scenario. The P-score is the proportion of items selected that appeal to moral ideas and/or theoretical frameworks for resolving complex moral issues—specifically, items that appeal to consensus building procedures, insistence on due process, safeguarding minimal basic rights, and organizing social arrangements in terms of appealing to ideals. The P-score has internal consistency reliabilities ranging from .74 to .77 (Rest et al., 1999; University of Minnesota, n.d.). The second score from the DIT2 particularly salient to the WNSLAE is the N2-score. As with the P-score, the N2-score reflects the extent to which one exhibits high order moral reasoning. However, the relatively new N2-score also reflects the extent to which one rejects ideas because they are simplistic or biased (Bebeau & Thoma, 2003). The internal consistency reliability for the N2-score range from .77 to .81 (Rest, et al., 1999; University of Minnesota, n.d.). An extensive body of evidence supports the validity of the DIT in predicting principled ethical behavior in a number of areas. These include: resistance to cheating, peer pressure, and unlawful or oppressive authority; whistle-blowing on corruption; the keeping of contractual promises; helping behavior; community involvement; ethical behavior in several professions; clinical performance in nursing students; and social/political activism (see Pascarella & Terenzini, 1991, 2005, for a synthesis of this body of evidence, including citations to original


NSSE Validation

19

studies). The vast majority of the validity data on the DIT is based on the P-score. However, correlations between the P-score and the N2-score ranged from .91 to .92, suggesting they measure essentially the same construct. For the purposes of this study, we chose to analyze the more comprehensive and recently developed N2-score. With two exceptions, all 3,081 participants completed each WNSLAE outcome/dependent measure during both the initial data collection in fall 2006 and the follow-up data collection in spring 2007. The two exceptions were the CAAP Critical Thinking Test and the Defining Issues Test. Because each of these instruments took at least 40 minutes to complete, we were concerned with the amount of time required of students during each data collection. To remedy this concern, we randomly divided the study participants during the first data collection into two approximately equal samples. One random sample then took the CAAP Critical Thinking Test during both waves of data collections and the other random sample took the Defining Issues Test taken during both data collection waves. Of the 3,081 students participating in both data collections, 1,485 had useable responses on the CAAP and 1,584 had useable responses on the DIT. NSSE Measures of Good Practices in Undergraduate Education The independent variables in the study were four of the five NSSE benchmark scales of effective educational practice (Level of Academic Challenge, Active and Collaborative Learning, Student-Faculty Interaction, Supportive Campus Environment) and the three deep approaches to learning scales (Higher-Order Learning, Integrative Learning, Reflective Learning) previously described. The Enriching Educational Experiences scale was not used in individual-level analysis because it had a particularly low internal consistency reliability in the WNSLAE sample (alpha =


NSSE Validation

20

.44). However, we did use the Enriching Educational Experiences scale in institution-level analyses where scale reliability is less of an issue. Control Variables A particular methodological strength of the Wabash National Study of Liberal Arts Education is that it is longitudinal in nature. This permitted us to introduce a wide range of statistical controls, not only for student background and precollege traits and experiences, but also for other experiences during the first year of college. Our control variables in the present study included the following: •

A parallel precollege measure for each liberal arts outcome measure.

•

Tested precollege academic preparation. This was the student’s ACT score, SAT equivalent score, or COMPASS equivalent score for community college students.

•

Sex.

•

Race (coded as 1 = White, 0 = non-White).

•

Average parental education. This was computed as the average of the respondent’s parents’ education provided that the student provided a response for at least one parent. The item asked “What is the highest level of education each of your parents/guardians completed?” The response options are: 1 = did not finish high school, 2 = High school graduate/GED, 3 = Attended college but no degree, 4 = Vocational/technical certificate or diploma, 5 = Associate of other 2-year degree, 6 = Bachelors or other 4-year degree, 7 = Masters, 8 = Law, 9 = Doctorate).

•

High school involvement. This was a seven-item scale with an internal consistency reliability of .58 that measured involvement during high school. Examples of constituent items include: “During your last year in high school, how often did you


NSSE Validation

21

study with a friend?” “During your last year in high school, how often did you talk with teachers outside of class?” “During your last year in high school, how often did you participate in extracurricular activities?” Response options were “very often,” “often,” “occasionally,” “rarely,” or “never.” Scores on the scale were obtained during the initial data collection in fall 2006. •

Precollege academic motivation. This was an eight-item, likert-type scale in which respondents were asked to indicate the extent to which they agree or disagree (“strongly agree,” “agree,” “not sure,” “disagree,” “strongly disagree”) with statements about their academic motivation. These statements included: a willingness to work hard to learn material even if it doesn’t lead to a higher grade, the importance of getting good grades, reading more for a class than required, enjoyment of academic challenge, and the importance of academic experiences in college. The internal consistency reliability for the scale is .69, and scores on the scale were obtained during the initial data collection in fall 2006.

•

Hours per week during the first year of college one worked both on and off campus. There were eight response options from “zero” to “more than 30 hours.”

•

Lived in campus housing (coded 1) versus elsewhere (coded 0) during the first year of college.

•

The liberal arts emphasis of one’s first year coursework. [Operationalized as the total number of courses during the first year of college taken in traditional liberal arts areas: “Fine Arts, Humanities, and Languages” (e.g., art, music, philosophy, religion, history); “Mathematics/Statistics/Computer Science”; “Natural Sciences” (e.g.,


NSSE Validation

22

chemistry, physics); and “Social Science” (e.g., anthropology, economics, psychology, political science, sociology)] •

Institutional type. This was operationally defined as three dummy variables representing attendance at a research university, regional university, or community college (each coded 1), with attendance at a liberal arts college always coded 0.

Information on work responsibilities, place of residence, and first-year coursework was obtained during the follow-up data collection in spring 2007. Analyses Because of the limited number of institutions in the WNSLAE (N = 19), we considered hierarchical linear modeling (HLM) to be inappropriate for our data (although with sufficient numbers of institutions in the sample it would have been the analytical method of choice). Consequently, we conducted separate analyses with institutions and individuals as the unit of analysis. Since some of the items in the NSSE benchmark scales are also used in the deep learning scales, we conducted analyses for the benchmark scales and the deep learning scales separately. At the institutional level (N = 19), we first computed simple correlations between the average NSSE benchmark scales and the average deep learning scales at each institution and each liberal arts outcome. We then computed partial correlations between each of the average institutional-level benchmark and deep learning scale scores and the institutional average score on each outcome, controlling for the institutional average precollege score on an outcome. The results of these analyses provided what might be considered, respectively, as an upper- and lower-bounds estimate of the effects of the NSSE benchmark and deep learning scales on the liberal arts outcomes at the institutional level. In addition, we control for the pretest in the lower-bounds estimates, this is essentially the same as estimating the effects of the various


NSSE Validation

23

NSSE scales on the net change in each outcome measure during the first year of college (Pascarella, Wolniak, & Pierson, 2003). Because of the extremely small sample size in institutional-level analysis, the critical alpha level was set at .10. A similar set of analyses was conducted with individual students as the unit of analysis (N = 2,861 for all outcomes except critical thinking: N = 1,426, and moral reasoning: N = 1,446). To get an upper-bounds estimate, we simply regressed each liberal arts outcome on the NSSE benchmark scales (excluding the Enriching Education Experience scale) and the deep learning scales without controls. To get a lower-bounds estimate, we repeated the analyses introducing controls for: a parallel pretest, tested precollege academic preparation, sex, race, parental education, measures of high school involvement and precollege academic motivation, institutional type (research university, regional institution, or community college with liberal arts college as the reference group), place of residence and work responsibilities during the first year of college, and the liberal arts emphasis of one’s coursework at college. In all individual-level analyses, we also adjusted for the clustered or nested nature of our data. The “clustered” nature of our data results from the fact that the individuals in our sample were not drawn from a random individual sample but a sample in which their postsecondary institution was the primary sampling unit. Because students within a school are more similar than across schools, the error terms from the prediction model are correlated, which violates one of the assumptions of Ordinary Least Squares regression and results in underestimated standard errors (Ethington, 1997; Raudenbush & Bryk, 2001). We accounted for the nested nature of the data by using appropriate statistical techniques that adjust for this clustering (Groves et al., 2004).


NSSE Validation

24

Results Institutional-Level We summarize the upper- and lower-bounds estimates of the associations between the institutional average NSSE benchmark scale scores and the institutional average liberal arts outcome scores in Table 3. The upper-bounds estimate is the zero-order correlation (column “r” in Table 3), while the lower-bounds estimate is the partial correlation, controlling for the institutional average precollege score on each liberal arts outcome (column “pr” in Table 3). As the table shows, at least one NSSE benchmark had a significant, positive zero-order correlation with each of the 15 liberal arts outcome scores. Not surprisingly, when a control was introduced for the respective average precollege liberal arts outcome score, both the magnitude of the resultant partial correlations (column “pr”) and their statistical significance tended to be reduced across most, though not all, outcomes. Net of the influence of the precollege score, at least one NSSE benchmark had a significant partial correlation with all but five of the first-year liberal arts outcomes: need for cognition, and the congruence, commitment, collaboration and citizenship scales of the socially responsible leadership measure. Table 3 about here Of all the NSSE benchmark scales, the supportive campus environment benchmark had the most significant partial correlations (six) with the liberal arts outcomes. These statistically significant partial correlations were concentrated in three areas: psychological well-being (pr = .725); intercultural effectiveness (pr = .483 and .433, respectively, with the Miville-Guzman universality-diversity scale and the openness to diversity scale); and socially responsible leadership (pr = .435, .478, and .666, respectively, with consciousness of self, common purpose, and change scales). Both the level of academic challenge and enriching educational experiences


NSSE Validation

25

benchmarks had significant partial correlations with four liberal arts outcomes. Each benchmark had a significant partial correlation with first-year critical thinking (pr = .433 for academic challenge and .441 for enriching experiences). Aside from their common association with critical thinking, however, academic challenge and enriching educational experiences had significant partial correlations with different liberal arts outcomes. Academic challenge had significant partial correlations with positive attitude toward literacy (pr = .512), consciousness of self (pr = .499), and controversy with civility (pr = .625), while enriching educational experiences had significant partial correlations with the Miville-Guzman universality-diversity scale (pr = .572), openness to diversity (pr = .410), and moral reasoning as measured by the DIT-N2 score (pr = .444). Both the active and collaborative learning and student-faculty interaction benchmarks had significant partial correlations with only one liberal arts outcome. Net of average institutionallevel precollege score, active and collaborative learning significantly predicted the openness to diversity score (pr = .559), while student-faculty interaction had a significant partial correlation (pr = .424) the change score of the socially responsible leadership measure. Table 4 summarizes the estimated upper- and lower-bounds estimates of the associations between the institutional average NSSE deep learning scale scores and the institutional average liberal outcomes scores. Again, the upper-bounds estimate is the zero-order correlation (column “r”), while the lower-bounds estimate is the partial correlation, controlling for the institutional average precollege score on each liberal arts outcome (column “pr”). Overall, with institutions as the unit of analysis, the NSSE deep learning scales did not predict the first-year liberal arts outcomes as well as did the NSSE benchmarks. The deep learning scales had significant, positive zero-order correlations with 11 of the 15 liberal arts outcome measures. However, when a control


NSSE Validation

26

for the respective average precollege liberal arts outcome score was introduced, the deep learning scales had statistically significant, and positive partial correlations with only four of the liberal arts outcome measures. Moreover, the significant, positive partial correlations appeared to be concentrated in three areas: inclination to inquire and lifelong learning; intercultural effectiveness, and socially responsible leadership. Significant, positive partial correlations were found between the need for cognition scale and the integrative learning (pr = .465), reflective learning (pr = .562), and the deep learning total score (pr = .502). Similarly, the deep learning total score (pr = .546), the integrative learning score (pr = .591), and the higher order learning score (pr = .435) each had a significant, positive partial association with the openness to diversity scale. Finally, all four deep learning scales had a significant, positive partial correlation with the controversy with civility scale of the socially responsive leadership measure: total score, pr = .689; higher order learning, pr = .660; integrative learning, pr = .549; and reflective learning, pr = .705. In contrast with the NSSE benchmarks, the NSSE deep learning scales had no significant relationship, net of precollege scores, with critical thinking, psychological well being, positive attitude toward literacy, the Miville-Guzman universality-diversity scale, or moral reasoning as measured by the DIT-N2 score. Furthermore, all four deep learning scales had a significant negative partial correlation with the collaboration core of the socially responsible leadership measure. Table 4 about here Individual-Level We present the upper- and lower-bounds estimates of the standardized (β) associations between four individual student-level NSSE benchmark scores and individual-level liberal arts


NSSE Validation

27

outcome scores in Table 5. (Recall we excluded the enriching educational experiences benchmark from the individual-level analyses because of a low reliability in the present sample.) The upper-bounds estimate (column “U” in Table 5) controls only for the other NSSE benchmark scales. The lower-bounds estimate (column “L” in Table 5) controls not only for the other benchmark scales, but also for a parallel precollege measure of the respective liberal arts outcome measure, tested precollege academic preparation, race, sex, parental education, precollege academic motivation, high school involvement, place of residence and work responsibilities during college, the liberal arts emphasis of first-year college coursework, institutional type. Both the upper- and lower-bounds estimates account for the clustering or nesting effect of the data. As the table shows, at least one of the four NSSE benchmark scales included in the individual-level analyses had a significant, positive, upper-bounds estimate standardized association with each of the 15 liberal arts outcome scores. As expected, when additional controls were introduced to obtain the lower-bounds estimates, a substantial number of these associations became smaller and nonsignificant (column “L”). However, even with the substantially more stringent controls in the lower-bounds estimates, all 15 liberal arts outcome measures had a significant and positive, though modest, association with at least one of the four benchmark scales. Table 5 about here Students’ perceived levels of academic challenge had the widest range of estimated effects in the lower-bounds equations. Net of an extensive battery of controls, level of academic challenge had significant, positive standardized associations with psychological well-being (β = .050), both measures of intercultural effectiveness (the Miville-Guzman, β = .110, and the


NSSE Validation

28

openness to diversity scale, β = .119), and all eight subscales of the socially responsible leadership measure (βs ranging from .055 to .152). The active and collaborative learning benchmark tended to demonstrate the strongest lower-bounds associations with cognitive oriented outcomes, having small but statistically significant beta’s with critical thinking (β = .056), need for cognition (β = .041), and moral reasoning as measured by the DIT-N2 score (β = .073). In addition, active and collaborative learning also had significant, positive lower-bounds associations with psychological well-being (β = .087), openness to diversity (β = .057), and the congruence, collaboration, citizenship, and change scale of socially responsible leadership (βs from .068 to .096). At the individual level, students’ perceptions of support in the campus environment had the most consistent lower-bounds associations with intercultural effectiveness—having small but significant beta associations with both the Miville-Guzman universality-diversity scale (β = .050), and the openness to diversity scale (β = .085). Additionally, the supportive campus environment benchmark also had significant lower-bounds associations with psychological wellbeing (β = .130), positive attitude toward literacy (β = .050), and the collaboration, common purpose, controversy with civility, and change scales of socially responsible leadership (βs ranging from .062 to .093). Finally, consistent with the institution-level analyses, the student-faculty interaction benchmark demonstrated only limited predictive power. Indeed, it had only one significant standardized association, and that was a negative association (β = -.049) with critical thinking. We display the upper- and lower-bounds estimates of the standardized (β) associations between the individual student-level NSSE deep learning scales and the individual-level liberal arts outcome scores in Table 6. Overall, it would appear that at the individual-student level, the


NSSE Validation

29

deep learning scales predicted the liberal arts outcomes as well or perhaps even slightly better than the benchmark scales. One possible reason for this is that the deep learning scales had somewhat higher reliabilities in the WNSLAE sample than did the benchmarks (See Tables 1 and 2). As with the NSSE benchmarks, at least one of the deep learning scales had a significant, positive, upper-bounds association with each of the 15 liberal arts outcome scores. Similarly, even with the introduction of more stringent controls in the lower-bounds estimates, all 15 liberal arts outcome measures had a significant, positive, though modest, association with at least one of the deep learning scales. Table 6 about here Reflective learning was the most consistently effective deep learning scale in terms of predicting liberal arts outcomes in the lower-bounds estimates—having a statistically significant, though modest, association with all 15 measures. These β associations ranged from .053 with critical thinking to .234 with openness to diversity. It was also the only deep learning scale with significant lower-bounds associations with all cognitive oriented outcomes: critical thinking, moral reasoning, need for cognition, and positive attitude toward literacy (βs ranging from .053 to .101). Both the higher order learning and the integrative learning scales had somewhat fewer significant lower-bounds associations with liberal arts outcomes than did reflective learning. The higher order learning scale had significant lower-bounds associations with psychological wellbeing (β = .067), openness to diversity (β = .052), and the consciousness of self, congruence, collaboration, common purpose, and controversy with civility scale of the socially responsible leadership measure (βs ranging from .087 to .132). Integrative learning had significant lowerbounds associations with need f or cognition (β = .072), positive attitude toward literacy (β =


NSSE Validation

30

.108), the Miville-Guzman universality-diversity scale (β = .075), openness to diversity (β = .094), moral reasoning (β = .085), and the collaboration, controversy with civility, and change scores of socially responsible leadership (βs ranging from .056 to .064). Finally, the deep learning total score, which combines all 12 items, had significant lower-bounds associations with all liberal arts outcomes except critical thinking. The βs for these significant associations ranged from .101 with moral reasoning to .294 with openness to diversity. Discussion The use of the National Survey of Student Engagement (NSSE) to measure effective practices in undergraduate education has become wide spread in American postsecondary institutions. The NSSE benchmark scales and deep learning scales purport to measure educational practices that are linked to desirable learning and developmental outcomes. To date, however, there is only a limited body of internally valid research that has attempted to validate this claim in ways that permit the control of important confounding influences such as precollege level of development. The present study sought to address this relative dearth of validity evidence concerning the NSSE by estimating “upper-” and “lower-bounds” associations between the NSSE benchmarks and deep learning scales on the one hand and a comprehensive array of cognitive and personal outcomes typically associated with a liberal arts education on the other. The longitudinal nature of our data permitted an introduction of statistical controls for important confounding influences (including a precollege measure of each outcome) that has typically not been possible in previous research on the predictive validity of the NSSE scales. Our overall findings clearly suggest that both the NSSE benchmark and deep learning scales are, in fact, significantly and positively linked to measures tapping each of the six basic liberal arts outcomes considered in the study: effective reasoning and problem solving, well


NSSE Validation

31

being, inclination to inquire and lifelong learning, intercultural effectiveness, leadership, and moral character. With both institutions and individuals as the unit of analyses, there was a general tendency for these associations to persist even after introducing controls for important confounding influences. With only 19 colleges in the sample, this control at the institutional level consisted of the average precollege score on each liberal arts outcome. At the individual student level, the controls consisted not only of the precollege score on each outcome, but also tested precollege academic preparation and academic motivation, demographic characteristics, family educational background, high school involvement, college place of residence, work responsibilities, and coursework emphasis, and the clustering effect. Moreover, it would appear that exposure to good practices, as measured by the NSSE benchmark and deep learning scales, is significantly linked to growth in liberal arts outcomes as early as the first year of postsecondary education. Although institutional use of the NSSE is usually oriented toward a sample of students from all undergraduate classes, our results suggest that the NSSE may also be used to focus on the effectiveness of the first year of college—a period of time in which some multi-institutional evidence suggests that the greatest developmental impact of postsecondary education occurs (Flowers, Osterlind, Pascarella, & Pierson, 2001; Pascarella & Terenzini, 2005). Clearly, the small number of colleges in the WNSLAE (N = 19) limited the sophistication of our institution-level analyses. In order to provide sufficient residual degrees of freedom (16) to detect significant associations between NSSE scales and liberal arts outcomes, we controlled only for the most important confounding influence: the average institutional precollege score on each liberal arts outcome. Yet even with such low statistical power we found that, net of average institutional precollege score, at least one NSSE benchmark scale had a significant, positive


NSSE Validation

32

partial correlation with 10 of 15 liberal arts outcome measures. At the institutional level, the deep learning scales performed substantially less effectively than the benchmarks, having significant, positive partial correlations with only 4 of 15 liberal arts outcome scores. Interestingly, however, three of the deep learning scales had significant, positive partial correlations with need for cognition, the only cognitively oriented outcome unrelated to the benchmarks. Because there is no within-institution score variability, one might expect discernibly stronger associations among variables with institutions rather than individuals as the unit of analysis. Nevertheless, the significant partial correlations between the NSSE benchmark and deep learning scales on the one hand and liberal arts outcomes on the other were substantial in magnitude. For the benchmarks, the 18 statistically significant partial correlations ranged from .410 to .725 with a median of .532, while for the deep learning scales, the 11 significant positive correlations with liberal arts outcomes ranged from .423 to .705 with a median of .562. With individuals as the level of analysis, the significant, positive lower-bounds associations between the benchmark and deep learning scales on the one hand and liberal arts outcomes were numerous but modest in magnitude. For the four benchmarks considered, the 27 significant, positive lower-bounds β coefficients ranged from .041 to 152 with a median of .082, while for the three independent deep learning scales the 30 significant, positive lower-bounds β coefficients ranged from .052 to .183, with a medium of .091. For the total deep learning scale (a composite of the three independent scales), the 14 significant, positive lower-bounds associations with the 15 liberal arts outcome measures ranged from .100 to .294, with a median of .187. This suggests that, at the individual student level, at least, the deep learning total score had the single most salient net influence on liberal arts outcomes of all the NSSE scales.


NSSE Validation

33

There are several possible explanations for the relatively modest net, lower-bounds associations between the NSSE scales and liberal arts outcomes when students are the level of analysis. The first is that the internal consistency reliabilities of the NSSE scales in the WNSLAE sample were somewhat depressed. Across all eight NSSE scales used in the individual level analyses, the reliabilities ranged from .57 to .82, with a median of .725. Indeed, the deep learning total score, which had the highest alpha (.82), also had the strongest net positive associations with the liberal arts outcomes. Second, our lower-bounds estimates at the individual level of analysis introduced statistical controls for an extensive set of confounding influences that included a precollege measure of each outcome, actual tested precollege academic preparation, academic motivation, demographic characteristics, high school experiences, and other first-year college experiences. The introduction of such controls (particularly a precollege measure of each outcome) tends to produce very conservative estimates of the impact of specific college experiences, such as those measured by the NSSE benchmark and deep learning scales (Pascarella & Terenzini, 1991, 2005). Finally, it is worth noting that the present study estimated the validity of good practices as measured by the NSSE scales only during the first year of college. One might reasonably expect good practices to demonstrate somewhat stronger impacts on student development during the subsequent years of college when such practices have had longer periods of time to exert their influence. Implications for Policy The U.S. News and World Report (USNWR) annual ranking of postsecondary institutions have strongly shaped public notions of a quality undergraduate education in the U.S.


NSSE Validation

34

These rankings, however, operationally define “quality” largely in terms of resources, reputation, and, particularly, the academic selectivity of an institution’s undergraduate student body. Indeed, there is sound evidence to suggest that the USNWR rankings can be essentially reproduced simply by knowing the average ACT/SAT score of each institution’s entering first year class (Pascarella, Cruce, Umbach, Wolniak, Kuh, Carini, Hayek, Gonyea, and Zhao, 2006; Webster, 2001). This means that, insofar as the USNWR rankings shape public understanding of what constitutes a quality undergraduate education, it will be defined by the characteristics students bring to college and not by the actual effectiveness of the academic and non-academic experiences students have after they arrive on campus. The good practices scales of the National Survey of Student Engagement (NSSE) were designed specifically to measure these effective academic and non-academic experiences, and there is little evidence that such experiences are substantially linked to the academic selectivity of the college one attends (Pascarella et al., 2006). Our findings suggest the dimensions of undergraduate experience measured by the NSSE scales are, in fact, precursors to desirable liberal arts outcomes, and thus a more valid conception of what truly constitutes a quality undergraduate education than the measures used by USNWR. Furthermore, the NSSE benchmark and deep learning scales point to areas of effective academic and non-academic experiences that may be amenable to improvement through purposeful changes in institutional policies and practices. On the other hand, resources and academic selectivity—measures frequently used by USNWR to rank colleges—may be much harder to change and therefore may form a much more deterministic or stable context in which an institution must learn to function. For example, state legislatures often set the admissions standards, and therefore the selectivity, of many public institutions. However, a dynamic context, grounded in an institution’s commitment toward improvement, may be able to create an


NSSE Validation

35

institutional culture that continuously strives to engage students in the practices of the NSSE benchmark and deep learning scales, yielding improved student learning and, thus, improved institutional effectiveness. “Institutional effectiveness” and “accountability through the assessment of student learning” have ceased to be simply buzzwords. The U.S. Department of Education, accrediting agencies, and educational stakeholders from the Business-Higher Education Forum (BHEF) to the State Higher Education Executive Officers (SHEEO) are calling for accountability of student learning as a necessary condition for the United States to remain competitive in meeting economic and labor force trends in the global economy (McPherson & Shulenburger, 2006). To address these calls, the American Association of State Colleges and Universities (AASCU) and the National Association of State Universities and Land-Grant Colleges (NASULGC) have created a voluntary system of accountability, of which NSSE plays a key role. Students’ experiences and their perceptions of those experiences, along with institutional data from one of three learning outcomes instruments, contribute significantly to the Voluntary System of Accountability’s College Portrait (VSA, 2008). However, due to cost, some institutions may not feel it is feasible to collect all data necessary to complete their institutional portrait. It’s in this regard that the findings from the present study can be of great use. Our findings confirm the primary assumption of NSSE that in measuring student exposure to and engagement in empirically-vetted good practices, one is essentially measuring experiences that yield desired cognitive and personal development during college. Thus, collecting data on student experiences provides a window from which institutions can extrapolate to student learning. This kind of inference is particularly useful for institutions that might only have resources to collect student experience data. Although not an exhaustive set, our research


NSSE Validation

36

demonstrates the NSSE student experience scales to be valid predictors of a host of outcomes theoretically and conceptually associated with a liberal arts education. Thus, institutional emphasis on improving the condition, practices, and pedagogy that engage students pays dividends in student learning. Moreover, the WNSLAE outcomes mirror many attributes American companies have identified in which U.S. college graduates lack development: leadership and teamwork (measured by the SRLS), problem solving and analytical thinking (measured by the CAAP), and global consciousness (measured by the MGUDS and ODC) (BHEF, 2003). Because of WNSLAE’s pretest-posttest longitudinal design, our results suggest institutions that improve the degree to which students engage in these good practices, as measured by NSSE, in effect, improve student learning and development in areas necessary for success in the interdependent global economy and society. As resources for higher education grow ever tighter (National Center for Public Policy in Higher Education [NCPPHE], 2005), identifying efficiencies in demonstrating accountability and effectiveness in student learning is paramount.


NSSE Validation

37

REFERENCES Anaya, G. (1999, April). Within-college, curricular and co-curricular correlates of performance on the MCAT. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada. Arnold, J., Kuh, G., Vesper, N., & Schuh, J. (1993). Student age and enrollment status as determinants of learning and personal development at metropolitan institutions. Journal of College Student Development, 34, 11-16. Astin, A. (1993). What matters in college? Four critical years revisited. San Francisco: JosseyBass. Astin, A., Astin, H., Boatsman, K., Bonous-Hammarth, M., Chambers, T., Goldberg, S., et al. (1996). A social change model of leadership development: Guidebook (Version III). Los Angeles: University of California at Los Angeles, Higher Education Research Institute. Astin, A., & Lee, J. (2003). How risky are one-shot cross-sectional assessments of undergraduate students? Research in Higher Education, 44, 657-672. Avalos, J. (1996). The effects of time-to-degree completion, stopping out, transferring, and reasons for leaving college on students' long-term retention, educational aspirations, occupational prestige, and income. Unpublished doctoral dissertation, University of California at Los Angeles. Bebeau, M., & Thoma, S. (2003). Guide for the DIT-2. Minneapolis, MN: Center for the Study of Ethical Development, University of Minnesota. Bray, G., Pascarella, E., & Pierson, C. (2004). Postsecondary education and some dimensions of literacy development: An exploration of longitudinal evidence. Reading Research Quarterly, 39, 306-330.


NSSE Validation

38

Business-Higher Education Forum (2003). Building a nation of learners: The need for changes in teaching and learning to meet global challenges. Washington, DC: Author.

Cabrera, A., Crissman, J., Bernal, E., Nora, A., Terenzini, P., & Pascarella, E. (2002). Collaborative learning: Its impact on college students' development and diversity. Journal of College Student Development, 43(1), 20-34. Cacioppo, J., Petty, R., Feinstein, J., & Jarvis, W. (1996). Dispositional differences in cognitive motivation: The life and times of individuals varying in need for cognition. Psychological Bulletin, 119, 197-253. Carini, R., Kuh, G., & Klein, S. (2006). Student engagement and student learning: Testing the linkages. Research in Higher Education, 47, 1-32. Chickering, A., & Gamson, Z. (1987). Seven principles for good practice in undergraduate education. AAHEBulletin, 39(7), 3-7. Chickering, A., & Gamson, Z. (1991). Applying the seven principles for good practice in undergraduate education. San Francisco: Jossey-Bass. Chickering, A., & Reisser, L. (1993). Education and identity (2nd ed.). San Francisco: JosseyBass. Cruce, T., Wolniak, G., Seifert, T., & Pascarella, E. (2006). Impacts of good practices on cognitive development, learning orientations, and graduate degree plans during the first year of college. Journal of College Student Development, 47, 365-383. d'Apollonia, S., & Abrami, P. (1997). Navigating student ratings of instruction. American Psychologist, 52, 1198-1208. Dugan, J. (2006). Explorations using the social change model: Leadership development among college men and women. Journal of College Student Development, 47, 217-225.


NSSE Validation

39

Elias, S., & Loomis, R. (2002). Utilizing need for cognition and perceived self-efficacy to predict academic performance. Journal of Applied Social Psychology, 32, 1687-1717. Ethington, C.A. (1997). A hierarchical linear modeling approach to studying college effects. In J.C. Smart (Ed.), Higher education: Handbook of theory and research, Volume XII (pp. 165-194). New York: Agathon Press. Feldman, K. (1997). Identifying exemplary teachers and teaching: Evidence from student ratings. In R. Perry and J. Smart (Eds.), Effective teaching in higher education: Research and practice (pp. 368-395). Edison, NJ: Agathon Press. Flowers, L., Osterlind, S., Pascarella, E., & Pierson, C. (2001). How much do students learn in college? Cross-sectional estimates using the College Basic Academic Subjects Examination. Journal of Higher Education, 72(5), 565-583. Fuertes, J., Miville, M., Mohr, J., Sedlacek, W., & Gretchen, D. (2000). Factor structure and short form of the Miville-Guzman Universality-Diversity Scale. Measurement and Evaluation in Counseling and Development, 33, 157-169. Groves, R.M., Fowler, F.J. Jr., Couper, M.P., Lepkowski, J.M., Singer, E., & Tourangeau, R. (2004). Survey methodology. Hoboken, NJ: Wiley-Interscience. Gurin, P., Dey, E., Hurtado, S., & Gurin, G. (2002). Diversity and higher education: Theory and impact on educational outcomes. Harvard Educational Review, 72, 330-366. Hagedorn, L., Siadat, M., Nora, A., & Pascarella, E. (1997). Factors leading to gains in mathematics during the first year of college. Journal of Women and Minorities in Science and Engineering, 3(3), 185-202.


NSSE Validation

40

Hake, R. (1998). Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66(1), 64-74. Hayek, J., Carini, R., O’Day, P., & Kuh, G. (2002). Triumph or tragedy: Comparing student engagement levels of members of Greek-letter organizations and other students. Journal of College Student Development, 43, 643-663. Johnson, D., Johnson, R., & Smith, K. (1998a). Active learning: Cooperation in the college classroom (2nd ed.). Edina, MN: Interaction Book Company. Johnson, D., Johnson, R., & Smith, K. (1998b). Cooperative learning returns to college. Change, 30(4), 26-35. Johnstone, K., Ashbaugh, H., & Warfield, T. (2002). Effects of repeated practice and contextualwriting experiences on college students' writing skills. Journal of Educational Psychology, 94(2), 305-315. Jones, S., & McEwen, M. (2000). A conceptual model of multiple dimensions of identity. Journal of College Student Development, 41, 405-414. Keyes, C., Shmotkin, D., & Ryff, C. (2002). Optimizing well-being: The empirical encounter of two traditions. Journal of Personality and Social Psychology, 82, 1007-1022. King, P., Kendall Brown, M., Lindsay, N., & VanHecke, J. (2007). Liberal arts student learning outcomes: An integrated approach. About Campus, 12(4), 2-9. Kuh, G. (2001). The National Survey of Student Engagement: Conceptual framework and overview of psychometric properties. Bloomington, IN: Indiana University, Center for Postsecondary Research


NSSE Validation

41

Kuh, G., & Gonyea, R. (2003). The role of the academic library in promoting student engagement in learning. College and Research Libraries, 64, 256-282. Kuh, G., & Hu, S. (2001). The effects of student-faculty interaction in the 1990s. Review of Higher Education, 24(3), 309-332. Kuh, G., Kinzie, J., Schuh, J., Whitt, E., & Associates (2005). Student success in college: Creating conditions that matter. San Francisco: Jossey-Bass. Kuh, G., Pace, C., & Vesper, N. (1997). The development of process indicators to estimate student gains associated with good practices in undergraduate education. Research in Higher Education, 38(4), 435-454. Kuh, G., Schuh, J., Whitt, E., & Associates. (1991). Involving colleges: Successful approaches to fostering student learning and development outside the classroom. San Francisco: JosseyBass. LaNasa, S., Olson, E., & Alleman, N. (2007). The impact of on-campus student growth on firstyear student engagement and success. Research in Higher Education, 48, 941-966. Lang, M. (1996). Effects of class participation on student achievements and motivation. Unpublished honors thesis, University of Western Ontario, London, Ontario. McPherson, P., & Shulenburger, D. (2006, April). Improving student learning in higher education through better accountability and assessment: A discussion paper for the National Association of State Universities and Land-Grant Colleges. Washington, DC: NASULGC. Miville, M., Gelso, C., Pannu, R., Liu, W., Touradji, P., Holloway, P., Fuertes, J. (1999). Appreciating similarities and valuing differences: The Miville-Guzman UniversalityDiversity Scale. Journal of Counseling Psychology, 46(3), 291-307.


NSSE Validation

42

Murray, H., & Lang, M. (1997). Does classroom participation improve student learning? Teaching and Learning in Higher Education, 20(Feb.), 7-9. National Center for Public Policy in Higher Education (2005). State capacity for higher education policy. Washington, DC: Author. National Survey of Student Engagement. (2006). The college student report: 2006 codebook. Bloomington, IN: Indiana University Center for Postsecondary Research. National Survey of Student Engagement. (2007). Experiences that matter: Enhancing student learning and success. Bloomington, IN: Indiana University Center for Postsecondary Research. Nelson Laird, T., Shoup, R., & Kuh, G. (2006, May). Measuring deep approaches to learning using the National Survey of Student Engagement. Paper presented at the annual Forum of the Association for Institutional Research, Chicago, IL. Nelson Laird, T., Shoup, R., Kuh, G., & Schwarz, M. (2008). The effects of discipline on deep approaches to student learning and college outcomes. Research in Higher Education, 49, 469-494. Pascarella, E. (2001). Using student self-reported gains to estimate college impact: A cautionary tale. Journal of College Student Development, 42(5), 488-492. Pascarella, E., Bohr, L., Nora, A., & Terenzini, P. (1995). Cognitive effects of 2-year and 4-year colleges: New evidence. Educational Evaluation and Policy Analysis, 17, 83-96. Pascarella, E., Cruce, T., Umbach, P., Wolniak, G., Kuh, G., Carini, R., Hayek, J., Gonyea, R., & Zhao, C. (2006). Institutional selectivity and good practices in undergraduate education: How strong is the link? Journal of Higher Education, 77, 251-285.


NSSE Validation

43

Pascarella, E., Edison, M., Nora, A., Hagedorn, L., & Terenzini, P. (1996). Influences on students’ openness to diversity and challenge in the first year of college. Journal of Higher Education, 67, 174-195. Pascarella, E., Palmer, B., Moye, M., & Pierson, C. (2001). Do diversity experiences influence the development of critical thinking? Journal of College Student Development, 42(3), 257-271. Pascarella E., & Terenzini, P. (1991). How college affects students: Findings and insights from twenty years of research. San Francisco: Jossey-Bass. Pascarella, E., & Terenzini, P. (2005). How college affects students (Vol. 2): A third decade of research. San Francisco: Jossey-Bass. Pascarella, E., Wolniak, G., & Pierson, C. (2003). Explaining student growth in college when you don’t think you are. Journal of College Student Development, 44, 122-126. Pike, G. (2006). The convergent and discriminant validity of NSSE scalelet score. Journal of College Student Development, 47, 550-563. Pike, G., & Kuh, G. (2005). First- and second-generation college students: A comparison of their engagement and intellectual development. Journal of Higher Education, 76, 276-300. Pike, G., Kuh, G., & Gonyea, R. (2007). Evaluating the rationale for affirmative action in college admissions: Direct and indirect relationships between campus diversity and gains in understanding diverse groups. Journal of College Student Development, 48, 166-182. Qin, Z., Johnson, D., & Johnson, R. (1995). Cooperative versus competitive efforts and problem solving. Review of Educational Research, 65, 129-143. Raudenbush, S.W., & Bryk, A.S. (2001). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage Publications.


NSSE Validation

44

Rest, J., Narvaez, D., Thoma, S., & Bebeau, M. (1999). DIT2: Devising and testing a revised instrument of moral judgment. Journal of Educational Psychology, 91, 644-659. Rubin, J. (2000). The emerging leaders: An evaluation of the social change model of leadership. Unpublished doctoral dissertation, The Union Institute. Ryff, C. (1989). Happiness is everything, or is it? Explorations on the meaning of psychological well-being. Journal of Personality and Social Psychology, 57, 1069-1081. Ryff, C., & Keyes, C. (1995). The structure of psychological well-being revisited. Journal of Personality and Social Psychology, 69, 719-727. Seifert, T., Goodman, K., Lindsay, N., Jorgensen, J., Wolniak, G., Pascarella, E., & Blaich, C. (2008). The effects of liberal arts experiences on liberal arts outcomes. Research in Higher Education, 49, 107-125. Springer, K., & Hauser, R. (2006). An assessment of the construct validity of Ryff’s Scales of Psychological Well-Being: Method, mode, and measurement effects. Social Science Research, 35, 1080-1102. Springer, K., Hauser, R., & Freese, J. (2006). Bad news indeed for Ryff’s six-factor model of well-being. Social Science Research, 35, 1120-1131. Terenzini, P., Springer, L., Yaeger, P., Pascarella, E., & Nora, A. (1994). The multiple influences on students' critical thinking skills. Paper presented at the annual meeting of the Association for the Study of Higher Education, Orlando, FL. Tyree, T. (1998). Designing an instrument to measure socially responsible leadership using the social change model of leadership development. Unpublished doctoral dissertation, University of Maryland-College Park.


NSSE Validation

45

Umbach, P., & Kuh, G. (2003, May). Student experiences with diversity at liberal arts colleges: Another claim for distinctiveness. Paper presented at the Annual Forum of the Association for Institutional Research, Tampa, FL. Umbach, P., & Kuh, G. (2006). Student experiences with diversity at liberal arts colleges: Another claim for distinctiveness. Journal of Higher Education, 77, 169-192. Umbach, P., & Wawrzynski, M. (2004, June). Do faculty matter: The role of college faculty in student learning and engagement. Paper presented at the Annual Form of the Association for Institutional Research, Boston, MA. University of Minnesota, Center for the Study of Ethical Development (n.d.). New index (N2). Retrieved January 14, 2008 from http://www.centerforthestudyofethicaldevelopment.net/New%20Index.htm. Webster, T. (2001). A principal component analysis of the U.S. News & World Report tier rankings of colleges and universities. Economics of Education Review, 20(3), 235-244. Whitmire, E., & Lawrence, J. (1996, Nov.). Undergraduate students' development of critical thinking skills: An institutional and disciplinary analysis and comparison with academic library use and other measures. Paper presented at the annual meeting of the Association for the Study of Higher Education, Memphis, TN. Whitt, E., Edison, M., Pascarella, E., Terenzini, P., & Nora, A. (2001). Influences on students’ openness to diversity in the second and third years of college. Journal of Higher Education, 72, 172-204. Zhao, C., & Kuh, G. (2004). Adding value: Learning communities and student engagement. Research in Higher Education, 45, 115-138.


NSSE Validation

46

Table 1 National Survey of Student Engagement Benchmark Scales and Component Itemsa Response Options

Scale/Items Level of Academic Challenge (alpha reliability = .70)b Number of assigned textbooks, books, or book-length packs of course readings

None, Between 1 and 4, Between 5 and 10, Between 11 and 20, More than 20

Number of written papers or reports of 20 PAGES OR MORE


Number of written papers or reports BETWEEN 5 AND 19 PAGES


Number of written papers or reports of FEWER THAN 5 PAGES


Coursework emphasized: ANALYZING the basic elements of an idea, experience, or theory, such as examining a particular case or situation in depth and considering its components

Very little, Some, Quite a bit, Very much

Coursework emphasized: SYNTHESIZING and organizing ideas, information, or experiences into new, more complex interpretations and relationships


Coursework emphasized: MAKING JUDGMENTS about the value of information, arguments, or methods, such as examining how others gathered and interpreted data and assessing the soundness of their conclusions


Coursework emphasized: APPLYING theories or concepts to practical problems or in new situations


Worked harder than you thought you could to meet an instructor’s standards or expectations

Never, Sometimes, Often, Very often


NSSE Validation

Response Options

Scale/Items Hours per 7-day week spent preparing for class (studying, reading, writing, doing homework or lab work, analyzing data, rehearsing, and other academic activities)

0, 1-5, 6-10, 11-15, 16-20, 21-25, 26-30, more than 30

Institutional emphasis: Spending significant amounts of time studying and on academic work


Active and Collaborative Learning (alpha = .57)b Asked questions in class or contributed to class discussions


Made a class presentation


Worked with other students on projects DURING CLASS


Worked with classmates OUTSIDE OF CLASS to prepare class assignments


Tutored or taught other students (paid or voluntary)


Participated in a community-based project (e.g., service learning) as part of a regular course


Discussed ideas from your readings or classes with others outside of class (students, family members, co-workers, etc.)


Student-Faculty Interaction (alpha = .69)b Discussed grades or assignments with an instructor


Discussed ideas from your readings or classes with faculty members outside of class



47

NSSE Validation

48

Response Options

Scale/Items Talked about career plans with a faculty member or advisor


Received prompt written or oral feedback from faculty on your academic performance


Worked with faculty members on activities other than coursework (committees, orientation, student life activities, etc.)


Work on a research project with a faculty member outside of course or program requirements

Have not decided, Do not plan to do, Plan to do, Done

Enriching Educational Experiences (alpha = .44)b Had serious conversations with students who are very different from you in terms of their religious beliefs, political opinions, or personal values


Had serious conversations with students of a different race or ethnicity than your own


Institutional emphasis: Encouraging contact among students from different economic, social, and racial or ethnic backgrounds


Hours per 7-day week spent participating in co-curricular activities (organizations, campus publications, student government, fraternity or sorority, intercollegiate or intramural sports, etc.)

0, 1-5, 6-10, 11-15, 16-20, 21-25, 26-30, More than 30

Used an electronic medium (listserv, chat groups, Internet, instant messaging, etc.) to discuss or complete an assignment


Practicum, internship, field experience, co-op experience, or clinical assignment

Have not decided, Do not plan to do, Plan to do, Donec


NSSE Validation

49

Response Options

Scale/Items Community service or volunteer work


Participate in a learning community or some other formal program where groups of students take two or more classes together


Foreign language coursework


Study abroad


Independent study or self-designed major


Culminating senior experience (capstone course, senior project or thesis, comprehensive exam, etc.)


Supportive Campus Environment (alpha = .76)b Institutional emphasis: Providing the support you need to thrive socially


Institutional emphasis: Providing the support you need to help you succeed academically


Institutional emphasis: Helping you cope with your non-academic responsibilities (work, family, etc.)


Quality: Your relationships with other students

Unfriendly, Unsupportive, Sense of Alienation, 2, 3, 4, 5, 6, Friendly, Supportive, Sense of Belonging

Quality: Your relationships with faculty members

Unavailable, Unhelpful, Unsympathetic, 2, 3, 4, 5, 6, Available, Helpful, Sympathetic

Quality: Your relationships with administrative personnel and offices

Unhelpful, Inconsiderate, Rigid, 2, 3, 4, 5, 6, Helpful, Considerate, Flexible


NSSE Validation

a

Source: National Survey of Student Engagement (2008) Experiences that matter: Enhancing student engagement and success, Annual report 2007. Bloomington, IN: Indiana University, Center for Postsecondary Research. b Alpha reliabilities are based on the WNSLAE sample. c Coded: “Done” = 1, all other = 0.


50

NSSE Validation

51

Table 2 National Survey of Student Engagement Deep Approaches to Learning Scales and Component Items a Scale/Items Higher Order Learning (alpha =.75) b Analyzed the basic elements of an idea, experience, or theory, such as examining a particular case or situation in depth and considering its components Synthesized and organized ideas, information, or experiences into new, more complex interpretations and relationships Made judgments about the value of information, arguments, or methods, such as examining how others gathered and interpreted data and assessing the soundness of their conclusions Applied theories or concepts to practical problems or in new situations Integrative Learning (alpha = .67) b Worked on a paper or project that required integrating ideas or information from various sources Included diverse perspectives (different races, religions, gender, political beliefs, etc.) in class discussion or writing assignments Put together ideas or concepts from different courses when completing assignments or during class discussions Discussed ideas from your readings or classes with faculty members outside of class discussions Discussed ideas from your readings or classes with others outside of class (students, family members, co-workers, etc.) Reflective Learning (alpha = .76) b Examined the strengths and weaknesses of your own views on a topic or issue Tried to better understand someone else’s views by imagining how an issues looks from his or her perspective


NSSE Validation

52

Scale/Items Learned something that changed the way you understand an issue or concept Deep Learning Total Score (alpha = .82) b Includes all items in the scales above a

Source: Nelson Laird, T., Shoup, R., Kuh, G., & Schwarz, M. (2008). The effects of discipline on deep approaches to student learning and college outcomes. Research in Higher Education, 49, 469-494. b Alpha reliabilities are based on the WNSLAE sample.


NSSE Validation

53

Table 3 Zero-order (r) and Partial (pr) Correlations Between NSSE Benchmark Scales and First-Year Liberal Arts Outcomes (Institutions as Unit of Analysis, n = 19)

Liberal Arts Outcome

Level of Academic Challenge r pra

Active and Collaborative Learning r pra

Student-Faculty Interaction r pra

Enriching Educational Experiences r pra

Supportive Campus Environment r pra

Critical thinking (CAAP)

.693*** .433*

.180

.392

.157

.351

.602***

.441*

.052

Psychological well-being (RYFF-SPWB)

.483**

.401*

.305

.394*

.311

.552**

.393

.635*** .725***

Need for cognition scale

.702*** .267

.283

.084

.372

.203

.357

.024

-.147

.303

Positive attitude toward literacy scale

.711*** .512**

.367

.329

.421*

.174

.326

.296

-.115

.389

Miville-Guzman universalitydiversity scale

.542**

.283

.349

.328

.277

.436*

.572**

.093

.483*

Openness to diversity scale

.643*** .302

.415*

.559**

.468**

.355

.509**

.410*

.051

.433*

Socially responsible leadership scales: Consciousness of self

.541**

.456**

.397

.473**

.343

.588***

.300

.512**

.435*

-.010

.411*

.233

Congruence

.219

.313

.325

.499** -.241

.317

-.042

.195

-.110

.430*


.280

NSSE Validation

54

Table 3 (continued)


Level of Academic Challenge r pra

Active and Collaborative Learning r pra

Student-Faculty Interaction r pra

Enriching Educational Experiences r pra

Supportive Campus Environment r pra

Commitment

.057

.044

.001

-.254

.155

-.290

.254

-.075

.448*

Collaboration

-.157

-.363

.006

-.263

.027

-.246

.176

-.166

.578*** .379

Common purpose

.225

-.127

.310

-.050

.226

-.054

.529**

.127

.649*** .478**

Controversy with civility

.638*** .625***

.236

.314

.415*

.342

.367

.385

-.079

Citizenship

.634*** .075

.485**

.026

.562**

.010

.714***

.217

.448*

.269

Change

.336

.299

.259

.281

.486**

.424*

.209

.182

.350

.666***

.677*** .394

.194

.098

.206

.303

.464**

.444*

-.167

Moral reasoning (DIT-N2 score)

a

Controlling for the specific liberal arts outcome precollege score. *p < .10. **p < .05. ***p < .01.


.241

.125

.053

NSSE Validation

Table 4 Zero-order (r) and Partial (pr) Correlations Between NSSE Deep Learning Scales and First-Year Liberal Arts Outcomes (Institutions as Unit of Analysis, n = 19) Higher Order Learning r pra


Integrative Learning r pra

Reflective Learning r pra

Deep Learning Total Score r pra

Critical thinking (CAAP)

.445**

.280

.185

.283

.341

.067

.332

.235


.413*

.254

.194

.050

.106

.066

.250

.122


.735*** .310

.718***

.465*

.653***

.562**

.800*** .502**


.758*** .384

.764**

.124

.865***

.124

.841*** .252


.632*** .342

.663***

.269

.714***

.082

.710*** .271


.746*** .435*

.769*** .591***

.741***

.387

.802*** .546**


.586*** .423*

.392*

.287

.293

.193

.449*

.317

.221

.033

-.216

.026

-.231

.094

-.240

Congruence

-.235


55

NSSE Validation

Table 4 (continued) Higher Order Learning r pra


Integrative Learning r pra

Reflective Learning r pra

Deep Learning Total Score r pra

Commitment

-.079

-.146

-.299

-.257

-.318

-.148

-.252

-.203

Collaboration

-.208

-.463*

-.354

-.515**

-.429*

-.457*

-.352

-.509**

Common purpose

.176

-.124

-.099

-.196

-.234

-.275

-.057

-.208


.750*** .660***

.749***

Citizenship

.571**

-.165

.384

Change

.526**

.346

.622***

.345

.555**

.278

.609*** .355

.552**

.317

.404*

.355

.545**

.122

.521**


.549** -.124

.859*** .705***

.830*** .689***

.385

.457**

-.243

a

Controlling for the specific liberal arts outcome precollege score. *p < .10. **p < .05. ***p < .01.


-.186

.302

56

NSSE Validation

Table 5 Upper (U) and Lower (L) Bounds Estimates of the Standardized (β) Associations Between NSSE Benchmark Scales and First-Year Liberal Arts Outcomes (Individuals as Unit of Analysis)

Level of Academic Challenge Ua Lb

Liberal Arts Outcome Critical thinking (CAAP)

.197**


.082

Student Faculty Interaction Ua Lb

Supportive Campus Environment Ua Lb

.056***

-.096

.191*** .050**

.118*** .087***

.032


.233**

.047

.136***

.041**

.093**

.058

-.074

.004


.150**

.020

.123***

.041

.111**

.060

-.030

.050**


.195**

.110***

.119**

.038

.039

.009

.021

.050**


.198**

.119***

.093

.057**

.066

.033

.056

.085***


.202*** .136***

.061

.004

.060

.016

.104**

.052

.241*** .126***

.060

.068**

.018

.012

.046

.014

Congruence

-.012

Active and Collaborative Learning a U Lb

-.049**

-.025

-.001

.208*** .130***


.020

57

NSSE Validation

Table 5 (continued)

Level of Academic Challenge Ua Lb


Active and Collaborative Learning a U Lb

Student Faculty Interaction Ua Lb

Supportive Campus Environment Ua Lb

Commitment

.281*** .152***

.014

.027

.013

.015

.051

.034

Collaboration

.185**

.100**

.082***

.044

.031

.132**

.093***

Common purpose

.236*** .125***

.077

.068

.015

.014

.103**

.070**


.206

.067**

.032

.056

.064

.066**

.062**

Citizenship

.203*** .064**

.176*** .099***

.096**

.034

.036

.035

Change

.133**

.133**

.092*

.053

.097**

.072***


.094***

.120***

.055**

.213*** .050

.088***

.108*** .073***

-.034

.001

a

-.132*** -.042

Upper-bounds estimates. Regression equations include controls only for all other benchmark scales. Lower-bounds estimates. Regression equations include controls for: all other benchmark scales, a parallel precollege measure of each liberal arts outcome, tested precollege academic preparation, sex, race (White, non-White), average parental education, a measure of high school involvement, a measure of precollege academic motivation, hours of on- and off-campus work during college, place of residence during college, the liberal arts emphases of one’s first-year coursework in college, institutional type (research university, regional university, or community college vs. liberal arts college), and the clustering effect. **p < .05. ***p < .01.

b


58

NSSE Validation

Table 6 Upper (U) and Lower (L) Bounds Estimates of the Standardized (β) Associations Between NSSE Deep Learning Scales and First-Year Liberal Arts Outcomes (Individuals as Unit of Analysis) Higher Order Learning a U Lb

Liberal Arts Outcome Critical thinking (CAAP)

-.051


Reflective Learning Ua Lb

.009

.207***

.165*** .067**

.126***

.037

.105*** .070***

.316*** .138***


.090

.021

.180*** .072***

.261*** .101***

.421*** .152***


-.019

-.016

.244*** .108***

.251*** .071***

.377*** .127***


.069

.052

.151*** .075***

.291*** .156***

.402*** .224***


.079**

.052**

.114*** .094***

.377*** .234***

.471*** .294***


.179*** .132***

.105***

.014

.113*** .088***

.317*** .188***

.143**

.099***

.055

.151*** .103***

.312*** .195***

.087**

.053**

Deep Learning Total Score Ua Lb

.024

Congruence

-.029

Integrative Learning a U Lb


.141**

.023

59

NSSE Validation

60

Table 6 (continued) Higher Order Learning a U Lb


Integrative Learning a U Lb

Reflective Learning Ua Lb

Deep Learning Total Score Ua Lb

Commitment

.159*** .084

.089***

.048

.105*** .085***

.282*** .174***

Collaboration

.166*** .094***

.122***

.062**

.117*** .093***

.324*** .198***

Common purpose

.163*** .099***

.074**

.023

.161*** .108***

.316*** .183***


.129*** .098***

.101**

.056**

.274*** .183***

.398*** .266***

Citizenship

.083

.029

.168***

.057

.183*** .112***

.344*** .156***

Change

.160*** .078

.135***

.064**

.164*** .098***

.364*** .191***

.019

.091

.085**

.148***

.205*** .100**


-.020

.057**

a

Upper-bounds estimates. Regression equations include controls only for all other benchmark scales (except in the case of the total score). Lower-bounds estimates. Regression equations include controls for: all other benchmark scales (except in the case of the total score), a parallel precollege measure of each liberal arts outcome, tested precollege academic preparation, sex, race (White, non-White), average parental education, a measure of high school involvement, a measure of precollege academic motivation, hours of on- and off-campus work during college, place of residence during college, the liberal arts emphases of one’s first-year coursework in college, institutional type (research university, regional university, or community college vs. liberal arts college), and the clustering effect. **p < .05. ***p < .01.

b


NSSE Validation RUNNING HEAD: NSSE Validation ... - CiteSeerX

NSSE Validation RUNNING HEAD: NSSE Validation ... - CiteSeerX

Suggest Documents

Running head: STUDENT AND FACULTY PERSPECTIVES ... - NSSE

Proportional Weight Calculation - NSSE

Proportional Weight Calculation - NSSE

Running head: CONSTRUCT VALIDATION IN ...

Running head: CONSTRUCT VALIDATION IN RESEARCH - SSRN

Running head: ROUTINE VALIDATION OF AUDIO-VISUAL

A cautionary exploration of how faculty - NSSE

Fostering student engagement campuswideâannual results ... - NSSE

Fostering student engagement campuswideâannual results ... - NSSE

Intercollegiate Athletes and Effective Educational Practices - NSSE

Validation of the Online Efficacy 1 Running Head ... - Creative Wisdom

The Contributions of Writing to Learning and Development - NSSE

Analyzing and Interpreting NSSE Data - Thompson Rivers University

The Institutional Quality Debate The Institutional Quality Debate - NSSE

STUDENT-FACULTY INTERACTIONS Faculty Lend a Helping ... - NSSE

DEEP Practice Brief Using Financial and Other Resources - NSSE

A Comparison of International Student and American Student - NSSE

STUDENT-FACULTY INTERACTIONS Faculty Lend a Helping ... - NSSE

Validating the National Survey of Student Engagement (NSSE) - ERIC

Nahj Al-Balagha Semantic Search Engine(NSSE) - Iraq Academic ...

Major Differences: Examining Student Engagement by Field of ... - NSSE

Major Differences: Examining Student Engagement by Field of ... - NSSE

Resource Allocation at DEEP Colleges and Universities - NSSE

The Institutional Quality Debate The Institutional Quality Debate - NSSE

NSSE Validation RUNNING HEAD: NSSE Validation ... - CiteSeerX