DISTANCE TO PROMOTION: EVIDENCE FROM MILITARY GRADUATE EDUCATION MARIGEE BACOLOD and LATIKA CHAUDHARY∗
Using a unique dataset of U.S. military officers enrolled in graduate programs at the Naval Postgraduate School, we find students enrolled in distance education programs are 19 percentage points less likely to graduate compared to students enrolled in comparable traditional resident programs. Interestingly, distance education students receive a larger proportion of As on their courses. But, they are also more likely to fail and withdraw from their courses compared to their resident counterparts. The negative effects of distance education are worse for students enrolled in more technical engineering programs compared to less technical business programs. Although distance students are more likely to separate from the military after completing their education compared to traditional students, there are no significant differences in job promotion within the military between the two groups. Our results highlight the challenges of designing effective distance education programs in technical fields. (JEL I20, I23) I.
has the potential to lower education costs that have increased in the past few decades (Deming et al. 2015). Despite the growth in new education technologies, policymakers and academics continue to debate whether and how medium of instruction affects course outcomes. While some studies find better student outcomes in online courses, many others find no significant difference in outcomes between the two modes of instruction. Our paper contributes to this literature using a rich student-level dataset to study the effect of distance and online education on academic and subsequent job-related outcomes.2 We follow seven cohorts of military students who began graduate programs at the Naval Postgraduate School (NPS) between 2006 and 2013. We match student transcript data on course outcomes and graduation to their demographic and career information reported in the military personnel database
INTRODUCTION
Online and distance education has exploded over the last decade in the United States and across the world. According to recent estimates, 11% of degree-seeking undergraduates are enrolled in completely online courses. While for-profit colleges have led the charge, public and nonprofit colleges are slowly catching up.1 Indeed, the latter colleges are now offering complete online degree programs similar to forprofit chains such as the University of Phoenix (McPherson and Bacow 2015). Given the cost advantage of online education, the transition away from traditional face-to-face instruction ∗ We thank two anonymous referees and the Editor for valuable feedback. We also thank Roberto Pedace, NPS seminar participants, and conference participants at the 2016 Western Economic Association Meetings for comments. All errors are our own. The views expressed in this article are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government. Bacolod: Associate Professor, Graduate School of Business and Public Policy, Naval Postgraduate School, Monterey, CA 93943. Phone 831 656 3302, Fax 831 656 3407, Email
[email protected] Chaudhary: Associate Professor, Graduate School of Business and Public Policy, Naval Postgraduate School, Monterey, CA 93943. Phone 831 656 7647, Fax 831 656 3407, E-mail
[email protected]
2. Hereafter we refer to online and distance education as “distance learning.”
ABBREVIATIONS GPA: Grade Point Average MBA: Masters in Business Administration MOS: Military Occupation Specialty NPS: Naval Postgraduate School OLS: Ordinary Least Squares
1. For example, the nonprofit company edX led by Harvard and MIT offers many top quality online courses to students worldwide. 1 Contemporary Economic Policy (ISSN 1465-7287)
doi:10.1111/coep.12275 © 2018 Western Economic Association International
2
CONTEMPORARY ECONOMIC POLICY
(Defense Manpower Data Center). This allows us to observe career outcomes such as promotion within the military and separation from the military after students leave NPS. NPS offers numerous master programs in technical and nontechnical fields.3 Students who enroll in traditional resident programs take courses at NPS’s physical location in Monterey, California. Students also enroll in distance programs, taking courses using a mix of online software and video teleconferencing. Distance learning students are physically separated from their instructors, living in or around military bases across the United States. Unlike full-time resident students, distance learning students typically have 1 day of academic instruction per week and continue working in their regular jobs the rest of the week. To ensure apples to apples comparison, we focus on students enrolled in four degree programs that offer a comparable resident and distance learning option with both leading to the same degree. The four programs are masters in business administration (MBA), masters in space systems operations, masters in systems engineering analysis, and masters in mechanical engineering. Since students enroll in either the resident or distance learning option for the entire degree program, we focus on graduation as the main academic outcome. In addition, we look at the share of courses each student failed and withdrew from, and the share of As, Bs, and Cs. Our main empirical challenge is the endogeneity of distance learning. The choice to enroll in a distance learning program may be endogenous to other factors correlated with academic and career success such as ability. To address such concerns we include a rich set of covariates including undergraduate grade point average (GPA), undergraduate mathematics background, the selectivity of their undergraduate institution, military rank, military occupation, and standard demographics such as race, age, and marital status. Undergraduate GPA and mathematics background are decent, though not ideal, proxies for student ability. We also estimate propensity score matching models matching students on the same set of covariates within programs and first year of study. That said, we recognize selection problems may remain, and our findings are perhaps more suggestive than conclusive on the causal effect of distance education.
Our results across both the ordinary least squares (OLS) and matching models point to a negative effect of distance education on academic outcomes. These effects are especially large and negative for students in engineering programs. Distance learning students in MBA programs are 10 percentage points less likely to graduate, while those in engineering programs are 27 percentage points less likely to graduate. Interestingly these differences are not driven by worse course performance. Distance MBA students receive a larger proportion of As compared to resident students. Rather, distance students are less likely to graduate because they are more likely to fail and withdraw from their courses with similar effect sizes for both MBA and engineering programs. We believe this is perhaps driven by the competing demands of job and school. Other studies have also noted the problem of endurance and motivation among students taking online courses (Bettinger et al. 2017; Xi and Jaggars 2013). Our results also echo Anstine and Skidmore (2005) that find MBA students taking an online statistics course (more technical) perform worse than students taking an online managerial economics course (less technical). In terms of job outcomes, we find no robust effects of distance education on promotion within the military. While the matching results point to negative and marginally significant effects, the OLS results are smaller and statistically indistinguishable from zero. In contrast to promotion, distance students in engineering programs are 6–11 percentage points more likely to separate from the military. It is unclear if separation here is a positive or negative labor market outcome, however. If students are leaving the military to pursue better civilian opportunities, then distance learning students fare better. But, if distance students are being pushed out of the military, it is a negative outcome. Unfortunately, we cannot distinguish between the two possibilities using our data. To test for heterogeneous differences, we pool students from both the MBA and engineering programs, and then split them by ability such as students with an average undergraduate GPA of an A and students with a more rigorous undergraduate mathematics background. In these models we include fixed effects for the four academic programs.4 Similar to both MBA and engineering students, high ability students
3. In 2013 NPS offered 81 Master’s and 16 Ph.D. degree programs.
4. We do not have sufficient observations in each cell to construct samples by ability and engineering/MBA programs.
BACOLOD & CHAUDHARY: DISTANCE TO PROMOTION
enrolled in distance programs are less likely to graduate. Such students earn a larger proportion of As compared to their resident counterparts, but they are also more likely to withdraw from a larger proportion of their courses. We find similar patterns for students with an undergraduate mathematics background. Our paper contributes to the large literature on online and distance education. While our setting is unique with its focus on military graduate students, it offers a few advantages. First, we have access to rich data on student ability that allow us to control for selection into a distance program, at least on the ability dimension. Second, we follow the career trajectory of students after they leave NPS. While there are numerous studies linking the mode of education delivery to academic outcomes, few studies track job outcomes. Taken together our results point to significant negative effects of distance education on academic outcomes with larger effects for students enrolled in more technical engineering programs. While online programs are cheaper to operate, our findings suggest students struggle to complete such programs especially in more challenging fields. Moreover, the negative effects on academic outcomes hold even among high ability students. Interestingly, the negative academic outcomes do not translate into worse job outcomes such as promotion within the military. The rest of the paper is organized as follows. We review the related literature in the next section. Section III describes our data and institutional setting. We present the results in Section IV and conclude in Section V. II.
RELATED LITERATURE
A large and growing literature has studied the impact of course delivery on learning outcomes. Older studies focused on distance education when universities had remote campuses for students who were unable or unwilling to enroll in traditional campuses with face-to-face instruction. As the technology of delivering course and degree content has evolved, several studies have compared the impact of pure online instruction, blended instruction (combining some elements of online instruction with some faceto-face meetings), and traditional instruction on student outcomes as captured by either test scores or course grades. There are several meta-analyses of how the mode of delivery impacts student outcomes. Since the NPS distance programs incorporate
3
elements of both distance and online education, we begin by discussing two relevant metaanalyses. First, Bernard et al. (2004) compares the performance of distance education to traditional instruction. A defining characteristic of distance education is instructors being in separate locations from their students using some form of video teleconferencing for either lecture delivery or office hours. That said, in recent years distance education has embraced aspects of online education with taped lectures posted on course websites, online course assignments, and tests.5 So distance education today can be viewed as a type of online education. Distance education courses are either offered as synchronous courses with the instructor teaching in the same fashion as in a traditional course via a video teleconferencing program and students viewing the lecture in real time. In contrast, asynchronous distance education courses allow students to watch previously recorded lectures at their own convenience. Although Bernard et al. (2004) find no significant difference in outcomes between distance and traditional courses, the average effects mask differences between asynchronous and synchronous courses. Asynchronous distance education is correlated with better student outcomes compared to traditional courses, while synchronous distance education leads to worse student outcomes. Their study also notes the wide range of estimates in the literature that are often driven by different estimation strategies. The older studies tend to use observational data without statistically correcting for the selection of students into courses, making it difficult to draw strong conclusions. A meta-analysis by Means et al. (2010) comparing online and traditional instruction explicitly focuses on a small number of studies with a randomized or quasi-randomized study design. They find no significant difference in outcomes between online and traditional courses, but they find small and positive effects of blended courses on outcomes relative to traditional courses. As in other policy contexts, randomizedcontrol trials are the gold standard for estimating causal effects. Although the number of randomized studies is increasing (Alpert, Couch, and Harmon 2016; Bowen et al. 2014; Figlio, Rush, and Yin 2013; and Joyce et al. 2015), randomized experiments in this particular context 5. For example, the platform Aplia offers online assignments for many economics courses that are used at NPS by instructors offering traditional and distance/online courses.
4
CONTEMPORARY ECONOMIC POLICY
have their shortcomings. First, the sample sizes are small because of low participation rates. Recent efforts by Figlio, Rush, and Yin (2013) get around this problem by offering students extra credit to participate in the experiment. As in our context, external validity is another problem because most randomization and observational studies focus on individual courses offered at a few institutions, or focus on a single institution. Second, experiments suffer from contamination because it is quite likely that freshmen enrolled in the same section, for example, of an introductory economics course speak to each other about their class experiences (online or traditional). Survey questions rarely ask about such interactions. Third, students in traditional courses access all sorts of online materials so these studies are rarely estimating the impact of pure online instruction because we do not know how many students access online materials (e.g., Khan Academy videos on YouTube). Despite the limitations, randomized studies find no significant differences on average between the different instructional modes. However, there are differences by student background. Students from disadvantaged backgrounds do worse in online and blended classrooms compared to under traditional instruction (Alpert, Couch, and Harmon 2016). In contrast, some studies in economics have used observational data with corrections for the selection problem. For example Coates et al. (2004) had students take tests before and after enrolling in an introductory principles course across three campuses. Students choose whether to take the course online or face-to-face, but the authors used a rich background survey instrumenting for course choice using commute time to campus and general familiarity with online platforms. Correcting for selection, they found students perform worse in online courses. Importantly, they show that their OLS estimates suggest no significant difference in outcomes, which highlights the importance of correcting for selection when using observational data. In a similar vein Cosgrove and Olitsky (2015) combine difference-in-difference with matching to address the selection problem. Using test scores at multiple points in time, they also find no significant differences in outcomes between blended and traditional courses. However, they find that students in traditional introductory economics courses have higher
content retention compared to students enrolled in blended classrooms. Finally Bettinger et al. (2017) employ an instrumental variables approach on a large database of courses taken by students at a nonselective for-profit college, with the interaction of students’ distance to the local college and course offerings as an instrument for online coursetaking. They find that grades are significantly lower both for the course taken online and future courses. Furthermore taking a course online reduces the probability of remaining enrolled in college a year later by over 10 percentage points. Our study is both similar and different to these observational studies. We also use rich student information combined with propensity score matching to address selection into degree type like Cosgrove and Olitsky (2015). Unlike other studies, however, we study labor market outcomes in addition to graduation and other academic outcomes. Since our students enroll in a distance or resident program, we cannot separately identify the impact of delivery mode on individual courses. This is especially true for courses in a sequence where the online mode may work well for one course in the sequence but not the other. That said we show the range of effects on the proportion of As, Bs, and Cs received by a student in addition to their graduation. A recent study using fictitious resumes finds that online business degree holders are less likely to get job call-backs (Deming et al. 2016). Although very interesting, their study speaks to the perception of online degrees in the labor market. In contrast, our study focuses on online degrees offered by an accredited university with both distance and resident programs. We speak directly to the impact of such nonresident degrees on subsequent labor market success.
III.
INSTITUTIONAL BACKGROUND AND DATA
A majority of NPS students are military officers. Although civilian students also attend NPS, we focus on U.S. military officers because we have access to their demographic and career information. The selection of military officers for graduate study varies by branch of service. It is usually based on outstanding job performance, promotion potential, academic background, and military needs. NPS determines academic eligibility, while each service has a selection board or an assignment officer that screens eligible officers. Hence for many resident students they have
BACOLOD & CHAUDHARY: DISTANCE TO PROMOTION
limited choice in coming to NPS and in their program of study. Distance education students either choose to pursue an NPS graduate degree, or are encouraged to do so by their command. In both cases, students require approval from their military supervisors attesting they can juggle academic courses with their regular job. These students spend 1 day a week taking courses either online or via a video-teleconferencing program. Both options normally involve a live instructor at the other end. Their typical course load is two courses per quarter. In contrast, resident students typically take four courses per quarter. After receiving their degree, resident and distance students complete an obligated military service commitment ranging from 3 to 4 years. Since resident students are full-time students, they presumably have more time to devote to their course work. This difference is explicit in our context, but students in nonmilitary contexts, such as in online degree programs, are also more likely to work full-time or part-time compared to students enrolled in traditional degree programs. In terms of job outcomes, the career paths of resident and distance learning students are similar within Military Occupation Specialty (MOS). An individual is assigned an MOS when they enter the military. And, it is based largely on their performance during initial officer training. Since the military is an internal labor market with no lateral entry at advanced ranks, promotion to higher officer ranks are based on military needs for those occupations, the officer’s time in service, and his or her job performance. Since we match within MOS in addition to rank and age, our results on promotion are likely to be good signals of job productivity in the military after NPS. While NPS offers many degrees, we focus on four degrees with a comparable resident and distance education program. They are (1) MBA, (2) space systems operations, (3) mechanical engineering, and (4) systems engineering analysis.6 The normal time to degree completion ranges from 18 to 24 months across these programs. As we explain in Section IV, we estimate the distance 6. The mechanical engineering degree is short for M.S. in Engineering Science with a focus on mechanical engineering. Students in this distance learning option of this degree program have normally completed an officer’s course at the Naval Nuclear Power School and must have a B.S. in Engineering. This degree program is offered to distance students while they are deployed. While an engineering degree is recommended for the resident mechanical engineering students, it is not required.
5
effects separately for the MBA and engineering programs. Our data include all students that enrolled in the resident and distance learning options of the four mentioned programs beginning in the academic year 2006 through 2013. We used the Defense Manpower Data Center personnel files to obtain information on gender, age, race, marital status, dependents, military rank, military occupation, promotion after leaving NPS and separation from the military after leaving NPS. Our demographic data such as marital status and number of dependents are drawn as of 6 months prior to students beginning their studies at NPS. Our data also include the name of the institution that granted each student’s undergraduate degree.7 To capture the selectivity of a student’s undergraduate education, we code each individual’s undergraduate institution as either public, private nonprofit, private for-profit, or military academy. The coding is based on the Integrated Postsecondary Education Data System (IPEDS) classification of colleges. In addition, we construct measures of student ability using information on their undergraduate performance. NPS constructs an academic profile code for each student based on the GPA reported on their previous college transcripts. While we do not have a student’s exact undergraduate GPA, we know whether they were an A average, B average, or below B average student. Specifically, we create an indicator for students with an average undergraduate GPA of 3.2 and above. We refer to this group as the high ability A students. We also create indicators for students with an average undergraduate GPA of 2.6–3.19 (B students) and students with an undergraduate GPA below 2.59 (C students). And, we create an indicator for students that were undergraduate math majors or minors or had taken both upper level and lower level math courses. We refer to this group as students with an undergraduate mathematics background. While course grades and student GPAs are available in our data, we focus on course withdrawals and incompletes, in addition to graduation and later career outcomes in our analysis. Ideally we would like to be able to measure student learning, but grades would at best be very noisy signals of what students learn in their courses. The assignment of grades also varies 7. If a student obtained more than one degree or attended multiple institutions, this would be the last institution attended that granted the most recent undergraduate degree.
6
CONTEMPORARY ECONOMIC POLICY
TABLE 1 Characteristics of Resident and Distance Learning Students MBA Only
White Female Black Hispanic Married Age Age—missing Undergraduate—Public University UG—Service Academy UG—IPEDS missing Navy Military Occupation—Surface Military Occupation—Submarine Military Occupation—Aviation Military Occupation—Ground Combat Military Rank—O3 Military Rank—O4 and above UG—GPA, A UG—GPA, B UG—Math UG—Grade, missing System engineering Space systems operations Mechanical engineering Observations
Engineering Only
Resident Mean
DL Mean
Resident Mean
DL Mean
0.695 0.083 0.101 0.067 0.754 32.18 0.034 0.462 0.146 0.059 0.59 0.105 0.033 0.099 0.064 0.513 0.473 0.364 0.432 0.221 0.019
0.717 0.045 0.053 0.064 0.749 30.615 0.096 0.392 0.298 0.064 0.985 0.053 0.029 0.709 0.012 0.572 0.427 0.34 0.403 0.321 0.129
912
729
0.751 0.096 0.074 0.07 0.659 30.106 0.004 0.391 0.346 0.074 0.883 0.276 0.084 0.174 0.033 0.736 0.217 0.354 0.436 0.524 0.049 0.515 0.188 0.297 511
0.764 0.051 0.038 0.058 0.781 31.045 0.048 0.226 0.366 0.295 0.747 0.092 0.325 0.257 0.014 0.558 0.38 0.421 0.397 0.425 0.062 0.473 0.116 0.411 292
Notes: Entries are means within each cell. UG refers to undergraduate degree. Please refer to the text for specific details on the student sample. DL, distance learning.
considerably from program to program, instructor to instructor, course to course, and importantly in our context, distance versus face-to-face even for the same instructor and course. In addition, overall GPA varies among our students depending on when they withdraw from their programs. For all these reasons we have chosen not to focus on GPAs as outcomes. IV.
FINDINGS
A. Differences between Distance and Resident Students Table 1 reports summary statistics by resident and distance separately for the MBA and engineering programs. Since the engineering programs are more technical than the MBA, we conduct the analysis separately for the two groups. In terms of demographics, distance students are different from resident students in a few ways. Women are more represented in resident programs, at 8.3% female in resident MBA programs compared to 4.5% in distance. Minorities are also more represented in resident programs. For
instance, Blacks constitute only 3.8% of distance engineering students but are 9.6% of resident engineering students. In engineering Hispanics are slightly more represented in resident programs at 7% compared to 5.8% in distance. Marital status is similar across the two groups. Distance students are a tad younger on average, but there is also more variation in their age distribution than among residents. Navy students are disproportionately represented in the distance MBA programs, at 98% compared to 59% in the resident programs. Within military occupations, pilots are more likely to enroll in distance education often so they can maintain flight hours, a necessary requirement for their jobs. In contrast, surface warfare and ground combat officers are more likely to attend resident programs. A majority of students in both programs are early career officers of rank O3, which corresponds to a Lieutenant in the Navy and a Captain in the other services. Finally, Table 1 shows a mix of patterns in terms of academic preparation between distance and resident students. NPS students from
BACOLOD & CHAUDHARY: DISTANCE TO PROMOTION
service academies that generally would be considered selective undergraduate institutions are more likely to be represented among distance MBA students at 29.8% compared to 14.6% among resident MBA students. Interestingly, the engineering distance student has a higher undergraduate GPA. Since we have a few students reporting missing age and undergraduate GPA information, we code them as 0s with a separate missing indicator for each of those variables (age and ability).8 Next we turn to examining differences in outcomes of these groups. B. Estimation Methods To compare the academic and job outcomes of distance and resident students, we estimate the following linear regression: Yi = α + β′ Xi + τDistancei + εi , where Y i are the outcomes: graduation, course grade distribution including withdrawals and incompletes, and after leaving NPS, military job promotion, and separation from the military. X i denotes student i’s observable characteristics. Here we include a rich set of covariates including age, sex, race, marital status, number of dependents, indicators for undergraduate institution attendance at public colleges and service academies, indicators for Navy officers, for rank O3 and O4/O5, indicators for the military occupations of aviation, surface warfare, submarine, and ground combat, and indicators for students with an undergraduate GPA of A, undergraduate GPA of B, and an undergraduate mathematics background. Finally, we include cohort fixed effects for the year an individual began their graduate program at NPS. Below, we report both our OLS and nearest-neighbor matching estimates of τ. In our implementation of nearest-neighbor matching, we match with replacement and match with all ties. We also regression-adjust the standard errors of our matching estimates of τ per the Abadie and Imbens (2006, 2009) correction. The adjustment is necessary for removing biases, because the propensity score in the first stage of matching is an estimate.9 Including these particular covariates in the OLS and matching models is important, as they 8. Our results are unchanged if we drop students with missing age and undergraduate GPA information. 9. Please refer to Appendix S1, Supporting information, for more details on our matching estimation.
7
serve to control for each student’s characteristics that influence both their assignment to a resident versus distance learning program as well as their educational and subsequent job performance outcomes. The demographic factors are standard in the literature, and are important to include as they are likely to be correlated with our outcomes and with selection into distance learning. Undergraduate institution type, math background, and GPA capture characteristics of a student’s prior education and ability. We also include indicators for MOS and service branch because as described above, career paths are largely deterministic within these occupations. Identifying the effects of distance education on student educational outcomes and subsequent job performance is fraught with empirical difficulties. For example, if students who are more motivated and productive tend to attend resident programs, then the differences in outcomes may be due to unobserved motivation. At the same time, positive selection into distance learning may also occur. This would be the case if the more productive students (e.g., those who are more able to manage their time) tend to enroll in distance programs. An appeal of our setting is that enrolling in a distance versus resident degree program is often a function of military needs rather than individual student characteristics. Our matching estimates use the same set of covariates to match as those included in our OLS models, namely demographics, military rank, military occupation, service, and undergraduate background. This addresses some selection issues and also controls for student heterogeneity. Finally, the differences between distance and resident students shown in Table 1 do not provide evidence of strong positive or negative selection into either degree mode. C. Differences in Academic and Job Outcomes Table 2 presents our first results on students enrolled in the MBA program. We focus on graduation as the main academic outcome because it offers a clear interpretation compared to GPA that may be contaminated due to differences in tests and grading practices across resident and distance programs. We also show results on the share of courses failed, withdrawn, and not completed in addition to the distribution of grades. Apart from the academic outcomes, we also show results on the job outcome variables described above. As seen in Table 2, distance MBA students are 6–10 percentage points less likely to graduate,
8
CONTEMPORARY ECONOMIC POLICY
TABLE 2 Differences in Student Outcomes, MBA Degrees Dependent Variable Mean
Distance Learning Coefficient
Dependent Variables
Resident (1)
Distance Learning (2)
OLS (3)
Matching (4)
Education outcomes Graduate (0/1)
88.49%
83.40%
−0.1046*** [0.0226]
−0.0615*** [0.0187]
Share of courses Failed
0.09%
0.14%
Withdrawn
0.24%
2.20%
Incomplete
0.02%
0.00%
As
58.60%
78.75%
Bs
18.29%
17.84%
Cs and Ds
1.31%
0.81%
0.0004 [0.0009] 0.0278*** [0.0062] −0.0003** [0.0001] 0.1734*** [0.0119] 0.0156 [0.0101] 0.0004 [0.0024]
0.0003 [0.0005] 0.0169*** [0.0038] −0.0002* [0.0001] 0.1665*** [0.0112] 0.0396*** [0.0109] −0.0031** [0.0013]
Employment outcomes Promoted after NPS
52.61%
52.42%
Separated from Military
19.55%
28.33%
−0.0244 [0.0347] 0.0444 [0.0305]
−0.0908*** [0.0344] 0.1503 [0.1452]
Notes: Robust standard errors in brackets. OLS models include an indicator for female, black and Hispanic, age, an indicator if age was missing, an indicator if the student attended a public undergraduate institution, an indicator if the student attended a selective military service academy, an indicator for Navy officers, indicators for the military occupations of Aviation, Surface Warfare, Submarine, and Ground Combat, indicators for military rank O3, O4/O5 (one variable), an indicator if the undergraduate GPA was an A, an indicator if the undergraduate GPA was a B, an indicator if student had an undergraduate mathematics background, indicators if the student met the academic requirements of their NPS degree program, and a missing indicator of the academic preparation code used to construct the undergraduate and academic requirement variables. We also include fixed effects for the year in which the student begins their program at NPS. We used nearest neighbor matching reporting the correct Abadie-Imbens standard errors. We match on the same set of covariates including indicators for the different programs and year of entry in our matching estimation. We use 1,641 observations for the education outcomes, and 1,540 observations for employment outcomes. *p < .1, **p < .05, ***p < .01.
a sizeable effect given the mean graduation rate of 85%. The course-level data suggest the lower graduation rates are not driven by worse grades. Rather, distance MBA students earn a larger share of As in their course work compared to resident students. But, distance students are also more likely to fail and withdraw from their courses (1.7 percentage points higher than resident MBA students). This suggests distance students have lower motivation to complete their course work and graduate. Several plausible mechanisms could explain the lower graduation rates of distance students. First, distance and resident students face different time constraints. While distance students have a lower typical course load (2 courses per quarter vs. 4 for resident), these officers also have to perform the duties and tasks of their regular job the rest of the week. In contrast, resident students have relatively minimal military job tasks to perform at NPS, such as physical training
and occasionally standing “on duty” at the barracks. Second, the distant mode of educational delivery could significantly alter the nature of interactions between students and professors, and students and their peers. Distance students likely feel less oversight by their professors and diminished engagement with their student peers, lowering their persistence in their graduate program. While we find a large and negative impact of distance MBA education on graduation, we find mixed results on career outcomes after leaving NPS. While the matching estimate on job promotion indicates MBA distance students are significantly less likely to be promoted after NPS, both OLS estimates on promotion and military separation are statistically indistinguishable from 0. Officers enrolled in distance MBA programs are not significantly more likely to separate after NPS.
BACOLOD & CHAUDHARY: DISTANCE TO PROMOTION
9
TABLE 3 Differences in Student Outcomes, Engineering Degrees Dependent Variable Mean
Distance Learning Coefficient
Dependent Variables
Resident (1)
Distance Learning (2)
OLS (3)
Matching (4)
Education outcomes Graduate (0/1)
82.58%
44.18%
−0.2731*** [0.0341]
−0.2653*** [0.0605]
Share of courses Failed
0.19%
1.19%
Withdrawn
0.42%
2.93%
Incomplete
0.04%
0.23%
As
49.69%
47.35%
Bs
19.46%
12.65%
Cs and Ds
1.77%
0.34%
0.0134*** [0.0051] 0.0231*** [0.0078] 0.0033 [0.0023] −0.0105 [0.0181] −0.0442*** [0.0146] −0.0054** [0.0026]
0.0099* [0.0060] 0.0139** [0.0064] 0.0047 [0.0038] −0.0177 [0.0279] −0.0408* [0.0210] −0.0008 [0.0067]
Employment outcomes Promoted after NPS
50.29%
38.85%
Separated from Military
17.09%
21.22%
−0.058 [0.0442] 0.0641* [0.0375]
−0.1628*** [0.0332] 0.1196** [0.0505]
Notes: Robust standard errors in brackets. OLS models include an indicator for female, black and Hispanic, age, an indicator if age was missing, an indicator if the student attended a public undergraduate institution, an indicator if the student attended a selective military service academy, an indicator for Navy officers, indicators for the military occupations of Aviation, Surface Warfare, Submarine, and Ground Combat, indicators for military rank O3, O4/O5 (one variable), an indicator if the undergraduate GPA was an A, an indicator if the undergraduate GPA was a B, an indicator if student had an undergraduate mathematics background, indicators if the student met the academic requirements of their NPS degree program, and a missing indicator of the academic preparation code used to construct the undergraduate and academic requirement variables. We also include fixed effects for the three engineering programs and for the year in which the student begins their program at NPS. We used nearest neighbor matching reporting the correct Abadie-Imbens standard errors. We match on the same set of covariates including indicators for the different programs and year of entry in our matching estimation. We use 803 observations for the education outcomes, and 787 observations for employment outcomes. * p < .1, ** p < .05, *** p < .01.
In Table 3 we present our second set of results on students enrolled in engineering programs. Here we find an even larger negative effect of distance education on academic outcomes. Graduation rates for students enrolled in engineering distance programs are 26.5 percentage points lower than resident students. In the case of MBA students the gap was only 6 percentage points. We also find differences in job outcomes. The promotion rates of former distance students from engineering programs are 16.3 percentage points lower than resident students, while the promotion rates of distance MBA students are 9.1 percentage points lower. That said, these effects are not robust across the matching and OLS models. With regard to separation, officers enrolled in distance engineering programs are 12 percentage points more likely to separate from the military than resident students. But, there is no significant difference in separation among
MBA students. These differences by programs suggest the negative effects of distance learning as a mode of delivery on both academic and job outcomes are concentrated among officers in the engineering programs. We find even more interesting results on the grade distribution. Distance MBA students are more likely to get As than comparable resident students. In contrast, distance students in the engineering programs are more likely to fail their courses than resident students. However, distance students in both the MBA and engineering programs withdraw from a larger proportion of their courses. Rather than a simple story of negative selection into distance learning programs, these results point to the importance of endurance in distance programs. Student performance tends to be more extreme in distance education with not only more As, but also more Fs and course withdrawals.
10
CONTEMPORARY ECONOMIC POLICY
TABLE 4 Differences in Student Outcomes By Student Ability
Dependent Variables Education outcomes Graduate (0/1) Share of courses Failed Withdrawn Incomplete As Bs Cs and Ds Employment outcomes Promoted after NPS Separated from Military
Undergrad GPA = A
Undergrad Math = 1
Distance Learning Coefficient
Distance Learning Coefficient
OLS (1)
Matching (2)
OLS (3)
Matching (4)
−0.1852*** [0.0332]
−0.1719*** [0.0464]
−0.2295*** [0.0284]
−0.2248*** [0.0273]
−0.0003 [0.0005] 0.0284*** [0.0090] 0.0008* [0.0005] 0.0901*** [0.0179] −0.0206 [0.0126] −0.0050** [0.0020]
−0.0003 [0.0007] 0.0181*** [0.0070] 0.0020* [0.0011] 0.0998*** [0.0204] −0.0107 [0.0122] −0.0048*** [0.0011]
0.0033 [0.0022] 0.0185*** [0.0062] 0.0032 [0.0022] 0.0383** [0.0178] 0.0144 [0.0136] −0.0028 [0.0023]
0.0046** [0.0023] 0.0180*** [0.0052] 0.0037 [0.0023] 0.0622*** [0.0178] 0.0033 [0.0147] −0.0031** [0.0014]
−0.0885** [0.0450] 0.0797** [0.0384]
−0.0650 [0.0542] 0.1133*** [0.0439]
−0.0531 [0.0411] −0.0078 [0.0381]
−0.1076** [0.0479] 0.0741 [0.0518]
Notes: Robust standard errors in brackets. OLS models include an indicator for female, black and Hispanic, age, an indicator if age was missing, an indicator if the student attended a public undergraduate institution, an indicator if the student attended a selective military service academy, an indicator for Navy officers, indicators for the military occupations of Aviation, Surface Warfare, Submarine, and Ground Combat, indicators for military rank O3, O4/O5 (one variable), an indicator if the undergraduate GPA was an A, an indicator if the undergraduate GPA was a B, an indicator if student had an undergraduate mathematics background, indicators if the student met the academic requirements of their NPS degree program, and a missing indicator of the academic preparation code used to construct the undergraduate and academic requirement variables. We also include fixed effects for the four degree programs and for the year in which the student begins their program at NPS. We used nearest neighbor matching reporting the correct Abadie-Imbens standard errors. We match on the same set of covariates including indicators for the different programs and year of entry in our matching estimation. * p < .1., ** p < .05, *** p < .01
It seems there is no negative stigma associated with a distance degree in the internal military labor market. In our context, this finding is perhaps unsurprising. Since the military regularly sends officers to NPS for graduate education, they are familiar with the degree programs offered by the school. In these cases the distance degrees are less likely to represent a signal to future employers as it may in other labor markets. Unlike recent studies that have documented a negative effect of online degrees on job call-backs (Deming et al. 2016), our labor market results suggest students in distance programs have worse learning outcomes but they do not perform worse in their subsequent military jobs. D. Differences in Outcomes by Ability In Table 4 we find the negative effects of distance are also visible among high ability students,
those with an average undergraduate GPA of an A and those with an undergraduate mathematics background. Both groups of students in distance programs have lower rates of graduation similar to the results reported in Tables 2 and 3. Students with a high undergraduate GPA are more likely to withdraw from their courses, but not fail. And, they are also more likely to receive As. Interestingly, we observe this group of students is also more likely to separate from the military. It is unclear if that is due to better civilian labor market opportunities or their distance education. Students with strong prior math backgrounds are more likely to fail their courses in distance learning programs, but there is no significant effect of distance learning on promotion or separation for this population. Hence, ability seems to matter less in explaining outcomes compared to the technical rigor of programs.
BACOLOD & CHAUDHARY: DISTANCE TO PROMOTION V. CONCLUSION
Using a unique student-level dataset on academic and job-related outcomes, we estimate the impact of distance education programs on economic outcomes ranging from graduation to job-promotion. In such studies the main challenge of estimating causal effects is the nonrandom assignment of students to different program types. To address this problem, we implement propensity score matching using a rich set of observable characteristics of students. Our matching exercise suggests a large, negative, and significant impact of distance education programs on academic outcomes but no robust negative effects on job performance. Students that enroll in distance programs at NPS are less likely to graduate, and are more likely to fail and withdraw from their courses. The negative effects are more significant for students enrolled in technical engineering programs compared to those in less technical MBA programs. Even high ability students struggle with their courses in distance programs withdrawing from a larger proportion of their courses. But, interestingly we find no robust negative effects of distance education on promotion with the military. Overall, our results suggest less technical degrees may be more suitable to online or distance education than more technical engineering degrees. REFERENCES Abadie, A., and G. Imbens. “Large Sample Properties of Matching Estimators for Average Treatment Effects.” Econometrica, 74(1), 2006, 235–67. . “Matching on the Estimated Propensity Score.” NBER Working Paper 15301. Revised 2011, 2009. Alpert, W. T., K. A. Couch, and O. R. Harmon. “A Randomized Assessment of Online Learning.” American Economic Review Papers and Proceedings, 106(5), 2016, 378–82. Anstine, J., and M. Skidmore. “A Small Sample Study of Traditional and Online Courses with Sample Selection Adjustment.” Journal of Economic Education, 36(2), 2005, 107–27. Bernard, R. M., P. C. Abrami, Y. Lou, E. Borokhovski, A. Wade, L. Wozney, P. A. Wallet, M. Fiset, and B. Huang. “How Does Distance Education Compare with
11
Classroom Instruction? A Meta-Analysis of the Empirical Literature.” Review of Educational Research, 74(3), 2004, 379–439. Bettinger, E., L. Fox, S. Loeb, and E. Taylor. “Virtual Classrooms: How Online College Courses Affect Student Success.” American Economic Review, 107(9), 2017, 2855–75. Bowen, W. G., M. M. Chingos, K. A. Lack, and T. I. Nygren. “Interactive Learning Online at Public Universities: Evidence from a Six-Campus Randomized Trial.” Journal of Policy Analysis and Management, 33(1), 2014, 94–111. Coates, D., B. R. Humphreys, J. Kane, and M. A. Vachris. “‘No Significant Distance’ between Face-to-Face and Online Instruction: Evidence from Principles of Economics.” Economics of Education Review, 23(5), 2004, 533–46. Cosgrove, S., and N. H. Olitsky. “Knowledge Retention, Student Learning, and Blended Course Work: Evidence from Principles of Economics Courses.” Southern Economic Journal, 82(2), 2015, 556–79. Deming, D. J., N. Yuctman, C. Goldin, and L. F. Katz. “Can Online Learning Bend the Higher Education Cost Curve?” American Economic Review: Papers & Proceedings, 105(5), 2015, 496–501. Deming, D. J., N. Yuchtman, A. Abulafi, C. Goldin, and L. F. Katz. “The Value of Postsecondary Credentials in the Labor Market: An Experimental Study.” American Economic Review, 106(3), 2016, 778–806. Figlio, D., M. Rush, and L. Yin. “Is It Live or Is It Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning.” Journal of Labor Economics, 31(4), 2013, 763–84. Joyce, T., S. Crockett, D. A. Jaeger, O. Altindag, and S. D. O’Connell. “Does Classroom Time Matter?” Economics of Education Review, 46, 2015, 64–77. McPherson, M. S., and L. S. Bacow. “Online Higher Education: Beyond the Hype Cycle.” Journal of Economic Perspectives, 29(4), 2015, 135–54. Means, B., Y. Toyama, R. Murphy, M. Bakia, and K. Jones. Evaluations of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies. Washington, DC: U.S. Department of Education, 2010. Xi, D., and S. S. Jaggars. “Adaptability to Online Learning: Differences across Types of Students and Academic Subject Areas.” CCRC Working Paper No. 54, 2013.
SUPPORTING INFORMATION Additional Supporting Information may be found in the online version of this article: Appendix S1. Distance to Promotion: Evidence from Military Graduate Education
