Quality Determinants of Short-Cycle Programs in Latin America and the Caribbean
Box 4.1 Assessing the Effectiveness of Online and On-Campus SCPs in Colombia Enrollment in higher education has expanded dramatically in Colombia in recent decades. It doubled between 2000 and 2013, partly due to the opening of new short-cycle programs (SCPs) and the expansion of existing ones (Ferreyra et al. 2017; Carranza and Ferreyra 2019). An important percentage of SCP students take classes online. According to the Colombian Ministry of Education, about 359,020 students were enrolled in some form of hybrid or fully online higher education in 2017 (bachelor’s programs and SCPs), accounting for roughly 15 percent of the country’s total enrollment in higher education (SNIES 2019). The increased popularity of online education raises the question of its effectiveness. Despite the existence of several studies aiming to measure the effects of online education on student outcomes in Colombia, none of them has addressed the important issue of selfselection and thus, they do not provide causal estimates. In a background paper for this book, Cellini and Grueso (2020) implement propensity score weighting methods to address self-election into on-campus and online SCPs, to estimate the effectiveness of online relative to face-to-face on-campus programs. The authors exploit the unique system of college exit exams in Colombia to compare student performance on the exit exam for online and on-campus students. Their data include many students, programs, and institutions. Cellini and Grueso (2020) compare exam performance in quantitative analysis, reading, writing, and technical skills for students in online and on-campus SCPs. They find mixed results. First, in most institutions, online students appear to have significantly lower exit exam test scores (by about 0.04 standard deviation) than on-campus students, a result that holds for math, reading, and writing. These results are similar to the findings from Bettinger et al. (2017) using data from the United States. However, after including institution fixed effects, the estimations indicate positive and significant test score gains from online enrollment by about 0.09 to 0.11 standard deviation. To explore how institutional characteristics might be affecting the results, the authors analyze the relative effect of online programs in two samples. The first includes all programs offered by the largest public vocational institution, the National Learning Service (SENA, Servicio Nacional de Aprendizaje), and the second contains all non-SENA programs. The estimations using the second sample indicate that online students perform worse than those enrolled in on-campus programs on exit exams by about 0.06 standard deviation. Yet, in the first sample, SENA students who take online classes have exit test scores that are about 0.169 standard deviation higher than SENA students taking only on-campus courses. This result holds and is remarkably consistent in magnitude even when differences are considered only within the same degree and major. In other words, online programs are more effective than on-campus programs at SENA, but they are less effective at non-SENA institutions. Although Cellini and Grueso (2020) cannot rule out that SENA online students may be positively selected, interviews with SENA’s staff at its Department of Training, Virtual and Distance Training Group suggest that their online programs might have features that make them more successful than other online programs. For example, most online SENA classes are synchronous, meaning that students are (virtually) face to face with an instructor rather than watching prerecorded videos. SENA classes make project-based learning central to the student experience.
133
