Estimating the Impact of COVID-19 on Students’ Academic Outcomes
In May 2020, Ithaka S+R partnered with the Council of Independent Colleges (CIC) and a group of its member institutions to assess the impact of the COVID-19 pandemic’s disruption to the Spring 2020 term on students’ academic outcomes, and to inform subsequent institutional decision-making. We concluded our work with the first group of thirteen institutions earlier this month, and each participating institution received a customized confidential report of findings. A second group of up to six CIC institutions will join the project in early fall.
In this blog post, we describe the project and our methodology, and present preliminary high-level findings from the initial set of institutions. We also discuss our plans for similar research that aims to assist institutions in tracking student outcomes as they adjust course during the ongoing pandemic.
For this project, we collected and analyzed de-identified student demographic and outcomes data of degree-seeking undergraduates enrolled during the spring terms of 2016 through 2020. We estimated how students’ outcomes in the spring of 2020 compare to their otherwise anticipated outcomes in the absence of the pandemic. To do this, we regressed student outcome data during the spring terms of 2016 to 2019 on students’ demographic characteristics. The resulting constant and coefficient estimates were used to predict the outcomes of students enrolled in the Spring 2020 term given their student demographic characteristics. We also benchmarked each institution’s results against a set of self-selected peer institutions to allow institutions to further contextualize the results and assign meaning to them. All analyses were conducted across each institution’s full sample, as well as institutional subsamples to uncover potential disparate outcomes among particular student groups (e.g. Pell recipients and students from underrepresented minority groups). For more information on the student outcomes we analyzed and how the findings are presented, you can download a sample report.
The results described below reflect preliminary analyses across the twelve bachelor’s degree-granting institutions in our sample (one of the thirteen institutions in the first group is an associate degree-granting institution so we excluded them from this analysis). A few high-level findings comparing students’ actual outcomes to their predicted outcomes in the absence of the pandemic stand out. It’s worth noting that the differences described below are not large and have not been tested for statistical significance, and like all short-term findings, it’s unclear at this time whether they will persist beyond the Spring 2020 term.
1. Is grade inflation responsible for students’ higher-than-predicted term GPA?
Across the entire sample, students’ Spring 2020 term GPA was 0.16 GPA points higher than predicted despite students having a lower-than-predicted credit completion ratio. This may be driven partly by the fact that students passed half a credit more using the pass/fail option than predicted. Students typically use this option instead of a letter grade when they anticipate earning a relatively low grade in a particular course – as courses completed with a “pass” do not affect students’ GPA. Anecdotal evidence suggests that institutions gave students greater flexibility and time to use this option. Though it may have had a positive impact on GPA, it’s unclear what other impacts it may have on students’ long-term academic outcomes, such as enrolling in upper-level courses or applying to graduate schools that require students pass prerequisite courses with letter grades. An alternative explanation for the higher-than-predicted GPAs is that faculty graded students more leniently during what they may have perceived to be unfair or inadequate conditions for evaluating students.
2. Higher-than-predicted graduation rates were not evenly experienced across student subgroups.
Across the entire sample, the share of fourth-year standing students (typically defined at institutions on the semester system as students who have earned at least 90 cumulative credits) who graduated at the conclusion of the Spring 2020 term was about a percentage point higher than predicted. But this was not evenly experienced across student subgroups. In fact, the graduation rate of fourth-year standing Pell recipients was 1.2 percentage points lower than predicted; among underrepresented minority students, it was nearly 1 percentage point lower, and among female students, it was more than half a percentage point lower than predicted.
3. Pell recipients were most negatively impacted.
Based on the student outcomes we collected and analyzed, Pell recipients in our sample were most adversely affected by the pandemic. Besides having the largest deficit between actual and predicted graduation rates, the credit completion ratio of Pell recipients was nearly one percentage point lower than predicted, more than any other student subgroup. And while the GPA of Pell recipients in the Spring 2020 term was higher than predicted, the gap for this group was smaller than it was for the entire sample and any other student subgroup.
Helping institutions understand the impact of the pandemic on students’ outcomes is all the more important considering that many schools are offering most, if not all, Fall 2020 courses online as well. Although the Spring 2020 term is much more accurately described as emergency remote instruction than online learning per se, the Fall term retains emergency elements and uncertainty as the pandemic continues to unfold. Understanding impacts, and especially disparate impacts, during the Spring 2020 term can help institutions target resources and supports to the most affected students.
To help institutions understand whether the plans they developed and implemented over the summer were effective, and how their students are continuing to fare as the consequences of the COVID-19 pandemic continue to unfold, Ithaka S+R is developing a similar project to track and benchmark student outcomes on a continuous basis over the next several academic years. It will build off the lessons learned in the current project and include additional features and supports so institutions can make strategic use of project findings.
Ithaka S+R is seeking additional partnerships along with CIC’s continued involvement. If your institution is part of a broader group, network, or consortium of institutions that may be interested in participating, we invite you to contact us through the comments section below or at email@example.com to learn more.