Postsecondary outcomes for lower-income students have been disproportionately harmed by the COVID-19 pandemic. Intensifying and systematizing evidence-based student supports is a promising practice for helping these students. While initially conceived prior to the pandemic, Monitoring Advising Analytics to Promote Success (MAAPS) is one such project that aimed to learn whether and how technology-enhanced advising could better support low-income and first-generation students and promote equity.

From 2015 through 2022, the University Innovation Alliance (UIA) and its institutional members tested the effectiveness of MAAPS, an intervention consisting of intensive and proactive technology-enhanced advisement meant to increase the academic achievement, persistence, and degree completion of Pell-eligible and first-generation students. The project was tested as part of a randomized-controlled trial funded by a US Department of Education First in the World Grant with additional support from Arnold Ventures. Ithaka S+R has served as the independent evaluator, conducting impact and implementation studies and publishing reports on the findings (see previous reports here, here, and here).

Today we are publishing the final report on the impact findings of the MAAPS intervention on students’ outcomes after six academic years. We used data from the National Student Clearinghouse (NSC) to track the graduation and persistence rates of MAAPS students six years after they originally enrolled in a MAAPS institution, which included enrollment and graduation information for students who left their original MAAPS institution and transferred elsewhere. These two measures serve as the study’s primary outcomes. The report also summarizes findings from the implementation study conducted during the first three years of the project, which included interviews with project staff, student advising surveys, focus groups of participating students, and analysis of advising interaction data logged by MAAPS advisors.

Impact study findings

For the full study sample, assignment to the MAAPS advising intervention had no significant impact on either graduation or persistence.

Exploratory analyses of student subgroups revealed that first-generation students in the treatment group had a persistence rate that was two percentage points lower than first-generation students in the control group, driven in large part by enrollment differences in the Spring 2022 term.

Secondary analyses revealed significant impacts at Georgia State University (Georgia State), the only institution to run the intervention for six years. At Georgia State, after six academic years, treatment group students had a graduation rate that was seven percentage points higher than control group students. This impact was driven by Black students in the treatment group, who had a graduation rate that was 15 percentage points higher and a persistence rate that was 11 percentage points higher than their counterparts in the control group. Additional analyses using data collected through the implementation study found that Black students received a relatively high dosage of the intervention, compared to other students in the treatment group, even though the intervention design was blind to race, and Georgia State does not use race as a factor in its early alert models. This may at least partially explain why they benefitted from the intervention.

Implementation study findings

With the exception of Georgia State, the intervention period ran for only three years at the participating institutions. The pandemic brought further disruption to the final two and a half years of the study period, which, among many other challenges, made it more difficult for the institutions to deliver advising services to students. In addition, these sites struggled to identify and implement early data systems to inform proactive and early advisement, and experienced high advisor and staff turnover. The shorter intervention period and these implementation challenges may have been responsible for the differential findings. By contrast, Georgia State, in addition to having six years of implementation, benefited from having designed the MAAPS intervention as an extension and enhancement of its pre-existing advising approach, and therefore already had in place the institutional infrastructure, culture, and data tools and systems that eased implementation.

Despite the lack of observable impacts on the study’s outcomes, most institutions reported that the MAAPS project will have a lasting positive impact on their institutional programming, policies, and practices, especially around supporting historically underserved students through academic and financial advising. Over the last few years, MAAPS institutions have simplified certain curricula and degree plans; filled gaps in the type of information being collected on student success indicators; developed and deployed new tools to collect and share information between advisors; reassessed and modified their approach to advising, including tailoring advising services to close equity gaps; and brought together different parts of the institution to better support students.

Avenues for future research

We have also identified a number of areas for future research. These include evaluating the MAAPS protocol using the primary model at institutions other than Georgia State. Georgia State was one of only two institutions to have MAAPS advisors serve as students’ primary and sole advisor (as opposed to serving as a supplemental advisor), which contributed to its high fidelity of implementation. As the only institution where we observed positive impacts, it raises the question of whether similar results could be replicated at other institutions that employ a primary model.

As one of the first studies examining the causal impact of technology-enhanced proactive advisement on student achievement, persistence, and completion, the MAAPS project—and findings from both the impact and implementation studies—will make a significant contribution to the higher education community’s understanding of such interventions and the conditions under which they are most effective.