The COVID-19 pandemic highlighted and in many cases widened socioeconomic and other disparities in higher education, making evidence-based approaches that support underserved students more important than ever before. One project that aimed to promote equity and student success across a set of large, public universities is Monitoring Advising Analytics to Promote Success (MAAPS).

In 2016, the University Innovation Alliance (UIA) and its institutional members began testing the effectiveness of MAAPS, an intervention consisting of intensive, proactive, technology-enhanced advisement meant to increase achievement, persistence, and completion of Pell-eligible and first-generation students. It was tested as part of a randomized-controlled trial funded by a US Department of Education First in the World Grant. While the grant was officially closed out in fall of 2020, Ithaka S+R continues to serve as the independent evaluator thanks to support from Arnold Ventures. As evaluator, Ithaka S+R has been responsible for conducting impact and implementation studies since the project’s inception and publishing reports on the findings (see previous reports here and here).

Today we are publishing a report on the impact of the MAAPS intervention on students’ outcomes after four academic years. New to the study, we use data from the National Student Clearinghouse (NSC) to track the graduation and persistence of students, whether students stayed at a MAAPS institution or transferred and graduated elsewhere. These two new measures serve as the study’s primary outcomes. In addition, we continue to use institutional data to measure the cumulative GPA, credit accumulation, persistence, and graduation of students at their initial MAAPS institutions, which serve as secondary outcomes. This report also presents findings from the implementation study based on advising interaction data logged by MAAPS advisors, student advising surveys, interviews with project staff, and focus groups of participating students.

There were no impacts on the study’s primary outcomes—bachelor’s degree receipt and persistence—after four academic years for the full sample or at any of the eleven participating institutions. There also were no impacts on the study’s secondary outcomes for the full sample or at ten of the eleven participating institutions. However, there were significant impacts on the study’s secondary outcomes at Georgia State University, the lead institution on the project. Specifically, treatment group students had a GPA that was 0.16 points higher and accumulated nearly six credits more than control group students. Follow-up analyses revealed that the impacts were most pronounced among Black students at Georgia State, who had a graduation rate that was eight percentage points higher, whether using NSC data or student administrative data, and a persistence rate that was at least 12 percentage points higher depending on the data source. In addition, they had a GPA that was 0.22 points higher and accumulated 12 more credits than their counterparts in the control group.

There are a number of factors, supported by findings from the implementation study, which likely reduced the potential impact of the MAAPS intervention on student outcomes at other institutions. In an effort to account for idiosyncrasies across large, public universities, the grant gave participating institutions flexibility in deciding how to implement the intervention on their own campuses. As a result, institutions implemented the intervention under varying circumstances and with varying degrees of fidelity. For example, some institutions attempted to provide treatment students with MAAPS advising and business-as-usual advising through multiple advisors, but this increased the complexity of implementation and undermined the delivery of the MAAPS intervention.

Institutions faced a number of other implementation challenges, including the inability to identify and implement early alert data systems to inform proactive and early advisement in the first half of the intervention, low student take-up of MAAPS advisement, and advisor and staff turnover. Finally, other than Georgia State, no participating institution offered MAAPS advisement to their students after the third year of the intervention when the First in the World grant funding ended. This diluted the treatment and disrupted relationships and momentum that MAAPS advisors developed with their students in the first three years. By contrast, Georgia State stood out for its implementation, facilitated by a set of institutional conditions that existed prior to the intervention.

Despite the lack of observable impacts on the study’s outcomes after four years, there is evidence that at least a subset of students and institutions benefited from the intervention. Among the approximately ten percent of students who responded to the student advising surveys, treatment group students across all participating institutions reported a positive experience and improved perceptions of institutional know-how, and higher levels of academic support and proactive advising than students in the control group in each of the three years they were surveyed. These findings are well aligned with findings from the student focus groups conducted at each participating institution. Some institutions also shared that the MAAPS project will have a lasting positive impact on their institution’s practices and policies.

The study will conclude with the publication of a report on findings after students’ sixth academic year, which will provide the most comprehensive picture of the impact of the MAAPS intervention on student outcomes.