Many higher education institutions are implementing advising interventions, if not complete redesigns, in an effort to advise their students in a more timely and targeted manner. While the approaches can take various forms, they have increasingly relied on technology to alleviate the burden of large caseloads by helping advisors easily and quickly identify which students need what type of support, and when. In an ambitious experiment, the 11 institutions that form the University Innovation Alliance (UIA) are testing such an intervention on their campuses, as part of a randomized-controlled trial funded by a U.S. Department of Education First in the World validation grant. Monitoring Advising Analytics to Promote Success (MAAPS) was designed to provide intensive, proactive technology-enhanced advisement to a cohort of over 5,000 low-income and first-generation students across the participating campuses. MAAPS advisement aims to help students navigate key academic choices and stay on track through regular and individualized degree planning activities; real-time and early alerts prompted in part through an analytics-based system; and timely, targeted advising interventions informed by degree-planning activities and early alerts.

The intervention and accompanying evaluation study were launched in the beginning of the Fall 2016 term, with Georgia State University serving as the lead institution, and Ithaka S+R as the independent evaluator of the project. As the evaluator, Ithaka S+R is responsible for examining the impact of the MAAPS intervention on GPA, credit accumulation and success, retention, progress to degree, and ultimately graduation for the more than 10,000 students randomly selected and assigned into the treatment and control groups, as well as studying the implementation of MAAPS across the participating institutions.

Today Ithaka S+R is releasing the second impact findings report, which shares the interim results from the first two years of the project and previews early results from the third academic year of the intervention (a report with findings from the first year of the intervention can be found here). The report, which also includes additional details on the core components of MAAPS advisement and findings from the implementation study, is accompanied by a technical supplement that presents detailed information on the outcomes measured and analytic methods employed, as well as data on attrition rates for each outcome and results from each analysis.

What are some of the big takeaways after two and a half years of the intervention?

  • After one, two, and two and a half academic years, we did not observe significant impacts from MAAPS advisement on any of the four interim outcomes in the aggregate sample across the 11 institutions.
  • We did, however, observe significant positive impacts from MAAPS at one site: Georgia State. More specifically, after two academic years of the intervention, Georgia State students in the treatment group had a credit success rate that was four percentage points higher, a cumulative GPA that was 0.17 points higher, and 2.19 additional cumulative credits earned than the control group. In addition, early findings from the third year of the intervention (through the Fall 2018 term) suggest that the significant positive impacts at Georgia State have continued to persist over time, with additional impacts on credit success rate and cumulative GPA since the completion of the first year of MAAPS. These early impacts may be the result of Georgia State’s established culture and infrastructure for proactive advising and degree mapping, including high-quality early alert tools, student data tools, and support services.
  • Many of the participating institutions faced notable challenges implementing the MAAPS protocol as designed, particularly during the first year of the intervention and around degree-planning advising activities. Decentralized advising systems and advisor turnover were key challenges to successful implementation. Nonetheless, many sites were able to problem solve and implement multiple key features of MAAPS advisement with fidelity.
  • The impact of MAAPS may be incremental whereby students make small gains each year that accumulate and only become significant over time. For example, after two and a half years of the MAAPS intervention, Oregon State University treatment group students had earned an additional 3.73 credits than students in the control group, while treatment group students at Purdue University earned a cumulative GPA that was 0.08 points higher than students in the control group. Most of these impacts were not observed until after the first year of the intervention.
  • MAAPS may have positive impacts on students that are not adequately captured by the interim outcomes currently measured. For instance, focus groups and surveys with subsets of participating students indicate that MAAPS advisement helps them gain skills and knowledge, including institutional know-how, which promote better decisions and choices in general – attributes that may contribute to fewer wasted credits and a more timely graduation, which are not captured by interim outcome measures.

These interim results, generated partway through students’ college careers, should be interpreted with caution—the positive results observed to date could fade, or new positive results could be observed later in students’ academic careers. Continued data collection on students’ outcomes later in their academic trajectories is necessary to provide evidence regarding the intent-to-treat impact of MAAPS advisement on students’ academic progress and degree completion.