Acknowledgements

This project is generously funded by a U.S. Department of Education First in the World validation grant.[1] We thank the project principal investigator, Dr. Timothy Renick of Georgia State University, for inviting Ithaka S+R to serve as its independent evaluator and for being an invaluable thought and project partner. We would like to acknowledge the key role of the University Innovation Alliance (UIA), which inspired the project by virtue of its collaborative nature, and the commitment of its member institutions to innovate together by being early testers and adopters of new ideas. We thank the leadership and staff of Arizona State University (ASU), Georgia State University (Georgia State), Iowa State University (Iowa State), Michigan State University (MSU), The Ohio State University (Ohio State), Oregon State University (Oregon State), Purdue University (Purdue), University of California Riverside (UCR), University of Central Florida (UCF), University of Kansas (KU), and University of Texas at Austin (UT Austin). In particular, we thank the MAAPS team members at each institution for committing to the project and overseeing its different aspects, and for working with us to facilitate data collection around student outcomes and program implementation. This includes the principal investigators, advising leads, data leads, data analysts, and MAAPS advisors at each site, as well as the UIA fellows and other institutional staff members who have assisted the MAAPS teams on project implementation and data collection as needed.

We extend a special thanks to all members of the MAAPS data teams for their valued collegiality and support. We also thank C. Lockwood Reynolds of Kent State University for his technical assistance and expertise.

Last but not least, we are grateful to the thousands of students who are participating in the MAAPS project and contributed their data to the evaluation study, and to those who also completed surveys and participated in focus groups with us. We wish them much success in their educational endeavors and beyond.

Summary

Monitoring Advising Analytics to Promote Success (MAAPS) is a multi-institutional project of the University Innovation Alliance (UIA), supported by a U.S. Department of Education First in the World Grant to Georgia State University, a UIA member. The large-scale, randomized-controlled trial was designed to test and validate the effectiveness of technology-enhanced, proactive advisement in increasing retention, progression, and achievement for incoming low-income and first-generation college students. The MAAPS intervention was officially launched during the Fall 2016 term at the 11 institutions that form the UIA, and presently concluded its third year. Over 5,000 students were randomly assigned to the intervention group, and received proactive outreach, degree-planning activities, and targeted interventions from their assigned MAAPS advisors in addition to business-as-usual advisement at their institution. An evaluation study accompanies the intervention and collects administrative data on students’ academic achievement and persistence, as well as qualitative data on the implementation of the project.

This report presents impact findings for the intent-to-treat effect of MAAPS advisement on participating students’ outcomes for the current entire sample of 10,042 students and at each participating institution after one and two academic years into the project, and previews early emerging findings from the third year of implementation. It also presents findings from the implementation study, with a focus on implementation challenges and strengths faced by participating sites. This report is accompanied by a technical supplement that presents detailed information on the methodology and findings of the evaluation study.

Overall, the findings indicate that across the entire sample, the MAAPS advising intervention has no impacts on the four interim student outcomes measures of interest after one, two, or two and a half academic years. Institutional sub-analyses, however, reveal significant positive impacts in the Georgia State sample during this timeframe, as well as positive impacts at two additional institutions in the first part of the third year of implementation. The implementation study indicates that sites faced significant implementation challenges, particularly in advisor turnover and degree-planning advising activities. Nonetheless, many sites were able to adequately problem-solve and implement multiple key features of MAAPS advisement with fidelity. Results from student surveys and focus groups suggest that a subset of students across are gaining valuable experiences and skills through MAAPS advisement.

We conclude the report with five takeaways from these interim findings and their implications for future research.

Introduction & Overview

Monitoring Advising Analytics to Promote Success (MAAPS) is a multi-institutional project of the University Innovation Alliance (UIA), and is supported by a U.S. Department of Education First in the World Grant to Georgia State University, the lead UIA member on this project. MAAPS is a large-scale randomized-controlled trial designed to validate the effectiveness of technology-enhanced proactive advisement in increasing retention, progression, and achievement for low-income and first-generation college students. Addressing documented obstacles to college completion that disproportionately impact at-risk populations, the four-year study tracks cohorts of low-income and first-generation students enrolled at the 11 large public universities that constitute the UIA: Arizona State University, Georgia State University, Iowa State University, Michigan State University, The Ohio State University, Oregon State University, Purdue University, University of California Riverside, University of Central Florida, University of Kansas, and University of Texas at Austin. Ithaka S+R serves as the independent evaluator of the study.

The MAAPS advising intervention and accompanying impact and implementation studies were officially launched at each participating institution at the start of the Fall 2016 term, after a year of planning and preparation. The advising intervention was offered to a randomly selected group of eligible students at each institution (treatment group), while control group students received business-as-usual advisement at their institution. The advising intervention concluded at the end of the Spring 2019 term at most participating institutions, after three years of implementation. Presently, two institutions plan to continue providing MAAPS advisement to their original cohort of treatment group students during the 2019-2020 academic year, after the conclusion of the original research grant.

This report presents 1) a high-level description of the intervention activities and study methodology; 2) a summary of interim findings regarding the impact of MAAPS advisement on participating students’ outcomes in the aggregate sample and at each participating institution after one and two academic years into the advising intervention; 3) a preview of early emerging findings from the third year of implementation; and 4) findings from the implementation study, including implementation challenges and strengths that help contextualize the impact study results. The interim student outcomes presented in this report include two academic achievement outcomes, credit success rate and GPA, and two persistence outcomes, credit accumulation and continuous enrollment. A future report with findings from three academic years of implementation will also include outcomes pertaining to students’ success in completing their degree requirements and their progress toward a bachelor’s degree. We conclude the report with key takeaways regarding the interim findings, and recommendations for future research.

The MAAPS Intervention

Background

The MAAPS intervention was envisioned and developed as a UIA initiative, with Georgia State as the lead institution on the project. It was grounded in empirical research findings on the positive impacts of proactive technology-enhanced advisement and degree planning on historically underserved students,[2] and in Georgia State’s advising redesign successes. The result was an intervention that aimed to offer intensive wrap-around supports to eligible low-income and first-generation students with a focus on degree-planning activities and proactive data-informed interventions to keep students from going off track. The participating institutions committed to testing this intervention at their respective campuses as part of the collective efforts to innovate around improving the success of less-advantaged students and inform the field of scalable solutions.

To that end, each participating institution assembled a team consisting of a project lead, an advising lead, two or three MAAPS advisors, and a data team. The project lead, a senior administrator working in the area of student success, oversaw the project and its staff. The advising lead, a senior or mid-level administrator with robust experience in student advisement, was responsible for the implementation of the MAAPS advisement intervention at their site, including hiring, training, and managing the MAAPS advisors. The MAAPS advisors had varied levels of experience at the start of the intervention, and were hired by each individual MAAPS team based on its needs and local context. Many institutions hired their initial MAAPS advisors internally, selecting them based on best fit for the project and to minimize training needs. In general, sites selected advisors based on their demonstrated interest and experience advising low-income and first-generation students and their willingness to participate in a research study. Finally, the data team members led their institutions’ data collection and submission efforts, and in most cases also provided data and technical support to the advisors (e.g. by creating dashboards with select data on treatment group students, or prepopulating the degree planning tool with administrative student data).

The launch of the intervention was preceded by a planning year, during which the Georgia State team developed a training manual outlining the goals and activities of the project and guidelines for providing MAAPS advisement to students. A kick-off meeting was held in February 2016 where initial MAAPS leadership convened as a group to discuss and refine the project and its various components. For example, the group redesigned the intervention to allow for varied MAAPS models based on the different institutions’ advisement cultures and models. As a result, most institutions adopted the “supplemental” MAAPS model instead of the original “primary” model, whereby MAAPS advisors would supplement the work of the existing primary advisors rather than replace them. During the kick-off meeting, data team leaders helped establish common project-wide data definitions. Follow-up webinars were subsequently held to finalize data collection plans and train newly hired MAAPS advisors on using the Degree Planner Excel tool developed for the project. Advising teams were otherwise given the discretion of training their advisors and developing specific proactive student intervention plans as they saw fit. After the start of the intervention in fall 2016, yearly meetings of MAAPS leadership were held to share practices, lessons learned, solutions, and early findings.

The MAAPS Cohort

Shortly before the start of the Fall 2016 term, a cohort of incoming first-time low-income and/or first-generation students was selected to participate in the MAAPS study at each institution. These students were randomly selected by the independent evaluator from a pool of eligible students, and then randomly assigned to the treatment or control group.[3] Cohort sizes vary across the institutions, ranging between 391 and 1,162 students, with a total of 10,489 students across all sites, 5,239 of which were assigned to receive MAAPS advisement (treatment group). The cohort sizes were designed in part to ensure that MAAPS advisors would have a caseload of 150 or fewer treatment group students. On the third day of the Fall 2016 term, all study students were informed of their participation in the study via email through a letter of information outlining the study and its purpose, their selection and participation in the study, and guidelines for opting out or requesting more information.

All treatment group students were set up to receive business-as-usual advisement as typically provided by their institution, as well as MAAPS advisement delivered through their assigned and dedicated MAAPS advisor. Control group students were set up to receive business-as-usual advisement as provided by their institution, as they otherwise would receive in the absence of the study, and did not have access to MAAPS advisement.[4] The MAAPS advising intervention was launched starting on the fourth day of the Fall 2016 term, when MAAPS advising staff began reaching out to treatment group students about the advising services or to schedule appointments.

The Intervention Activities

As originally designed, the MAAPS intervention provides intensive, proactive advisement to help students navigate key academic choices and stay on track through the following key activities: (1) regular and individualized degree planning activities; (2) real-time and early alerts prompted in part through an analytics-based system; and (3) timely, targeted advising interventions informed by degree planning activities and early alerts.

Degree Planning

Advising based on a detailed suggested schedule of classes is an essential element of the MAAPS project. Advisors were trained to a create and update degree plans for each treatment group student that lay out the most efficient and appropriate schedule of coursework to be completed for timely graduation with a bachelor’s degree. A “degree planner” Excel tool was developed for this project, and sites were encouraged to develop four-year templates for each undergraduate major at their institution that advisors could draw on before customizing a version for each student. As part of MAAPS advisement, advisors personalize students’ individual degree planners based on their specific chosen course of study and related institutional or departmental considerations, as well as on their individual strengths and preferences, academic requirements, and prior or ongoing credit accrual. If implemented with high levels of fidelity, advisors review the degree planners at least once per term and discuss them with their students at least once per academic year and whenever a progression issue arises.

Real-Time and Early Alerts

MAAPS advisement requires advisors to review notifications received based on in-class or within-term performance in time to intervene meaningfully. Early alerts may be automated via IPAS or an LMS-integrated analytics solution, or the early alert may be manual (via instructor emails or advisor dashboard login) or some combination of the two. Information generated through these systems would ideally include whether a student’s final grade in a course is not predictive of future success in their major or of on-time graduation, a student’s schedule adjustment undermines timely academic progression or financial aid eligibility (e.g. courses dropped), and a student has changed into a different major. Before registration changes are no longer possible, advisors also review students’ registration information for each upcoming term to ensure that students have registered for the appropriate course sequences and have a course load that will keep them on track. At this time, advisors also review databases that include early alerts for students who are at risk of not progressing on time.

Targeted Advising Interventions

Targeted advising interventions are evidence-based steps MAAPS advisors take to respond to information about a student’s progress generated through early alert systems or processes. MAAPS advisement focuses on early and targeted interventions that respond to specific information about student’s progress, but does not prescribe the type of intervention used or its format. Whenever an alert or source of data suggests that a student is off path or may be at risk of going off path, advisors select the appropriate intervention(s) for their institution and student and contact them in a timely manner.

Interim Analyses & Impact Findings

This section presents the key interim findings from our impact analyses after one, two, and two-and-a-half years of the MAAPS intervention in the aggregate sample of 10,000+ students remaining in the study at each time point, as well as at each participating institution (institutional subsamples). It is important to note that these interim findings are inconclusive, as the intervention was still ongoing at the time of data collection and students were still only partway through their intended trajectories toward the completion of a bachelor’s degree. As such, both the impacts of the intervention and their directionality must be interpreted with caution. Findings from the implementation study, described later in this report, shed light on the potential reasons behind the interim findings presented below and the specific patterns of these impacts. The conclusion section of the report outlines five main takeaways based on the impact and implementation findings to date.

Outcomes & Analytic Methods

The accompanying technical supplement presents detailed information on the outcomes measured and analytic methods employed, as well as data on attrition rates for each outcome and results from each analysis. In summary, the outcomes of interest for this report include (a) the proportion of credits attempted at the institution that the student successfully completed (credit success rate); the student’s cumulative GPA at their institution (cumulative GPA); (c) the total number of accumulated credits that have been accepted by the institution (credit accumulation); and (d) whether the student was continuously enrolled at the institution or not (continuous enrollment). We employed linear regression analyses to assess the intent-to-treat effect of the MAAPS intervention after one, two, and two-and-a-half academic years on the specified outcomes in the aggregate sample and at each of the 11 participating institutions. The final models include baseline demographic covariates, as well as institutional fixed effects for aggregate sample analyses.[5] Where relevant, we conducted additional exploratory analyses to further examine or explicate certain results.

Aggregate Sample Impacts

Assignment to the MAAPS advising intervention has no significant or near-significant impacts, on average, on the four interim measures of student outcomes after one, two, or two and a half academic years.[6] Similarly, no significant impacts were observed in the aggregate sample for any student subgroups of interest (i.e. Pell-eligible students, first-generation students, and students from underrepresented ethnic or racial minority groups).

Institutional Subsample Impacts

Significant impacts on the study’s interim outcome measures were observed mainly at Georgia State, the lead institution on the project. Significant and near-significant impacts were also observed in early analyses of year three data at two additional institutions, and provide additional information for interpreting the study findings to date. No significant impacts were observed on the four interim outcome measures at the remaining eight participating institutions over the course of the first two and a half years of the intervention.

Impacts at Georgia State

Assignment to the MAAPS advising intervention resulted in significant positive impacts on three interim outcome measures of interest in the sample of 964 students enrolled at Georgia State. More specifically, after the first academic year of the intervention (Fall 2016 through Spring 2017), treatment group students had a credit success rate that was three percentage points higher (p=0.013), and a cumulative GPA that was 0.17 points higher than control group students (p=0.001).[7] Additional analyses revealed that impacts on students’ cumulative GPA were concentrated among students in the lower half of the GPA distribution specifically (i.e. lower-achieving students). There were no significant impacts on students’ continuous enrollment or cumulative credits earned, although treatment group students earned an additional 1.20 institutional credits compared with the control group (p=0.013).[8]

By the end of the second academic year of the intervention these impacts were sustained in the Georgia State subsample, whereby treatment group students had a credit success rate that was four percentage points higher (p=0.002), a cumulative GPA that was 0.17 points higher (p<0.001), and 2.19 additional cumulative credits earned compared with the control group (p=0.06).[9] Preliminary findings from the third year of the intervention (through the Fall 2018 term) suggest that the significant positive impacts at Georgia State have continued to persist over time, with additional impacts on credit success rate and cumulative GPA after the first year of MAAPS.

Impacts at Other Institutions

Early analyses from the third year of the intervention reveal significant and near-significant impacts at two additional participating institutions that are worth noting. After two and a half years of the MAAPS intervention, compared with the control group, Oregon State University treatment group students had earned an additional 3.73 credits (p=0.10) while treatment group students at Purdue University had earned a slightly higher cumulative GPA (by 0.08 points; p=0.09). [10] Follow-up analyses show that most of the additional cumulative credits earned (2.97 credits) and half of the GPA gains at the respective institutions were achieved after the conclusion of the first academic year of MAAPS. These early findings support our hypotheses regarding the incremental nature of MAAPS’ impacts on student outcomes, as well as potential stronger impacts after the first year of the program when implementation challenges were especially prevalent at many participating institutions.[11]

Implementation Study Findings

Data for the implementation study were collected through yearly phone interviews with advising lead staff, yearly student advising surveys, an implementation form completed by each advising team, and site visits to the participating institutions that included interviews with MAAPS staff and focus groups with participating students. The methodology is described in more detail in Appendix A of the accompanying technical supplement. Additionally, all MAAPS advisors logged their advisement interactions in a common secure database, documenting the reason, format, and type of advisement provided through each interaction with a treatment group student. These data were used to conduct targeted exploratory analyses, some of which are discussed in the conclusion section of this report.

Fidelity of Implementation

The 14 criteria for implementing MAAPS advisement with high levels of fidelity are listed below.

  • Advisor caseload and training (three criteria): MAAPS advisors maintained individual caseloads at or below 160 students and did not carry caseloads of non-MAAPS students. Advising teams developed standard MAAPS advising training protocols at their institutions.
  • Frequency of student contacts (two criteria): MAAPS advisors communicated with more than 90 percent of their students at least once per academic term (e.g. to schedule an appointment or send registration information), and conducted an advising intervention with at least 80 percent of their students at least once during the first two years (e.g. recommended a particular course of action or academic plan for the student based on their records).
  • Calendar of proactive advisement (one criterion): Advising teams developed and followed a calendar of proactive advising interventions each academic term that prompts proactive interventions from MAAPS advisors that are aligned with and informed by pertinent institutional procedures and deadlines.
  • Monitoring student progress metrics (three criteria): MAAPS advisors monitored at least six different types of student progress metrics, which must include course registration, course grades, and course withdrawals. At least six different types of student progress metrics, including poor course grades, course registration issues, and course withdrawals, triggered an intervention on the part of MAAPS advisors. Additionally, more than 75 percent of the student progress metrics tracked were available to advisors before the end of the given term (i.e. acted as “early or within-term alerts”).
  • Data-informed advising interventions (one criterion): MAAPS advisors responded to at least 90% of actionable student progress metrics with targeted interventions with students.
  • Degree planning activities (four criteria): MAAPS advisors continuously updated students’ personalized academic maps/degree planners throughout the academic year, shared with students regularly, and always drew on them to inform their advising. More than 90 percent of students reviewed or received a copy of their updated degree plan prior to registration each academic year.

Fidelity of implementation was assessed at the conclusion of the second academic year of the intervention (Spring 2018 term). Three participating institutions, including Georgia State University, met 12-13 criteria, three institutions met 9-10 criteria, and five institutions met 7-8 criteria. Results from the implementation study indicate that many sites experienced challenges in the first year of the intervention, which persisted for some sites through the second year and beyond (e.g. low student take-up). In other cases, new challenges emerged in the second year and beyond (e.g. advisor turnover). These implementation challenges and resultant lower-than-optimal levels of fidelity to the MAAPS protocol may be partially responsible for the absence of interim impacts at most institutions by the end of 2018.

Despite these challenges however, six institutions implemented MAAPS in accordance with 9-13 of the 14 key fidelity criteria, and student survey results indicate that subsets of treatment group students are experiencing more personalized and supportive advisement at their institutions than their control group counterparts. The next sections describe in more detail our findings pertaining to these implementation challenges and strengths. We further discuss the implications of these findings in the conclusion section of this report.

MAAPS Implementation Challenges & Successes

Advising Take-Up

Although many sites successfully increased student take-up and engagement in MAAPS advisement after the first year, they continued to experience lower in-person interactions with students than anticipated. For instance, advising logs kept by MAAPS advisors at each site indicate that at four institutions, a full third to half of the treatment group students had not yet interacted with their MAAPS advisor in person by the end of the second academic year. While advisors have communicated with most of these students over email, phone, or text, low in-person interactions significantly reduced their ability to engage in individualized degree planning activities with their students with the regularity and intensity envisioned for MAAPS.

Sites that experienced higher-than-average student take-up, or notable increases in take-up, engaged in a set of problem-solving strategies, including placing registration holds during the first term of the study or the first term of each year, to ensure students met with their MAAPS advisors on a regular basis. This strategy required high levels of buy-in from the advising community, who had to approve and collaborate on this. Other sites engaged in creative outreach strategies, including embedding the MAAPS advisors in broader advising events, holding advisor office hours at different locations more accessible to students, and using text messaging to engage students and easily schedule appointments.

Advisor Turnover

Advisor turnover posed a notable implementation challenge for MAAPS, especially in the second year, with six sites facing advisor loss and/or turnover by the end of the Spring 2018 term. As a result, remaining advisors experienced a temporary or permanent increase in their caseload as well as the added responsibility of assisting in onboarding new advisors. The reasons for high advisor turnover appear to be multifold; it is due in part to the short-term nature of the project and advisors’ desire to secure longer-term positions. Potential dissatisfaction with the position may also play a role in advisor turnover, as MAAPS advising in the supplemental model involves more “behind the scenes” work reviewing student records and entering data and less time intervening with students through face-to-face interactions than business-as-usual advising.[12] In addition to straining advising teams and interrupting students’ personal relationships with advisors, such turnover in some instances moved advising teams farther away from the key elements of MAAPS advisement. This is because in the absence of a centralized formal training process for the project, training for new advisors at times focused more on supporting the existing practices of the MAAPS team at their institution than on the core features and requirements of MAAPS advisement. Additionally, new advisors bring fresh ideas and practices that are not focused on the core features of MAAPS but may be appealing to MAAPS teams looking to increase student or campus buy-in more generally.

Sites that experienced low or no advisor turnover were able to promote their MAAPS advisors, separately from the project grant, and/or created longer-term pathways for the them at the institution to draw on their increased expertise after the conclusion of the study. In many cases, MAAPS advisors were fully integrated in the broader advising community and elevated as experts on advising low-income and first-generation students, with opportunities to provide feedback to senior administrators and relevant institutional committees.

Advising Infrastructure

Most participating sites adopted the supplemental model of MAAPS advisement, especially after the first year, mainly due to their existing decentralized advisement models. As such, the vast majority of MAAPS students had primary advisors in their respective departments, often faculty experts in students’ specific majors, with MAAPS advisors providing an additional layer of advisement. Because of this, degree planning work often fell to students’ departmental/primary advisors, either due to pushback from the broader advising community or inadvertently as numerous MAAPS advisors did not feel as equipped in that respect. This is especially true when MAAPS advisors had not been previously employed at the institution, and for those assigned to departments with complex major and sub-major options or specialized tracks that departments wanted to oversee closely (e.g. pre-pharmacy). Student focus group interview results suggest that in the supplemental advising model, students also prefer to rely on their departmental advisors for such advisement, especially as they progress more in their specific majors. On the other hand, they prefer to rely on MAAPS advisors for more general academic issues as well as all non-academic ones.

At some sites, MAAPS advisors made efforts to ensure they engaged their students in degree planning by doing it informally instead, with the intention of preparing students to get the most out of their short conversations with their primary advisors who might not engage them in such an exercise otherwise, and who would then review and approve any such plans. In a few cases, MAAPS advisors developed students’ degree plans for their own purposes, to inform their advisement and make appropriate requests to students’ primary advisors (e.g. requests to address a course sequencing issue they identified, or revisit the student’s plan due to a change of major).

Advising Data and Tools

At a few institutions, early challenges in identifying and accessing adequate early alert data to inform proactive advisement by MAAPS advisors also persisted beyond the first year. At one site, the advising team described the process as “hunting and gathering” early alert data from multiple systems and offices at their institution in order to provide advisors with consistent access to actionable student data for the project. At many sites, existing early alert systems required advisors to sift through students’ information to identify actionable alerts, rather than push information to them. As a result, a number of sites successfully relied on their MAAPS data teams to manually generate regular data reports or set up data dashboards designed specifically for the project. In some cases, however, some data, such as student course withdrawals, were simply not available to advisors in a timely manner and thus prevented them from acting proactively based on that important information.

Lastly, the format of the degree-planning tool used for the project also decreased the fidelity with which sites provided proactive degree planning to the treatment group students. Many MAAPS team members reported that the tool was not user friendly for advisors and students, and was incompatible with existing institutional systems that students are accustomed to and that advisors prefer to use (e.g. institutional degree audit systems). Although many advisors saw significant value in the degree planners as a concept, and some reported that they are superior advising tools to existing systems when used correctly, their complexity and inconvenience resulted in low take-up by advisors. In fact, advising logs indicate that 38 percent of treatment group students did not receive any interventions using the MAAPS degree planner specifically, whether in person or otherwise (note: these students may have still received degree planning interventions that did not involve the degree planner tool).

To mitigate this, one site’s data team developed a process to pre-populate key parts of students’ degree planners with administrative student record data so that MAAPS advisors could spend less time populating them and more time reviewing them and gleaning actionable next steps from them. At another site, advisors translated key information from the degree planners into a simplified form that was easy to share and discuss with students. Other sites made efforts to embed degree planning conversations in their advising more broadly, but the absence of an efficient dynamic tool make this key component of MAAPS advisement especially difficult to implement.

Students’ Self-Reported Experiences

Findings from a student survey administered during the Spring 2018 term indicate that by year two, at least a subset of treatment group students across the sample is experiencing higher levels of proactive and personalized advising and gaining valuable information or skills. A total of 942 students across the 11 institutions participated in the student advising survey (9.4% response rate). Regression analyses with baseline covariates indicate that survey participants in the treatment group experience significantly higher levels of proactive advisement, of perceived personalized support by their advisors, and of overall advising satisfaction. Participating students also report higher levels of institutional know-how, which is a key intended outcome of MAAPS advisement.[13] While this subset of students is not representative of the full MAAPS sample, it suggests that even when not implemented with high levels of fidelity across the board, MAAPS may provide some students with an additional layer of advisement or a qualitatively different advising experience than business-as-usual advisement. These results are aligned with findings from focus groups MAAPS treatment group students, who feel their advisors provide a college success safety net and teach them skills for navigating the process on their own. These experiences have potential benefits for students’ academic outcomes in the long term, as well as short-term benefits on outcomes of value beyond what is captured by administrative data, including advising satisfaction and institutional know-how.

The interim impact findings presented in this report and accompanying implementation study findings shed important information on both the challenges and opportunities of advising redesign. They highlight the key role of institutional contexts and cultures, not only in how they shape implementation but also how they should be carefully considered in interpretations of study impact findings and outcomes measured. As the project progresses, the impact and implementation studies promise to continue to highlight best practices for redesigning advising at scale while leveraging data and promising technologies, and for studying the impacts of such redesign in ways that maximize the knowledge produced for the field more broadly.

Conclusion

The interim impact findings regarding the intent-to-treat effect of MAAPS advisement on student’s progress and achievement after approximately two and a half years of implementation, and the rich accompanying implementation data, lead us the draw the following conclusions:

  • Persistent implementation challenges during the first two years of the intervention may have reduced the potential impact of MAAPS at many institutions, as well as in the aggregate sample. Future exploratory analyses with longer-term student and advising data may shed light on whether and how certain implementation features relate to student outcomes.
  • MAAPS advisement can have an impact on students’ outcomes early on in their trajectories, with the promise of sustaining if not increasing these impacts over time, as demonstrated by findings from the Georgia State subsample. These unique early impacts may be a result of Georgia State’s established culture and infrastructure for proactive advisement and degree mapping. Georgia State was not alone in its high levels of implementation fidelity, nor in the frequency with which its advisors engaged in key MAAPS advising practices.[14] It did stand out, however, in the seeming quality of the early alert tools, student data tools, and student support resources available to its MAAPS advisors, which they appeared to use more frequently, rigorously, and consistently than business-as-usual advisors at their institution. The quality of these inputs may have a considerable impact on the subsequent quality and efficacy of advisors’ interventions, even when they are used with the same frequency by MAAPS advisors at other sites.
  • The impact of MAAPS may be incremental, as suggested by the early year three impacts observed at Oregon State[15] and Purdue, whereby students make small gains each year through MAAPS advisement that accumulate and become significant over time. These incremental gains started accruing mostly after the first year of the intervention, suggesting that delayed impacts could be observed at additional institutions. This could be due to a number of early implementation challenges being addressed by the end of the first year of the program, or due to MAAPS providing an especially important layer of support for students after the intensive first-year supports commonly offered to students are no longer available.
  • Data from interviews with MAAPS advisors and students indicate that MAAPS advising may impact students’ academic outcomes in ways that are not yet detectable or observable two and a half years into the intervention. More specifically, MAAPS advising may have differential impacts on different groups of students that are obscured by average-level analyses, but may translate to higher levels of progress and timely graduation for these different students by the end of the study. For example, participating institutions and departments within an institution differ in their policies regarding transferring-in students’ prior credits earned (e.g. AP, dual-enrollment, and summer transfer credits).[16] As a result, two and a half years into the intervention, it is possible that MAAPS advisement is helping treatment group students who transferred a large number of credits for their particular major accumulate fewer credits than average (e.g. by focusing them on successful course completion vs. unneeded credit accumulation) while helping treatment group students who transferred no or very few credits accumulate more credits (e.g. by helping them register for and earn 15 credits per term, or supplement with summer coursework in order to graduate on time). In this scenario, interim increases or decreases in credit accumulation could both lead to improvements in students’ longer term outcomes. MAAPS advisement may also influence some interim outcomes for only some groups of students based on their different academic needs. For example, MAAPS advisement may be helping a small group of students remain enrolled (e.g. by identifying and remedying a financial or registration hold), another small group of students improve their GPA (e.g. by referring them to adequate tutoring services early on), and another small group of students improve their credit success rate (e.g. by enrolling in an appropriate credit load at the right time). In either scenario, MAAPS could be helping students progress toward their degrees in a timely manner even though this impact is not currently captured by interim outcomes.
  • Lastly, it is possible that MAAPS advisement does not impact the interim measures assessed in this study, which are imperfect proxies for students’ progress toward their degrees and degree completion at this stage, but that it impacts other outcomes (e.g. general well-being and college satisfaction or engagement) that could be eventually associated with improved student progress and attainment down the road. For instance, results from the survey of MAAPS students indicates that treatment group students report feeling more supported by their advisors and more satisfied with their advising experience altogether.

The conclusions presented above highlight the inconclusive nature of the present interim findings as students are only partway through their intended trajectories toward the completion of a bachelor’s degree. As such, both the impacts of the intervention and their directionality must be interpreted with caution. Whether the positive impacts observed to date persist over time or translate to positive impacts on students’ progress toward degree completion, and if new impacts emerge in future analyses, remains to be seen. Longer-term research that assesses students’ four- and six-year postsecondary outcomes at large is necessary to provide conclusive evidence regarding the intent-to-treat impact of MAAPS advisement, as implemented through this project, on low-income and first-generation students’ academic trajectories.

Endnotes

  1. Grant Number: P116X150015 ↑
  2. Bettinger, Eric P., and Rachel B. Baker. “The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Advising.” Educational Evaluation and Policy Analysis 36, no. 1 (March 2014): 3–19. doi:10.3102/0162373713500523. ↑
  3. The accompanying technical supplement presents detailed information on the random selection and assignment procedures employed. ↑
  4. Most commonly, business-as-usual advisement at the participating institutions involves a larger advisor-to-student ratio, fewer communications from advisors, shorter advisor-student meetings, and lower levels of proactive outreach to students based on in-term and end-of-term student information. Through business-as-usual, students are also less likely to work on personalized and dynamic four-year degree plans with their advisors. ↑
  5. The baseline demographic covariates include high school achievement scores, low-income status at baseline, and the number of college-level credit hours transferred into the institution before the start of the Fall 2016 term. ↑
  6. Our final analytic samples included 10,069 students for the year 1 analyses, and 10,042 students for subsequent analyses. ↑
  7. After correcting for multiple outcomes in the same domain using the Benjamini-Hochberg correction, these impacts remain statistically significant at the p=0.013 and p=.0.02 levels, respectively. ↑
  8. Institutional credits here refer to credits earned at Georgia State specifically, during the Fall 2016 and Spring 2017 terms. ↑
  9. After applying the Hochberg-Benjamini correction, the impacts on credit success rate and cumulative GPA remain statistically significant at the p=0.002 and p<.0001 levels respectively. Impacts on cumulative credits earned are near-significant (p=0.122). ↑
  10. These impacts are no longer significant after applying the Hochberg-Benjamini correction, with respective p-values of 0.20 and 0.18. ↑
  11. See pp. 24-28 of the year 1 report for more information on early implementation findings and our hypotheses regarding incremental impacts: https://sr.ithaka.org/publications/monitoring-advising-analytics-to-promote-success/ ↑
  12. In the supplemental model, MAAPS advisors supplement the work of students’ existing primary advisors rather than replace them. ↑
  13. See Appendix A in the accompanying technical supplement for more details about the survey, sample, and analyses. ↑
  14. For example, analyses of the study advising logs indicate that MAAPS advisors at Georgia State did not report intervening with students more often, engaging students in more degree-planning activities, or intervening with students more often based on early alerts than advisors at the other two institutions with high implementation fidelity. ↑
  15. About half of the difference in cumulative credits earned are non-institutional credits, meaning students earned and transferred them from another institution. During the first year of the intervention, MAAPS advisors at Oregon State encouraged students facing financial-related issues to take advantage of Oregon State’s Degree Partnership Program (DPP), which allows students to enroll in courses and spread financial aid across both Oregon State and any of the local community college partner schools, where the average price per credit is much lower. The program makes it easy for students to transfer credits from the community college to Oregon State, and articulation tables put forth by Oregon State make it clear what courses at the partner schools satisfy major and minor requirements at Oregon State. ↑
  16. At baseline, participating students at some institutions had transferred-in an average of four to seven credits, while others had transferred-in an average of 16 to 23 prior credits. Similarly, within any given institution, students transferred-in between zero and at least 60 credits (and up to 134 at one institution). ↑