MAAPS Advising Experiment

Acknowledgements

This project is generously funded by a US Department of Education First in the World validation grant,^[1] with additional support from Arnold Ventures. We thank the project principal investigator, Dr. Timothy Renick of Georgia State University, for inviting Ithaka S+R to serve as its independent evaluator and for being an invaluable thought and project partner. We would like to acknowledge the key role of the University Innovation Alliance (UIA), which inspired the project by virtue of its collaborative nature and the commitment of its member institutions to innovate together by being early testers and adopters of new ideas. We thank the leadership and staff of Arizona State University (ASU), Georgia State University (Georgia State), Iowa State University (Iowa State), Michigan State University (MSU), The Ohio State University (Ohio State), Oregon State University (Oregon State), Purdue University (Purdue), University of California Riverside (UCR), University of Central Florida (UCF), University of Kansas (KU), and University of Texas at Austin (UT Austin). In particular, we thank the Monitoring Advising Analytics to Promote Success (MAAPS) team members at each institution for committing to the project, overseeing its different aspects, and for working with us to facilitate data collection around student outcomes and program implementation. This includes the principal investigators, advising leads, data leads, data analysts, and MAAPS advisors at each site, as well as the UIA fellows and other institutional staff members who have assisted the MAAPS teams on project implementation and data collection as needed.

We extend a special thanks to all members of the MAAPS data teams for their valued collegiality and support. We also thank C. Lockwood Reynolds of Kent State University for his technical assistance and expertise during the early years of the project.

Last but not least, we are grateful to the thousands of students who participated in the MAAPS project and contributed their data to the evaluation study and to those who also completed surveys and participated in focus groups with us. We wish them much success in their educational endeavors and beyond.

Summary

MAAPS is a multi-institutional project of the UIA funded by a US Department of Education First in the World Grant to Georgia State with additional support from Arnold Ventures. MAAPS is a large-scale randomized-controlled trial designed to validate the effectiveness of intensive, proactive, technology-enhanced advisement in increasing achievement, persistence, and completion of historically underserved students. The MAAPS advising intervention and accompanying impact and implementation studies began at the eleven UIA institutions at the start of the Fall 2016 term and concluded its fourth year at the end of the 2019-20 academic year. Over 5,000 low-income and/or first-generation students were randomly assigned to the treatment group and received proactive outreach, degree-planning activities, and targeted interventions from their assigned MAAPS advisors in addition to business-as-usual advisement at their institution, while over 5,000 students were assigned business-as-usual advisement only at their institution.

This report presents impact findings for the intent-to-treat effect of MAAPS advisement on participating students’ outcomes for the current analytic sample of 10,037 students and at each participating institution after four academic years. New to the study, two outcomes are based on data from the National Student Clearinghouse (NSC), which provides graduation and persistence information on MAAPS students at their original institutions, as well as those who left their original MAAPS institution and enrolled and graduated elsewhere. These two outcomes measured at four years serve as the study’s primary outcomes and will be superseded by the same outcomes at six years, as specified in the pre-analysis plan.^[2] In addition, we continue to use institutional data to measure the cumulative GPA, credit accumulation, persistence, and graduation of students at their initial MAAPS institutions, which serve as secondary outcomes. This report also presents findings from the implementation study based on advising interaction data logged by MAAPS advisors, student advising surveys, interviews with project staff, and focus groups of participating students.

Assignment to the MAAPS advising intervention had no significant impacts, on average, on either the primary outcomes measuring graduation and persistence using NSC data or the secondary outcomes using student administrative data after four academic years for the full sample. Secondary analyses examining the impact of the MAAPS intervention at each of the eleven participating institutions revealed that assignment to the MAAPS intervention had no significant impacts on the primary outcomes measuring graduation and persistence as measured by NSC data. Early analyses after two and a half years of the intervention revealed significant or near-significant impacts on secondary outcomes at three of the eleven participating institutions: Georgia State, Oregon State, and Purdue.^[3] First in the World grant funding ended after year three, and only Georgia State continued to offer MAAPS advisement interventions in year four. The interim positive effects faded at Oregon State and Purdue by the end of the fourth academic year but were still significant after four years at Georgia State. Follow-up analyses revealed that the impacts of the MAAPS intervention on Black students accounted for the intervention’s overall impacts at Georgia State. In addition to ten of the eleven participating institutions not offering MAAPS advisement in the fourth year of the intervention, the end of the fourth year occurred during the COVID-19 pandemic—a time during which campuses may have struggled to offer and deliver basic advising services and supports to students.

Based on findings from the implementation study, institutions faced a wide array of challenges implementing the MAAPS protocol, which may have been responsible for these differential findings. Despite these challenges, results from the student advising surveys and focus groups suggest that a subset of students gained valuable skills and information as a result of the MAAPS intervention and had a more favorable perception of their advising experience. In addition, some institutions shared that the MAAPS project will have a lasting positive impact on their institution’s practices and policies.

Project Background

MAAPS is a multi-institutional project of the UIA initially funded and supported by a US Department of Education First in the World Grant to Georgia State, the lead UIA member on this project. MAAPS is a large-scale randomized-controlled trial designed to validate the effectiveness of intensive, proactive, technology-enhanced advisement in increasing achievement, persistence, and completion of historically-underserved students. The study includes more than 10,000 low-income and first-generation students enrolled at the eleven large public universities that constitute the UIA: ASU, Georgia State, Iowa State, MSU, Ohio State, Oregon State, Purdue, UCR, UCF, KU, and UT Austin. Ithaka S+R serves as the independent evaluator of the study.

This report presents 1) a brief overview of the MAAPS intervention and its key activities; 2) an update on the current study and analytic samples; 3) findings from our impact analyses on the outcome measures after four academic years in the full sample and institutional subsamples; 4) implementation study findings; and 5) discussion of the results. A final MAAPS report will be published in 2023 that includes analyses of impacts after six academic years.

Overview of the MAAPS Intervention

The MAAPS intervention is grounded in empirical research findings on the positive impacts of intensive, proactive, technology-enhanced advisement and degree planning,^[4] and in the dramatic improvements in student success associated with Georgia State’s advising redesign.^[5] The intervention includes the following activities: (1) regular and individualized degree planning activities; (2) real-time and early alerts prompted in part through an analytics-based system; and (3) timely, targeted advising interventions informed by degree planning activities and early alerts.

The MAAPS advising intervention and accompanying impact and implementation studies began at each participating institution at the start of the Fall 2016 term, after a year of planning and preparation. The advising intervention was offered to a randomly selected group of eligible students at each institution (the “treatment group”) who were assigned a dedicated MAAPS advisor by their institution, most of whom had a caseload of 150 students or fewer. MAAPS advisors provided advisement including the intervention components (“MAAPS advisement”) to the treatment group; treatment group students also continued to receive business-as-usual advisement from their institution. Randomly selected control group students received only business-as-usual advisement at their institution.^[6] The advising intervention concluded at the end of the Spring 2019 term at most participating institutions, after three years of implementation. Georgia State University was the only institution to provide MAAPS advisement to its original cohort of treatment group students through the 2019-2020 academic year.^[7]

The original First in the World grant, scheduled to end following the Spring 2019 term, was extended to allow for the collection of student administrative data through the 2019-20 academic year. In addition, a new grant from Arnold Ventures supports the collection and analysis of data on MAAPS students from NSC through 2022—study participants’ sixth year after initial enrollment. This supplements existing MAAPS study data by providing graduation and persistence information after the 2019-20 and 2021-22 academic years on MAAPS students at their original institutions, as well as those who left their original MAAPS institution and enrolled and graduated elsewhere. This report is the first to include results based on NSC data.

The MAAPS Cohort: Study and Analytic Samples

In the summer of 2016, the eleven participating institutions identified more than 20,000 Pell-eligible and/or first-generation students who met the study’s eligibility criteria. Four weeks before the start of the Fall 2016 term, Ithaka S+R randomly selected a subset of 10,946 eligible students, stratified by institution and based on each institution’s desired sample size, and then randomly assigned them to the treatment or control group, also stratified by institution. Each eligible student had an equal chance of being selected into the study, conditional on their institution, and each selected student had an equal chance of being assigned to either the treatment or control group, conditional on their institution.^[8]

A total of 457 students were not eligible for the study and removed from the sample. These students either turned out to have baseline characteristics that rendered them ineligible to participate in the study or had not matriculated at the participating institution for which they were selected. This resulted in a final study sample of 10,489 students, including 5,239 in the treatment group who were assigned to receive MAAPS advisement. Cohort sizes vary across the institutions, ranging between 391 and 1,162 students. Students who opted out of the study and deceased students are included in the study sample but excluded from analytic samples and are thus considered attriters. There were a total of 452 attriters, resulting in a 4.3 percent overall attrition rate.^[9]

Table 1 provides a breakdown of final study and analytic samples at each site and across the study.

Table 1. Final Study and Analytic Samples by Assigned Group and Institution

	MAAPS-eligible at selection	Randomly selected and assigned	Included in study sample			Included in analytic sample
	Total	Total	C	T	Total	C	T	Total
ASU	3,845	1,037	507	504	1,011	501	494	995
Georgia State	1,998	1,040	492	502	994	476	488	964
ISU	1,520	1,230	584	578	1,162	561	532	1,093
KU	1,173	1,173	565	559	1,124	546	536	1,082
MSU	1,830	930	456	456	912	434	434	868
Ohio State	2,615	1,024	494	499	993	486	448	934
Oregon State	920	920	437	430	867	434	420	854
Purdue	964	964	472	469	941	472	436	908
UC Riverside	3,534	1,112	544	544	1,088	507	488	995
UCF	1,203	1,100	503	503	1,006	489	485	974
UT Austin	416	416	196	195	391	186	184	370
Total	20,018	10,946	5,250	5,239	10,489	5,092	4,945	10,037

Key Baseline and Outcome Measures

The measures used to capture student outcomes after four academic years, at the end of Spring 2020, are derived from two different data sources: student administrative data submitted directly by participating institutions to Ithaka S+R and, for the first time as part of this study, student data collected by NSC and submitted to Ithaka S+R. NSC data supplements existing MAAPS study data by collecting graduation and persistence information on students who left their initial MAAPS institution and continued college elsewhere. We generated two academic achievement variables and two persistence/credit accumulation variables using institutional student administrative data, and one academic achievement variable and one persistence/credit accumulation variable using NSC data, as detailed below. Per the study’s pre-analysis plan, the graduation and persistence measures using NSC data are the primary outcomes, and the remaining four measures using student administrative data are the secondary outcomes. Variables generated using NSC data are indicated next to the variable name.

Outcome Measures: Academic Achievement

Cumulative GPA: Student’s cumulative GPA as determined by their initial MAAPS institution. Scores range from 0 to 4.3. Continuous variable.

Graduation: Whether the student earned a bachelor’s degree by the end of the Spring 2020 term from their initial MAAPS institution, or not. Binary variable.

Graduation (NSC):^[10] Whether the student earned a bachelor’s degree by the end of the Spring 2020 term from their initial MAAPS institution or elsewhere, or not. ^[11] One of two primary outcomes. Binary variable.

Outcome Measures: Persistence/Credit Accumulation

Credit Accumulation: Total number of credits the student has earned to date as determined by their initial MAAPS institution.^[12] Continuous variable.

Persistence: Whether the student earned a bachelor’s degree by the end of the Spring 2020 term from their initial MAAPS institution or was enrolled in the Spring 2020 term according to the institution’s census, or not. Binary variable.

Persistence (NSC): Whether the student earned a bachelor’s degree by the end of the Spring 2020 term from their initial MAAPS institution or elsewhere or was enrolled at least half time^[13] at the end of the Spring 2020 term at their initial MAAPS institution or a degree-granting institution, or not.^[14] One of two primary outcomes. Binary variable.

Due to the nature of the selected study outcomes, which are specific to students’ postsecondary experiences and performance, we rely on one baseline measure of high school achievement for all outcomes.

Baseline Measure: Academic Achievement and Persistence/Credit Accumulation

High School Achievement and College Readiness: Student’s highest composite ACT score recorded by the participating institution where the student enrolled. For students who submitted SAT scores, concordance tables provided by the College Board were used to convert SAT composite scores to ACT composite scores.

Analyses and Impact Findings after Four Years

This section provides an update on the analytic samples, describes the analytic approach, and presents the key findings from our impact analyses in the full sample of 10,000+ students remaining in the study after four academic years, as well as in each institutional subsample.

Analytic Samples: Full Sample

Table 2 presents attrition information for the full sample reported in this section. All attrition rates are considered low according to What Works Clearinghouse (WWC) standards,^[15] yielding a tolerable threat of bias under cautious assumptions regarding the exogenous nature of the attrition.

Table 2. Analytic Sample and Attrition Information for all Outcome Measures – Full Sample

Outcome Measure	Control Group		Treatment Group		Diff. Attrition (pp)	Overall Attrition
	# original sample	# analytic sample	# original sample	# analytic sample
Cumulative GPA	5,250	5,058	5,239	4,922	2.4	4.9%
Graduation	5,250	5,092	5,239	4,945	2.6	4.3%
Graduation (NSC)	5,250	5,092	5,239	4,945	2.6	4.3%
Credit Accumulation	5,250	5,092	5,239	4,945	2.6	4.3%
Persistence	5,250	5,092	5,239	4,945	2.6	4.3%
Persistence (NSC)	5,250	5,092	5,239	4,945	2.6	4.3%

Analytic Approach: Full Sample

We employed linear regression analyses to assess the intent-to-treat effect of the MAAPS intervention after four academic years on the specified outcomes in the full sample. The primary model includes baseline demographic covariates collected at the start of the study in 2016 (high school achievement scores as determined by composite ACT score,^[16] low-income status as determined by expected family contribution (EFC) at baseline, and the number of college-level credit hours transferred into the institution before the start of the Fall 2016 term).^[17] Full sample analyses also include institutional fixed effects to account for idiosyncrasies across the eleven institutions in samples, implementation, and duration of the intervention, and policies regarding enrollment deadlines, credit accrual, and GPA calculations. We addressed missing baseline data in accordance with WWC standards by replacing missing values with a constant of zero and adding a missing data indicator for the given baseline measure in the analysis.^[18] All p-values<0.10 are corrected to adjust for multiple outcomes within a given outcome domain using the Benjamini-Hochberg method, also in accordance with WWC standards.^[19] Where relevant, we conducted additional exploratory analyses to further examine or explicate certain results. Regression tables for the full sample are presented in Appendix A.

The primary model for the full sample is estimated as follows:

Where Y is an outcome for individual i at institution j, TREATMENT indicates whether the student was in the treatment or control group, X is a vector of control variables, and INST represents the institutional fixed effects.

Impacts: Full Sample

Table 3 presents the impact analysis results estimating the intent-to-treat effect of MAAPS advisement on the three academic achievement outcomes, including graduation using NSC data, one of two primary outcomes.

Table 3. Intent-To-Treat Effect of MAAPS Advisement on Academic Achievement Outcomes: Full Sample

Outcome measure	Control Group			Treatment Group			T – C diff.	Std. diff.	p
	n	Mean	(SD)	n	Adj. mean	(SD)
Cumulative GPA	5,058	2.87	(0.86)	4,922	2.88	(0.86)	0.01	0.00	0.732
Graduation	5,092	0.40	(0.49)	4,945	0.41	(0.49)	0.01	0.01	0.223
Graduation (NSC)	5,092	0.42	(0.49)	4,945	0.43	(0.49)	0.01	0.01	0.260

Table 4 presents the impact analysis results estimating the intent-to-treat effect of MAAPS advisement on the three persistence/credit accumulation outcomes, including persistence using NSC data, one of two primary outcomes.

Table 4. Intent-To-Treat Effect of MAAPS Advisement on Persistence/Credit Accumulation Outcomes: Full Sample

Outcome measure	Control Group			Treatment Group			T – C diff.	Std. diff.	p
	n	Mean	(SD)	n	Adj. mean	(SD)
Credit Accumulation	5,092	109.5	(54.0)	4,945	109.5	(54.9)	-0.0	-0.6	0.963
Persistence	5,092	0.69	(0.46)	4,945	0.68	(0.46)	-0.01	-0.01	0.399
Persistence (NSC)	5,092	0.80	(0.40)	4,945	0.79	(0.41)	-0.01	-0.01	0.107

For the full sample, assignment to the MAAPS advising intervention had no significant impacts, on average, on any of the primary or secondary outcome measures after four academic years. Similarly, no significant impacts were observed in the full sample for any student subgroups of interest (i.e. Pell-eligible students, first-generation students, students from underrepresented ethnic or racial minority groups, and Black students specifically).^[20] Regression tables for the full sample are presented in Appendix A.

Institutional Subsamples

Significant impacts on the study’s secondary measures were observed only at Georgia State, the lead institution on the project and the only institution to offer MAAPS advisement to its students after the third year of the intervention. No significant impacts were observed on any of the primary or secondary outcome measures at the remaining ten participating institutions after four academic years, with the end of the fourth year disrupted by the COVID-19 pandemic.

Analytic Samples: Georgia State Subsample

Table 5 presents attrition information for the Georgia State sample reported in this section. All attrition rates are considered low according to WWC standards, yielding a tolerable threat of bias under cautious assumptions regarding the exogenous nature of the attrition.

Table 5. Analytic Sample and Attrition Information for all Outcome Measures: Georgia State Subsample

Outcome Measure	Control Group		Treatment Group		Diff. Attrition (pp)	Overall Attrition
	# original sample	# analytic sample	# original sample	# analytic sample
Cumulative GPA	492	466	502	485	1.9	4.3%
Graduation	492	476	502	488	0.5	3.0%
Graduation (NSC)	492	476	502	488	0.5	3.0%
Credit Accumulation	492	476	502	488	0.5	3.0%
Persistence	492	476	502	488	0.5	3.0%
Persistence (NSC)	492	476	502	488	0.5	3.0%

Analytic Approach: Georgia State Subsample

We employed linear regression analyses to assess the intent-to-treat effect of the MAAPS intervention after four academic years on the specified outcomes at Georgia State. The primary model includes the same baseline demographic covariates. We addressed missing baseline data in accordance with WWC standards by replacing missing values with a constant of zero and adding a missing data indicator for the given baseline measure in the analysis. All p-values<0.10 are corrected to adjust for multiple outcomes within a given outcome domain using the Benjamini-Hochberg method, also in accordance with WWC standards. Where relevant, we conducted additional exploratory analyses to further examine or explicate certain results. Regression tables for the Georgia State subsample are presented in Appendix B.

The primary model for the Georgia State subsample is estimated as follows:

Where Y is an outcome for individual i, TREATMENT indicates whether the student was in the treatment or control group, and X is a vector of control variables.

Impacts: Georgia State Subsample

Table 6 presents the impact analysis results estimating the intent-to-treat effect of MAAPS advisement at Georgia State on the three academic achievement outcomes, including graduation using NSC data, one of two primary outcomes.

Table 6. Intent-To-Treat Effect of MAAPS Advisement on Academic Achievement Measures: Georgia State Subsample

Outcome measure	Control Group			Treatment Group			T – C diff.	Std. diff.	p[21]
	n	Mean	(SD)	n	Adj. mean	(SD)
Cumulative GPA	466	2.94	(0.86)	485	3.10	(0.69)	0.16	0.17	0.001
Graduation	476	0.25	(0.44)	488	0.29	(0.46)	0.04	0.05	0.133
Graduation (NSC)	476	0.29	(0.45)	488	0.33	(0.47)	0.04	0.04	0.134

 Table 7 presents the impact analysis results estimating the intent-to-treat effect of MAAPS advisement at Georgia State on the three persistence/credit accumulation outcomes, including persistence using NSC data, one of two primary outcomes.

Table 7. Intent-To-Treat Effect of MAAPS Advisement on Persistence/Credit Accumulation Outcomes: Georgia State Subsample

Outcome measure	Control Group			Treatment Group			T – C diff.	Std. diff.	p[22]
	n	Mean	(SD)	n	Adj. mean	(SD)
Credit Accumulation	476	85.5	(41.5)	488	91.1	(39.0)	5.6	5.8	0.029
Persistence	476	0.62	(0.49)	488	0.66	(0.47)	0.04	0.04	0.202
Persistence (NSC)	476	0.76	(0.42)	488	0.80	(0.40)	0.04	0.04	0.184

Assignment to the MAAPS advising intervention resulted in significant positive impacts on one secondary academic achievement outcome and one secondary persistence/credit accumulation outcome in the sample of 964 students enrolled at Georgia State. After four academic years, treatment group students had a GPA that was 0.16 points higher than control group students, a difference that has continued to persist even after the end of the first academic year. Follow-up analyses revealed that differences were concentrated among students in the lower half of the GPA distribution. In addition, student subgroups of interest experienced similar results: Pell-eligible students, students from underrepresented minority groups, and first-generation students in the treatment group had a GPA that was 0.17, 0.14, and 0.19 points higher, respectively, than their counterparts in the control group.

In addition, treatment group students accumulated nearly six credits more than control group students, four of which were earned after the first academic year. Follow-up analyses revealed that this is true for Pell-eligible students, who earned six more credits than their counterparts in the control group. It is worth noting that these differences did not translate to statistically significant differences in graduation rates or persistence rates after four academic years, whether using NSC data or student administrative data. However, the differences are near significant, and if they persist, may become significant after six years. Regression tables for the Georgia State subsample are presented in Appendix B.

Table 8 presents the impact analysis results estimating the intent-to-treat effect of MAAPS advisement on Black students at Georgia State on the three academic achievement outcomes.

Table 8. Intent-To-Treat Effect of MAAPS Advisement on Academic Achievement Outcomes: Georgia State Subsample, Black Student Subgroup

Outcome measure	Control Group			Treatment Group			T – C diff.	Std. diff.	p[23]
	n	Mean	(SD)	n	Adj. mean	(SD)
Cumulative GPA	207	2.75	(0.82)	210	2.97	(0.58)	0.22	0.23	0.002
Graduation	208	0.21	(0.41)	210	0.29	(0.46)	0.08	0.09	0.056
Graduation (NSC)	208	0.23	(0.42)	210	0.31	(0.47)	0.08	0.08	0.076

Table 9 presents the impact analysis results estimating the intent-to-treat effect of MAAPS advisement on Black students at Georgia State on the three persistence/credit accumulation outcomes.

Table 9. Intent-To-Treat Effect of MAAPS Advisement on Persistence/Credit Accumulation Outcomes: Georgia State Subsample, Black Student Subgroup

Outcome measure	Control Group			Treatment Group			T – C diff.	Std. diff.	p[24]
	n	Mean	(SD)	n	Adj. mean	(SD)
Credit Accumulation	208	83.0	(40.6)	210	94.9	(35.8)	11.9	12.9	0.002
Persistence	208	0.60	(0.49)	210	0.72	(0.45)	0.12	0.12	0.012
Persistence (NSC)	208	0.69	(0.46)	210	0.82	(0.38)	0.13	0.14	0.001

Black students in the treatment group had a GPA that was 0.22 points higher and accumulated 12 more credits than their counterparts in the control group, including seven credits since the end of the first academic year. In addition, Black students in the treatment group had a graduation rate that was eight percentage points higher than their counterparts in the control group, whether using student administrative data or NSC data, and a persistence rate that was at least 12 percentage points higher depending on the data source. Follow-up analyses revealed that the impacts of the MAAPS intervention on Black students accounted for the intervention’s overall impacts at Georgia State. Regression tables for the Georgia State subsample are presented in Appendix B.

Some commentators have expressed concern that early alert and predictive analytic tools steer students from underrepresented minority groups—and Black students in particular—away from STEM majors and towards majors that are perceived to be easier.^[25] At Georgia State, however, there is no evidence of this. Black students at Georgia State were just as likely to have a STEM major in the Spring 2020 term as other Georgia State students (16 percent), across both the entire study sample and the treatment group.

Implementation Study Findings

The implementation study, conducted by Ithaka S+R, consisted of yearly phone interviews with advising lead staff, yearly student advising surveys, an implementation form completed by each advising team, and site visits to the participating institutions that included interviews with MAAPS staff and focus groups with participating students. In addition, MAAPS advisors logged advisement interactions in a common secure database that documented the reason, format, and type of intervention provided through each interaction with treatment group students. The implementation study concluded for most institutions in 2019, when implementation ended. Georgia State MAAPS advisors continued to log interventions through the end of the Spring 2020 term when the intervention ended at their institution.^[26]

Logged Advisement Interactions

We used advisement interaction data logged by Georgia State MAAPS advisors to further explore the impact of the MAAPS intervention on Black students at that institution.

We hypothesized that less academically prepared students are more likely to trigger alerts and therefore interact with their advisor and experience MAAPS advisement. Because Black students in the Georgia State treatment group were overrepresented among less academically prepared students,^[27] we further hypothesized that Black students received a higher dose of the intervention, and that this explains the greater magnitude of impact on their outcomes.

To test this, we compared the prevalence of the following key advising metrics between various subgroups of Georgia State students in the treatment group:

• In-person contact (at least one) in year 1
• Degree planner review at least once per year during years 1-3
• Total number of interactions in years 1-4
• Total number of interventions in years 1-4
• Total number of advising triggers that were not student-initiated (years 1-4)
• Total number of advising triggers that were student-initiated (years 1-4)

Black treatment group students at Georgia State were more likely to participate in at least one degree planner review per year (56.7 to 46.0 percent; p-value=0.02) and experienced more interactions (23.3 to 20.7; p-value<0.01) and interventions (47.0 to 41.4; p-value<0.01) per student, on average, than treatment group students at GSU who are not Black. There were no differences, however, when disaggregating the same set of advising metrics by students’ academic preparation rather than race. Further, the differences in advising interactions and interventions between Black treatment group students at Georgia State and all other students in the Georgia State treatment group persist after limiting the sample to those less academically prepared only, as well as limiting it to those more academically prepared only.^[28]

In other words, Black students triggered and experienced more advising interactions and interventions irrespective of their academic preparation, at least by the measurement of our analysis. It is worth noting that Georgia State does not use race as a factor in its early alert models; the alerts identify and notify advisors of students veering off path according to historical data. Despite differences in the number of interactions and interventions, analyses of student survey data did not reveal any differences in how Black survey respondents at Georgia State perceived and rated their advising experiences and support compared to other survey respondents at Georgia State, both for the full Georgia State sample and for treatment group students at Georgia State only.

Student Advising Survey Findings

The implementation study also includes a ten-minute student advising survey administered to all MAAPS students (in both the treatment and control groups) in the spring of 2017, 2018, and 2019. The surveys explored how treatment and control group students experience advising at their institution and whether their experiences are associated with their academic progress and achievement.^[29]

The number of students across the eleven participating institutions who completed the annual survey decreased each year, with 1,137 students completing the 2017 survey (11.3 percent response rate), 942 students completing the 2018 survey (9.4 percent response rate), and 788 students completing the 2019 survey (7.9 percent response rate). However, the assessments of their advising experiences by students who did complete the survey were fairly consistent across the three years. In all three surveys, those in the treatment group reported significantly higher levels of institutional know-how and reported experiencing higher levels of academic support and proactive advising than those in the control group, each assessed through a multi-item scale.^[30] Treatment group students also reported higher overall satisfaction with advisement than control group students and were more likely to report having an advisor contact them at least twice to set up an in-person meeting.

Table 10. Average Responses of Select 2019 Student Advising Survey Items by Assigned Group: Full Sample^[31]

Survey Item/Scale	Control Group		Treatment Group		p-value
	n	Mean	n	Mean
Institutional Know-How Scale	316	3.7	447	3.9	<0.01
Advisor Support Scale	287	3.6	398	4.0	<0.01
Proactive Scale	299	3.2	417	3.7	<0.01
Overall Satisfaction with Advisement	300	3.7	418	4.1	<0.01
Multiple Advisor Contact	277	0.65	409	0.89	<0.01

It is important to note that the survey subsamples are not representative of the larger MAAPS sample. For instance, female students and high-performing students were more likely to respond.^[32] To address this, we calculated the average responses by treatment group after correcting for any response bias related to the gender or academic performance of the student. However, the weighting procedure did not alter the results, so we report the unweighted responses. Although the survey subsamples are not representative, the results are aligned with findings from focus groups of both treatment and control group students across the participating institutions. Despite the lack of a significant impact of the MAAPS intervention on key outcome measures, these findings suggest that at least a subset of treatment group students have experienced the key features of MAAPS advisement, including proactive and personalized advisement, and have gained information and skills that they perceive as increasing their ability to navigate the complexities of a large, public university.

Implementation Challenges, Responses, and Successes

As discussed in much greater detail in previous MAAPS reports,^[33] the participating institutions experienced a host of implementation challenges, which likely reduced the potential impact of the MAAPS intervention on student outcomes.

Some institutions attempted to provide treatment students with MAAPS advising and business-as-usual advising through multiple advisors, which increased the complexity of implementation and undermined delivery of the MAAPS intervention. For example, prior to MAAPS, most participating institutions relied on departmental academic advisors to serve as students’ primary advisors in a decentralized advising model. Some of these institutions had treatment group students retain a primary advisor from their department, with MAAPS advisors offering supplemental advisement. At multiple institutions, primary advisors questioned the role and need of MAAPS advisors, and students reported preferring to go to their primary advisor over their MAAPS advisor for academic support and guidance. The institutions that engaged with advisors before the start of the intervention in the summer of 2016 to explain the study and the goal of MAAPS advisors—to support and complement, not hinder or duplicate, the work of primary advisors—were most successful in gaining their approval and trust.

Another issue a few institutions faced was the inability to identify and implement early alert data systems to inform proactive and early advisement in the first half of the intervention. Other institutions had early alert systems that were not effective in facilitating the type of proactive outreach described in the MAAPS protocol. For instance, some systems did not automatically push out information to advisors, so advisors had to sift through the data and identify relevant information themselves. Other institutions had multiple early alert systems for different units and offices which forced advisors to merge and synthesize information across sources. To overcome this series of challenges, institutions worked collaboratively with their MAAPS data team to produce data reports and dashboards that provided advisors with key information on the status and progress of their students.

At many institutions, students responded to advisor outreach at lower-than-expected levels, leading to low in-person interactions between MAAPS advisors and students, which served as another hurdle to providing MAAPS advisement to students. Some institutions were able to overcome this by placing holds on students’ registration accounts until they met with their MAAPS advisor. More than half of sites also faced advisor and staff loss or turnover, which resulted in increased caseloads, making it more difficult for advisors to provide personalized and proactive advisement. At institutions that had to replace advisors and/or staff, new advisors had trouble developing relationships with students while new project leads found it difficult to guide project staff because they were not as familiar with the MAAPS protocol.

Finally, other than Georgia State, no participating institution offered MAAPS advisement to their students after the third year of the intervention, diluting the treatment and disrupting relationships and momentum that MAAPS advisors developed with their students in the first three years. The COVID-19 pandemic brought further disruption to the end of the fourth year of the intervention as institutions shifted from face-to-face interaction to online delivery of courses and services in emergency fashion and as a result may have struggled to offer and provide basic advising services to students.

Georgia State stood out for its implementation, facilitated by a set of institutional conditions that existed prior to the intervention. Georgia State already had a centralized advising system in place, so the primary model was a natural extension. Moreover, they had cultivated a culture of proactive advisement and degree planning after years of doing similar activities with well-documented success. Finally, while Georgia State MAAPS advisors did not interact and intervene with their students more frequently than MAAPS advisors at other institutions, Georgia State set themselves apart for the quality of their early alert tools and student support systems.

Even with these challenges, some institutions shared that the MAAPS project will have a lasting positive impact on their institution’s practices and policies. A few noted that it has led to similar interventions in advising and across other areas on campus. For instance, one institution is training all academic advisors in proactive advising. Another institution is looking to deploy dashboards similar to the ones created for MAAPS so advisors can easily reference and pull up-to-date information on their students. Other institutions explained that the project exposed policies that were adversely impacting students, policies which they are currently working to address. For example, degree planning activities revealed the complexity and difficulty of navigating through some of their degree plans, which were often riddled with unnecessary barriers. Finally, and more generally, the MAAPS project brought together different parts of the institution that historically operated in silos, and as a result, has prompted conversations on how to work together to better support students.

Conclusion

Assignment to the MAAPS advising intervention had no significant impacts, on average, on either the primary outcomes measuring graduation and persistence using NSC data or the secondary outcomes using student administrative data after four academic years for the full sample. Secondary analyses revealed that the MAAPS intervention had no significant impacts on the primary outcomes after four years at the eleven participating institutions. However, significant impacts on the study’s secondary outcomes were observed at Georgia State, where treatment group students had a GPA that was 0.16 points higher and accumulated nearly six more credits than control group students. Follow-up analyses revealed that the impacts were most pronounced among Black students at Georgia State, who had a GPA that was 0.22 points higher and accumulated 12 more credits than their counterparts in the control group. In addition, they had a graduation rate that was eight percentage points higher, whether using NSC data or student administrative data, and a persistence rate that was at least 12 percentage points higher depending on the data source. In some ways, the positive outcomes of the MAAPS interventions at Georgia State likely underrepresent the potential positive impacts of similarly designed and implemented data-based proactive advising at other institutions. At Georgia State, even the control group students were receiving standard advising supports based on data and proactive outreach—just not with the intensity of the MAAPS cohort.

Implementation challenges at most participating institutions may have been responsible for these differential findings. These challenges include providing treatment students with MAAPS advising and business-as-usual advising through multiple advisors, which increased the complexity of implementation and undermined delivery of the MAAPS intervention; the inability to identify and implement early alert data systems to inform proactive and early advisement in the first half of the intervention; low student take-up of MAAPS advisement; advisor and staff turnover; and only three years of implementation. By contrast, Georgia State benefited from an institutional infrastructure, culture, and a set of high-quality data tools and systems that eased implementation.

Despite these implementation challenges, a subset of treatment group students across all participating institutions reported a positive experience and improved perceptions of institutional know-how, and higher levels of academic support and proactive advising than students in the control group in each of the three years they were surveyed.

Finally, there is a lot to learn from the Georgia State experience and outcomes, including the exceptionally positive impact for Black students in terms of credit accumulation, cumulative GPA, persistence, and most importantly graduation, which has the potential to inform institutional efforts around promoting equity and closing racial graduation gaps. Black students triggered and experienced more advising interactions and interventions than other students in the Georgia State treatment group, which may explain why they benefited from the intervention. Additional research is needed to better understand the types of interactions and interventions that were most impactful.

The study will conclude with the publication of a report on findings after students’ sixth academic year, which will provide the most comprehensive picture of the impact of the MAAPS intervention on student outcomes.

Appendix A. Results Tables: Full Sample

For the full sample, we present four regression models for each analysis, with each model presenting a different or additional set of control variables. Model 1 does not include control variables, model 2 includes baseline demographic covariates only (high school achievement scores as determined by composite ACT score, low-income status as determined by expected family contribution (EFC) at baseline, and the number of college-level credit hours transferred into the institution before the start of the Fall 2016 term), model 3 includes institutional fixed effects only, and model 4 includes both baseline demographic covariates and institutional fixed effects.^[34] For follow-up analyses, we present the results of model 4 only.

Table 11. Descriptive Statistics for Academic Achievement Outcomes

	Control Group		Treatment Group		Total Sample
	Mean (SD)	n	Mean (SD)	n	Mean (SD)	n	Range
Cumulative GPA	2.87 (0.86)	5,058	2.87 (0.86)	4,922	2.87 (0.86)	9,980	0 – 4.3
Graduation	0.40 (0.49)	5,092	0.41 (0.49)	4,945	0.40 (0.49)	10,037	0 – 1
Graduation (NSC)	0.42 (0.49)	5,092	0.43 (0.49)	4,945	0.42 (0.49)	10,037	0 – 1

Table 12. Descriptive Statistics for Persistence/Credit Accumulation Outcomes

	Control Group		Treatment Group		Total Sample
	Mean (SD)	n	Mean (SD)	n	Mean (SD)	n	Range
Credit Accumulation	109.5 (54.0)	5,092	108.9 (54.9)	4,945	109.2 (54.5)	10,037	0 – 322
Persistence	0.69 (0.46)	5,092	0.68 (0.46)	4,945	0.69 (0.46)	10,037	0 – 1
Persistence (NSC)	0.80 (0.40)	5,092	0.79 (0.41)	4,945	0.80 (0.40)	10,037	0 – 1

Table 13. Intent-To-Treat Effect of MAAPS Advisement on Cumulative GPA^[35]

VARIABLES	(1)	(2)	(3)	(4)
Treatment	0.00	0.01	-0.00	0.01
	(0.02)	(0.02)	(0.03)	(0.02)
Constant	2.87***	1.71***	2.87***	1.61***
	(0.01)	(0.06)	(0.01)	(0.14)
Observations	9,980	9,980	9.980	9,980
R-squared	0.00	0.09	0.00	0.10
Baseline Covariates	NO	YES	NO	YES
Institutional FE	NO	NO	YES	YES

Table 14. Intent-To-Treat Effect of MAAPS Advisement on Graduation

VARIABLES	(1)	(2)	(3)	(4)
Treatment	0.01	0.01	0.01	0.01
	(0.01)	(0.01)	(0.01)	(0.01)
Constant	0.40***	0.14***	0.40***	0.08
	(0.01)	(0.03)	(0.01)	(0.08)
Observations	10,037	10,037	10,037	10,037
R-squared	0.00	0.05	0.00	0.06
Baseline Covariates	NO	YES	NO	YES
Institutional FE	NO	NO	YES	YES

Table 15. Intent-To-Treat Effect of MAAPS Advisement on Graduation (NSC)

VARIABLES	(1)	(2)	(3)	(4)
Treatment	0.01	0.01	0.01	0.01
	(0.01)	(0.01)	(0.01)	(0.01)
Constant	0.42***	0.13***	0.42***	0.07
	(0.01)	(0.03)	(0.01)	(0.08)
Observations	10,037	10,037	10,037	10,037
R-squared	0.00	0.06	0.00	0.06
Baseline Covariates	NO	YES	NO	YES
Institutional FE	NO	NO	YES	YES

Table 16. Intent-To-Treat Effect of MAAPS Advisement on Credit Accumulation

VARIABLES	(1)	(2)	(3)	(4)
Treatment	-0.56	-0.36	-0.43	-0.05
	(1.09)	(1.00)	(1.37)	(1.03)
Constant	109.46***	80.60***	109.40***	55.69***
	(0.76)	(3.46)	(0.67)	(6.57)
Observations	10,037	10,037	10,037	10,037
R-squared	0.00	0.15	0.00	0.15
Baseline Covariates	NO	YES	NO	YES
Institutional FE	NO	NO	YES	YES

 Table 17. Intent-To-Treat Effect of MAAPS Advisement on Persistence

VARIABLES	(1)	(2)	(3)	(4)
Treatment	-0.01	-0.01	-0.01	-0.01
	(0.01)	(0.01)	(0.01)	(0.01)
Constant	0.69***	0.42***	0.69***	0.37***
	(0.01)	(0.03)	(0.01)	(0.07)
Observations	10,037	10,037	10,037	10,037
R-squared	0.00	0.04	0.00	0.04
Baseline Covariates	NO	YES	NO	YES
Institutional FE	NO	NO	YES	YES

Table 18. Intent-To-Treat Effect of MAAPS Advisement on Persistence (NSC)

VARIABLES	(1)	(2)	(3)	(4)
Treatment	-0.02*	-0.01*	-0.02*	-0.01
	(0.01)	(0.01)	(0.01)	(0.01)
Constant	0.80***	0.58***	0.80***	0.56***
	(0.01)	(0.03)	(0.00)	(0.05)
Observations	10,037	10,037	10,037	10,037
R-squared	0.00	0.04	0.00	0.03
Baseline Covariates	NO	YES	NO	YES
Institutional FE	NO	NO	YES	YES

Student Subgroups of Interest

Table 19. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Pell-Eligible Student Subgroup^[36]

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	0.02	0.02*✝	0.02*✝	0.72	-0.01	-0.01
	(0.03)	(0.01)	(0.01)	(1.31)	(0.01)	(0.01)
Constant	1.46***	-0.01	-0.02	50.71***	0.34***	0.51***
	(0.16)	(0.07)	(0.08)	(6.50)	(0.07)	(0.05)
Observations	8,024	8,071	8,071	8,071	8,071	8,071
R-squared	0.10	0.06	0.07	0.15	0.04	0.03
Baseline Covariates	YES	YES	YES	YES	YES	YES
Institutional FE	YES	YES	YES	YES	YES	YES

Table 20. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – First-Generation Student Subgroup

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	-0.02	0.01	0.01	-1.41	-0.02*✝	-0.02*✝
	(0.03)	(0.01)	(0.02)	(1.30)	(0.01)	(0.01)
Constant	1.77***	0.09	0.09	60.38***	0.38***	0.58***
	(0.14)	(0.08)	(0.08)	(7.44)	(0.07)	(0.05)
Observations	5,285	5,315	5,315	5,315	5,315	5,315
R-squared	0.09	0.05	0.06	0.14	0.04	0.03
Baseline Covariates	YES	YES	YES	YES	YES	YES
Institutional FE	YES	YES	YES	YES	YES	YES

Table 21. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Underrepresented Racial and Ethnic Minority Student Subgroup

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	0.02	0.01	0.01	0.35	-0.01	-0.00
	(0.03)	(0.01)	(0.01)	(1.68)	(0.01)	(0.01)
Constant	1.58***	0.01	-0.01	54.67***	0.37**	0.53***
	(0.23)	(0.13)	(0.13)	(11.89)	(0.12)	(0.09)
Observations	3,704	3,724	3,724	3,724	3,724	3,724
R-squared	0.07	0.04	0.05	0.12	0.03	0.02
Baseline Covariates	YES	YES	YES	YES	YES	YES
Institutional FE	YES	YES	YES	YES	YES	YES

Table 22. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Black Student Subgroup

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	-0.02	-0.01	-0.01	-0.12	0.00	0.01
	(0.09)	(0.04)	(0.04)	(4.60)	(0.04)	(0.05)
Constant	1.17***	-0.19	-0.19	23.71	0.08	0.43**
	(0.27)	(0.13)	(0.13)	(19.98)	(0.20)	(0.14)
Observations	1,259	1,264	1,264	1,264	1,264	1,264
R-squared	0.10	0.06	0.07	0.14	0.06	0.03
Baseline Covariates	YES	YES	YES	YES	YES	YES
Institutional FE	YES	YES	YES	YES	YES	YES

Conditional on Year One

Table 23. Intent-To-Treat Effect of MAAPS Advisement Conditional on Year One – Full Sample^[37]

VARIABLES	Outcome – since year 1
	Cumulative GPA	Credit Accumulation
Treatment	0.00	-0.14
	(0.01)	(0.73)
Outcome – year 1	0.90***	1.78***
	(0.01)	(0.12)
Constant	0.32***	34.46***
	(0.06)	(5.74)
Observations	9,965	10,037
R-squared	0.82	0.41
Baseline Covariates	YES	YES
Institutional FE	YES	YES

Appendix B. Results Tables: Georgia State Subsample

For the Georgia State subsample, we present two regression models for each primary analysis. Model 1 does not include control variables, model 2 includes baseline demographic covariates only (high school achievement scores as determined by composite ACT score, low-income status as determined by expected family contribution (EFC) at baseline, and the number of college-level credit hours transferred into the institution before the start of the Fall 2016 term). For follow-up analyses, we present the results of model 2 only.

Table 24. Descriptive Statistics for Academic Achievement Outcomes – Georgia State Subsample

	Control Group		Treatment Group		Total Sample
	Mean (SD)	n	Mean (SD)	n	Mean (SD)	n	Range
Cumulative GPA	2.94 (0.86)	466	3.11 (0.69)	485	3.03 (0.78)	951	0 – 4.3
Graduation	0.25 (0.44)	476	0.30 (0.46)	488	0.28 (0.45)	964	0 – 1
Graduation (NSC)	0.29 (0.45)	476	0.33 (0.47)	488	0.31 (0.46)	964	0 – 1

Table 25. Descriptive Statistics for Persistence/Credit Accumulation Outcomes – Georgia State Subsample

	Control Group		Treatment Group		Total Sample
	Mean (SD)	n	Mean (SD)	n	Mean (SD)	n	Range
Credit Accumulation	85.5 (41.5)	476	91.3 (39.0)	488	88.4 (40.4)	964	0 – 180
Persistence	0.62 (0.49)	476	0.66 (0.47)	488	0.64 (0.48)	964	0 – 1
Persistence (NSC)	0.76 (0.42)	476	0.80 (0.40)	488	0.78 (0.41)	964	0 – 1

Table 26. Intent-To-Treat Effect of MAAPS Advisement on Cumulative GPA – Georgia State Subsample^[38]

VARIABLES	(1)	(2)
Treatment	0.16***	0.16***
	(0.05)	(0.05)
Constant	2.94***	1.45***
	(0.04)	(0.19)
Observations	951	951
R-squared	0.01	0.09
Baseline Covariates	NO	YES

Table 27. Intent-To-Treat Effect of MAAPS Advisement on Graduation – Georgia State Subsample

VARIABLES	(1)	(2)
Treatment	0.04	0.04
	(0.03)	(0.03)
Constant	0.25***	-0.12
	(0.02)	(0.12)
Observations	964	964
R-squared	0.00	0.04
Baseline Covariates	NO	YES

Table 28. Intent-To-Treat Effect of MAAPS Advisement on Graduation (NSC) – Georgia State Subsample

VARIABLES	(1)	(2)
Treatment	0.05	0.04
	(0.03)	(0.03)
Constant	0.29***	-0.28**
	(0.02)	(0.12)
Observations	964	964
R-squared	0.00	0.06
Baseline Covariates	NO	YES

Table 29. Intent-To-Treat Effect of MAAPS Advisement on Credit Accumulation – Georgia State Subsample

VARIABLES	(1)	(2)
Treatment	5.74**	5.58**
	(2.60)	(2.55)
Constant	85.52***	59.02***
	(1.90)	(10.06)
Observations	964	964
R-squared	0.01	0.04
Baseline Covariates	NO	YES

Table 30. Intent-To-Treat Effect of MAAPS Advisement on Persistence – Georgia State Subsample

VARIABLES	(1)	(2)
Treatment	0.04	0.04
	(0.03)	(0.03)
Constant	0.62***	0.54***
	(0.02)	(0.12)
Observations	964	964
R-squared	0.00	0.00
Baseline Covariates	NO	YES

Table 31. Intent-To-Treat Effect of MAAPS Advisement on Persistence (NSC) – Georgia State Subsample

VARIABLES	(1)	(2)
Treatment	0.04	0.04
	(0.03)	(0.03)
Constant	0.76***	0.40***
	(0.02)	(0.10)
Observations	964	964
R-squared	0.00	0.02
Baseline Covariates	NO	YES

Student Subgroups of Interest

Table 32. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Georgia State Subsample, Pell-Eligible Student Subgroup

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	0.17***	0.04	0.05	6.45**	0.05	0.05*✝
	(0.05)	(0.03)	(0.03)	(2.70)	(0.03)	(0.03)
Constant	1.40***	-0.12	-0.26**	59.65***	0.49***	0.34***
	(0.18)	(0.11)	(0.11)	(9.78)	(0.12)	(0.10)
Observations	848	860	860	860	860	860
R-squared	0.09	0.04	0.07	0.05	0.00	0.03
Baseline Covariates	YES	YES	YES	YES	YES	YES

Table 33. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Georgia State Subsample, First-Generation Student Subgroup

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	0.19**	0.04	0.05	3.62	0.04	0.03
	(0.08)	(0.04)	(0.04)	(3.83)	(0.05)	(0.04)
Constant	1.47***	-0.11	-0.28	60.22***	0.48***	0.36**
	(0.29)	(0.17)	(0.18)	(14.79)	(0.18)	(0.14)
Observations	423	429	429	429	429	429
R-squared	0.08	0.03	0.04	0.04	0.00	0.02
Baseline Covariates	YES	YES	YES	YES	YES	YES

Table 34. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Georgia State Subsample, Underrepresented Racial and Ethnic Minority Student Subgroup

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	0.14**	0.04	0.03	5.97*✝	0.04	0.06*✝
	(0.06)	(0.04)	(0.04)	(3.22)	(0.04)	(0.04)
Constant	1.59***	-0.08	-0.19	58.11***	0.46**	0.36**
	(0.28)	(0.17)	(0.17)	(14.29)	(0.18)	(0.16)
Observations	561	562	562	562	562	562
R-squared	0.05	0.02	0.04	0.03	0.01	0.02
Baseline Covariates	YES	YES	YES	YES	YES	YES

Table 35. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Georgia State Subsample, Black Student Subgroup

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	0.22***	0.08*	0.08*	11.92***	0.12**	0.13***
	(0.07)	(0.04)	(0.04)	(3.76)	(0.05)	(0.04)
Constant	1.57***	-0.10	-0.17	57.01***	0.47**	0.52***
	(0.34)	(0.21)	(0.21)	(16.95)	(0.22)	(0.20)
Observations	417	418	418	418	418	418
R-squared	0.06	0.02	0.03	0.04	0.02	0.03
Baseline Covariates	YES	YES	YES	YES	YES	YES

Conditional on Year One

Table 36. Intent-To-Treat Effect of MAAPS Advisement Conditional on Year One – Georgia State Subsample

VARIABLES	Outcome – since year 1
	Cumulative GPA	Credit Accumulation
Treatment	0.02	3.91*
	(0.02)	(2.20)
Outcome – year 1	0.85***	1.74***
	(0.02)	(0.11)
Constant	0.09	46.21***
	(0.09)	(9.01)
Observations	948	964
R-squared	0.80	0.29
Baseline Covariates	YES	YES

Table 37. Intent-To-Treat Effect of MAAPS Advisement Conditional on Year One – Georgia State Subsample, Black Student Subgroup

VARIABLES	Outcome – since year 1
	Cumulative GPA	Credit Accumulation
Treatment	0.02	6.93**
	(0.03)	(3.30)
Outcome – year 1	0.84***	1.77***
	(0.03)	(0.20)
Constant	0.37**	31.89**
	(0.18)	(15.09)
Observations	417	418
R-squared	0.76	0.27
Baseline Covariates	YES	YES

Additional Analyses

Table 38. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Georgia State Subsample, Students who are not Black

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	0.11*✝	0.02	0.02	1.21	-0.02	-0.04
	(0.07)	(0.04)	(0.04)	(3.48)	(0.04)	(0.03)
Constant	1.63***	-0.10	-0.27*	55.77***	0.53***	0.37***
	(0.25)	(0.15)	(0.15)	(13.33)	(0.16)	(0.13)
Observations	534	546	546	546	546	546
R-squared	0.08	0.05	0.08	0.06	0.00	0.03
Baseline Covariates	YES	YES	YES	YES	YES	YES

Table 39. Quantile Regressions of MAAPS Advisement on Cumulative GPA – Georgia State Subsample

Outcome Measure	10th		30th		50th		70th		90th
	T-C diff	p	T-C diff	p	T-C diff	p	T-C diff	p	T-C diff	p
Cumulative GPA	0.42	0.00	0.17	0.03	0.11	0.05	0.05	0.26	0.04	0.36

Appendix C. Results Tables: Institutional Subsamples

After adjusting for multiple outcomes within a given outcome domain using the Benjamini-Hochberg method, regression results for ten of the eleven participating institutions did not reach statistical significance (p<0.10). We present the results of model 2 only.

Table 40. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Institutional Subsample 1^[39]

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	-0.08	0.00	-0.01	-3.55	-0.02	-0.03
	(0.06)	(0.03)	(0.03)	(2.81)	(0.03)	(0.03)
Constant	2.12***	0.22**	0.22**	61.41***	0.46***	0.54***
	(0.17)	(0.10)	(0.10)	(9.02)	(0.10)	(0.09)
Observations	991	995	995	995	995	995
R-squared	0.07	0.04	0.04	0.15	0.03	0.03
Baseline Covariates	YES	YES	YES	YES	YES	YES

Table 41. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Institutional Subsample 2

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	0.01	-0.03	-0.02	-1.35	-0.02	-0.01
	(0.06)	(0.03)	(0.03)	(2.72)	(0.03)	(0.03)
Constant	1.40***	0.09	0.12	48.98***	0.29***	0.49***
	(0.17)	(0.09)	(0.09)	(8.38)	(0.09)	(0.08)
Observations	1,092	1,093	1,093	1,093	1,093	1,093
R-squared	0.10	0.07	0.07	0.16	0.06	0.04
Baseline Covariates	YES	YES	YES	YES	YES	YES

Table 42. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Institutional Subsample 3

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	-0.06	-0.03	-0.04	-2.52	-0.03	-0.03
	(0.06)	(0.03)	(0.03)	(2.85)	(0.03)	(0.03)
Constant	1.03***	-0.17*	-0.14	21.57**	0.11	0.47***
	(0.21)	(0.10)	(0.10)	(10.31)	(0.10)	(0.10)
Observations	1,062	1,082	1,082	1,082	1,082	1,082
R-squared	0.15	0.10	0.11	0.18	0.06	0.05
Baseline Covariates	YES	YES	YES	YES	YES	YES

Table 43. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Institutional Subsample 4

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	-0.01	0.03	0.01	2.81	0.03	0.01
	(0.05)	(0.03)	(0.03)	(2.62)	(0.03)	(0.03)
Constant	1.34***	-0.02	-0.05	44.75***	0.42***	0.52***
	(0.19)	(0.11)	(0.11)	(9.87)	(0.11)	(0.10)
Observations	863	868	868	868	868	868
R-squared	0.14	0.07	0.08	0.11	0.02	0.02
Baseline Covariates	YES	YES	YES	YES	YES	YES

Table 44. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Institutional Subsample 5

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	0.06	0.05	0.04	5.56	0.05	0.02
	(0.05)	(0.03)	(0.03)	(4.26)	(0.03)	(0.03)
Constant	1.39***	0.01	0.00	76.98***	0.41***	0.69***
	(0.19)	(0.11)	(0.11)	(14.57)	(0.11)	(0.09)
Observations	852	854	854	854	854	854
R-squared	0.14	0.09	0.08	0.21	0.05	0.05
Baseline Covariates	YES	YES	YES	YES	YES	YES

Table 45. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Institutional Subsample 6

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	-0.08	0.04	0.03	-2.80	-0.03	-0.02
	(0.06)	(0.03)	(0.03)	(2.74)	(0.03)	(0.03)
Constant	1.20***	-0.29***	-0.30***	11.29	-0.09	0.23**
	(0.20)	(0.08)	(0.08)	(8.96)	(0.10)	(0.09)
Observations	930	934	934	934	934	934
R-squared	0.24	0.16	0.18	0.36	0.19	0.13
Baseline Covariates	YES	YES	YES	YES	YES	YES

Table 46. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Institutional Subsample 7

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	0.07	-0.01	0.00	-2.49	-0.04	-0.04*✝
	(0.05)	(0.03)	(0.03)	(2.62)	(0.03)	(0.02)
Constant	2.31***	0.62***	0.66***	90.32***	0.59***	0.87***
	(0.16)	(0.12)	(0.12)	(9.65)	(0.10)	(0.08)
Observations	905	908	908	908	908	908
R-squared	0.06	0.05	0.05	0.15	0.11	0.02
Baseline Covariates	YES	YES	YES	YES	YES	YES

Table 47. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Institutional Subsample 8

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	0.05	0.04	0.04	-0.81	-0.01	-0.03
	(0.04)	(0.03)	(0.03)	(2.40)	(0.03)	(0.02)
Constant	3.22***	0.51**	0.61***	67.19***	0.52***	0.91***
	(0.29)	(0.21)	(0.21)	(14.94)	(0.16)	(0.14)
Observations	974	974	974	974	974	974
R-squared	0.04	0.05	0.05	0.17	0.02	0.01
Baseline Covariates	YES	YES	YES	YES	YES	YES

Table 48. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Institutional Subsample 9

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	-0.01	-0.01	0.00	1.26	-0.01	-0.02
	(0.04)	(0.03)	(0.03)	(3.63)	(0.03)	(0.02)
Constant	2.13***	0.47***	0.41***	124.84***	0.71***	0.77***
	(0.15)	(0.12)	(0.12)	(12.99)	(0.10)	(0.08)
Observations	990	995	995	995	995	995
R-squared	0.03	0.02	0.02	0.06	0.01	0.01
Baseline Covariates	YES	YES	YES	YES	YES	YES

Table 49. Intent-To-Treat Effect of MAAPS Advisement on all Outcomes – Institutional Subsample 10

VARIABLES	Cumulative GPA	Graduation	Graduation (NSC)	Credit Accumulation	Persistence	Persistence (NSC)
Treatment	-0.02	0.05	0.05	-2.54	-0.04	-0.03
	(0.07)	(0.05)	(0.05)	(3.37)	(0.04)	(0.03)
Constant	2.14***	0.37	0.37	68.38***	0.67***	0.70***
	(0.36)	(0.28)	(0.28)	(15.34)	(0.17)	(0.16)
Observations	370	370	370	370	370	370
R-squared	0.05	0.04	0.04	0.19	0.03	0.03
Baseline Covariates	YES	YES	YES	YES	YES	YES

 

Endnotes

1. Grant Number: P116X150015. See “Department Awards $60 Million in First in the World Grants to 18 Colleges, Universities and Organizations,” US Department of Education, 21 September 2015, https://www.ed.gov/news/press-releases/department-awards-60-million-first-world-grants-17-colleges-universities-and-organizations.

   ↑

2. Rayane Alamuddin, Daniel Rossman, and Martin Kurzweil, “Monitoring Advising Analytics to Promote Success (MAAPS): Pre-Analysis Plan,” 13 November 2019, https://mfr.osf.io/render?url=https://osf.io/29zbw/?action=download%26mode=render.

   ↑

3. Rayane Alamuddin, Daniel Rossman, and Martin Kurzweil, “Interim Findings Report: MAAPS Advising Experiment,” Ithaka S+R, 27 June 2019, https://doi.org/10.18665/sr.311567.

   ↑

4. Eric P. Bettinger and Rachel B. Baker, “The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Advising,” Educational Evaluation and Policy Analysis 36, no. 1 (March 2014): 3–19, doi:10.3102/0162373713500523.

   ↑

5. Martin Kurzweil and D. Derek Wu, “Building a Pathway to Student Success at Georgia State University,” Ithaka S+R, 23 April 2015, https://doi.org/10.18665/sr.221053.

   ↑

6. Most commonly, business-as-usual advisement at the participating institutions involves a larger advisor-to-student ratio, fewer communications from advisors, shorter advisor-student meetings, and lower levels of proactive outreach to students based on in-term and end-of-term student information. Through business-as-usual, students are also less likely to work on personalized and dynamic four-year degree plans with their advisors.

   ↑

7. For detailed descriptions of the MAAPS advisement activities and additional background information on the study and intervention, please see earlier MAAPS reports: Rayane Alamuddin, Daniel Rossman, Martin Kurzweil, “Monitoring Advising Analytics to Promote Success (MAAPS): Evaluation Findings from the First Year of Implementation,” Ithaka S+R, 4 April 2018, https://doi.org/10.18665/sr.307005, and Rayane Alamuddin, Daniel Rossman, Martin Kurzweil, “Interim Findings Report: MAAPS Advising Experiment,” Ithaka S+R, 27 June 2019, https://doi.org/10.18665/sr.311567.

   ↑

8. Selected students were informed of their selection into the study, but not of their selection into the treatment or control group, via email, on the third day of the Fall 2016 term. This ensured that student matriculation at the participating institutions was not impacted by the study and allowed students to opt out of the study at that time irrespective of their assigned study group.

   ↑

9. For additional technical details on student sampling procedures, please see an earlier MAAPS report, Rayane Alamuddin, Daniel Rossman, and Martin Kurzweil, “Technical Supplement – Interim Findings Report: MAAPS Advising Experiment,” Ithaka S+R, 27 June 2019, https://doi.org/10.18665/sr.311566.

   ↑

10. There were 48 cases in which a student was categorized as a graduate in the student administrative data but not categorized as such in the NSC data because their administrative records were not successfully matched to records in the NSC database. To address this set of discrepancies, we recoded the 48 students’ NSC records to indicate that they earned a bachelor’s degree from their home MAAPS institution by the end of the Spring 2020 term.

    ↑

11. A total of 208 students graduated with a bachelor’s degree by the end of the Spring 2020 term from an institution other than their initial MAAPS institution.

    ↑

12. Includes credits for AP, dual-enrollment, and remedial/developmental coursework, as well as credits the student successfully earned at another institution that have been accepted at the student’s current institution.

    ↑

13. A student is considered enrolled if they were enrolled through the end of the Spring 2020 term. Students who withdrew from their institution(s) before the conclusion of the Spring 2020 term are considered not enrolled for that term. Half-time status is determined by the institution, but typically is considered at least six credits enrolled per term.

    ↑

14. There are 87 cases in which a student is categorized as having persisted in the student administrative data but not categorized as such in the NSC data. There are two causes for this set of discrepancies. The first is a timing issue: the persistence variable is based in part on whether a student was enrolled according to their institution’s Spring 2020 term census, usually taken a few weeks into the term, while the persistence variable using NSC data is based in part on whether a student was enrolled at the end of the Spring 2020 term. Students who were enrolled as of census and then left the institution soon after are marked as enrolled and as having persisted in the student administrative data but not enrolled and not having persisted in the NSC data. The second is that the persistence variable using NSC data has an additional condition: a student must be enrolled at least half time. Students enrolled less than half time are marked as having persisted in the student administrative data but not having persisted in the NSC data. In addition, there are 73 cases in which a student is categorized as having persisted at their initial MAAPS institution in the NSC data but not categorized as such in the student administrative data. There are two causes for this set of discrepancies as well. Again, the first is a timing issue. Students whose enrollment status post-census changed from not enrolled to enrolled are marked as not having persisted in the student administrative data but having persisted in the NSC data. The other is that the MAAPS study is focused on the impact of the MAAPS intervention on undergraduate outcomes so we do not collect data on students once they become graduate students. As a result, graduate students are marked as not enrolled and not having persisted in the student administrative data but may be marked as enrolled and having persisted in the NSC data. This includes 22 students enrolled in Purdue University’s Doctor of Pharmacy (PharmD) Program who are no longer considered undergraduates and therefore marked as not enrolled in the student administrative data once they enter the program. A total of 1,090 students persisted (either graduated with a bachelor’s degree or were enrolled at least half-time) by the end of the Spring 2020 term at a degree-granting institution other than their initial MAAPS institution.

    ↑

15. What Works Clearinghouse Standards Handbook, Version 4.1 (p.10), https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Standards-Handbook-v4-1-508.pdf.

    ↑

16. For students who submitted SAT scores, we used concordance tables provided by the College Board to convert to ACT composite scores.

    ↑

17. Controls also include a dummy variable capturing whether the student is one of 22 enrolled in Purdue University’s Doctor of Pharmacy (PharmD) Program. These students are no longer considered undergraduates once they enter the program. Unlike traditional graduate programs, this typically occurs two years into their college career, at which point their credits accumulated and cumulative GPA are frozen. In addition, they do not earn an undergraduate degree despite the fact that they are pursuing a PhD in Pharmacy. We decided to control for these students’ outcomes due to the unusual nature of the program and the fact that their progress is misleadingly low.

    ↑

18. What Works Clearinghouse Standards Handbook, Version 4.1 (p.38), https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Standards-Handbook-v4-1-508.pdf.

    ↑

19. We considered outcome measures generated by NSC data and equivalent administrative data outcomes in the same domain, so corrected p-values account for two other variables in the same domain.

    ↑

20. For the first time, student subgroups of interest included Black students after we were informed that the UIA is working on a student success initiative focused on supporting Black students. We were asked to explore whether MAAPS has benefitted this student subgroup and looked at the intervention’s impacts for the full sample and for each institutional subsample.

    ↑

21. Cumulative GPA result remains statistically significant at the p<0.10 level after correcting for multiple outcomes in the same domain. The Benjamini-Hochberg corrected p-value is 0.003.

    ↑

22. Credit accumulation result remains statistically significant at the p<0.10 level after correcting for multiple outcomes in the same domain.The Benjamini-Hochberg corrected p-value is 0.087.

    ↑

23. Results remain statistically significant at the p<0.10 level after correcting for multiple outcomes in the same domain. The Benjamini-Hochberg corrected p-values are 0.006 for cumulative GPA and 0.076 for graduation and graduation (NSC).

    ↑

24. Results remain statistically significant at the p<0.10 level after correcting for multiple outcomes in the same domain. The Benjamini-Hochberg corrected p-values are 0.003 for credit accumulation and persistence (NSC) and 0.012 for persistence.

    ↑

25. Todd Feathers, “Major Universities Are Using Race as a ‘High Impact Predictor’ of Student Success,” The Markup, 2 March 2021, https://themarkup.org/news/2021/03/02/major-universities-are-using-race-as-a-high-impact-predictor-of-student-success.

    ↑

26. For additional details on the methodology used in the implementation study, please see Appendix A of an earlier MAAPS report, Rayane Alamuddin, Daniel Rossman, and Martin Kurzweil, “Technical Supplement – Interim Findings Report: MAAPS Advising Experiment,” Ithaka S+R, 27 June 2019, https://doi.org/10.18665/sr.311566; and for a more comprehensive description of findings from the implementation study, please see Rayane Alamuddin, Daniel Rossman, Martin Kurzweil, “Interim Findings Report: MAAPS Advising Experiment,” Ithaka S+R, 27 June 2019, https://doi.org/10.18665/sr.311567.

    ↑

27. For this analysis, less academically prepared is defined as having an ACT composite score in the bottom 50th percentile.

    ↑

28. Among those more academically prepared, Black treatment group students at Georgia State also experienced more student and other initiated advising triggers.

    ↑

29. For additional details on the administration of the surveys and the survey items, please see earlier Ithaka S+R reports on the MAAPS study, including: Rayane Alamuddin, Daniel Rossman, Martin Kurzweil, “Monitoring Advising Analytics to Promote Success (MAAPS): Evaluation Findings from the First Year of Implementation,” Ithaka S+R, 4 April 2018, https://doi.org/10.18665/sr.307005, and Rayane Alamuddin, Daniel Rossman, and Martin Kurzweil, “Technical Supplement – Interim Findings Report: MAAPS Advising Experiment,” Ithaka S+R, 27 June 2019, https://doi.org/10.18665/sr.311566.

    ↑

30. The proactive scale was not introduced until the 2018 survey so for that scale, the findings were only consistent across two years.

    ↑

31. Excluded students who were not enrolled according to their institution’s Spring 2019 census.

    ↑

32. For this analysis, a high-performing student is defined as entering the spring term in which the particular survey was administered with a cumulative GPA in the upper half of the distribution.

    ↑

33. See, for example, Rayane Alamuddin, Daniel Rossman, and Martin Kurzweil, “Interim Findings Report: MAAPS Advising Experiment,” Ithaka S+R. 27 June 2019, https://doi.org/10.18665/sr.311567.

    ↑

34. Controls also include a dummy variable capturing whether the student is one of 22 enrolled in Purdue University’s PharmD Program.

    ↑

35. For all regression tables in Appendix A, robust standard errors are in parentheses, with *** indicating p<0.01, ** indicating p<0.05, and * indicating p<0.10. ✝ indicates that the result is not statistically significant at the p<0.10 level after correcting for multiple outcomes in the same domain.

    ↑

36. All regressions looking at Pell-eligible students only do not include low-income status at baseline as a control.

    ↑

37. All regressions looking at the effect on MAAPS advisement on credit accumulation conditional on year one do not include the number of credits earned prior to the Fall 2016 term as a control.

    ↑

38. For all regression tables in Appendix B, robust standard errors are in parentheses, with *** indicating p<0.01, ** indicating p<0.05, and * indicating p<0.10. ✝ indicates that the result is not statistically significant at the p<0.10 level after correcting for multiple outcomes in the same domain.

    ↑

39. For all regression tables in Appendix C, robust standard errors are in parentheses, with *** indicating p<0.01, ** indicating p<0.05, and * indicating p<0.10. ✝ indicates that the result is not statistically significant at the p<0.10 level after correcting for multiple outcomes in the same domain.

    ↑