Technical Supplement – Interim Findings Report: MAAPS Advising Experiment

Overview

Monitoring Advising Analytics to Promote Success (MAAPS) is a multi-institutional project of the University Innovation Alliance (UIA), supported by a U.S. Department of Education First in the World Grant to Georgia State University, the lead UIA member on this project. MAAPS is a large-scale randomized-controlled trial designed to validate the effectiveness of technology-enhanced proactive advisement in increasing retention, progression, and achievement for low-income and first-generation college students. Addressing documented obstacles to college completion that disproportionately impact at-risk populations, the four-year study tracks cohorts of low-income and first-generation students enrolled at the 11 large public universities that constitute the UIA: Arizona State University, Georgia State University, Iowa State University, Michigan State University, The Ohio State University, Oregon State University, Purdue University, University of California Riverside, University of Central Florida, University of Kansas, and University of Texas at Austin. Ithaka S+R serves as the independent evaluator of the study.

This technical supplement accompanies the MAAPS interim impact and implementation findings report, which describes the MAAPS intervention and its activities, a summary of impact findings after two and a half years, and findings from the implementation study. This technical document presents 1) a description of the student sampling procedures for MAAPS and the key baseline and outcome measures; 2) technical details on student and analytic samples, and the analytic approaches used to assess the impact of MAAPS; 3) detailed results of the impact of MAAPS advisement on participating students’ outcomes in the aggregate sample and at each participating institution after two academic years into the intervention (Fall 2016 – Spring 2018) ^[1]; and 4) a description of the implementation study methodology.

Student Sampling Procedures for MAAPS

This section provides a detailed description of the sampling for MAAPS, including student eligibility criteria, student selection and random assignment, and current MAAPS study samples by institution.

Student Eligibility

In order to be eligible for selection into the MAAPS study, students had to meet the following eligibility criteria at that time of selection in the summer of 2016:

1. Matriculate at one of the participating institutions as a first-time incoming freshman 2016-2017 bachelor-degree seeking student. Due to selection timing, matriculation was estimated based on students’ enrollment at the institution approximately four weeks before the start of the Fall 2016 term^[2].
2. Have completed a FAFSA application that was not subsequently rejected or pending.
3. Be either Pell-eligible or a first-generation student, or both, based on the FAFSA application^[3].
4. Not be an NCAA student athlete or part of another special student group that receives advising services incompatible with MAAPS advising^[4].

Student Selection & Random Assignment

In order to allow sites to launch MAAPS as early as possible during participating students’ first academic term, students were selected and assigned to study groups ahead of time during the summer of 2016 with the understanding that some selected students would eventually be dropped from the study due to ineligibility that could not be identified at the time of selection (e.g. non-matriculation).

Each site sent the evaluator a list of all its eligible students based on the criteria listed above, along with their desired final sample size for their site and expected rate of student non-matriculation in the fall of 2016. Sites were asked to include at least 900 students in their sample, but given the discretion to select more students should they have the advising capacity. One site decided to select students from a specific program that would include 500 or fewer students.

Because students were selected before they matriculated, sites used data from their 2015-2016 cohort to estimate the percentage of eligible students who would not end up matriculating in the fall of 2016 (and therefore be ineligible to participate). The evaluator oversampled students based on each site’s estimates and randomly selected and assigned students at each site using the same procedures. A total of 10,946 students were selected from a pool of 20,018 MAAPS-eligible students.

Students at each participating site location were assigned to intervention or control by lottery at the time of selection, approximately four weeks before the start of the Fall 2016 term at their institution. The evaluator randomly assigned students to each of the two conditions by sorting them into groups based on randomly generated numbers, using standard statistical software. The random assignment process was conducted in such a way that all selected MAAPS-eligible students had an equal chance of being assigned to the intervention or control condition, using the following identical procedures and steps across all participating sites:^[5]

1. The “set seed” Stata 13 command was used to ensure that sample selection and student assignment are uniform across sites, and reproducible. The seed, which is the number with which Stata starts its algorithms, is the random number “431309934” and was generated through the website “random.org.”
2. Using the “sample” command in Stata, a subsample of eligible students was randomly selected from the pool of eligible students at the given site. The desired sample size of selected students was identified based on the total number of students the site was willing to serve, and oversampling based on the expected percentage of selected students who would turn out to be ineligible for the study.
3. Using the “runiform” command in Stata, each student selected through step 2 was assigned a random number between 0 and 1. Selected students were then sorted based on their randomly generated number, in ascending order.
4. Using the “group” command in Stata, all selected students were grouped into two groups (1=control; 2=treatment) based on their random number generated in step 3.

After selection and random assignment, all students who were selected into the study were informed of their participation in the study on the third day of the Fall 2016 term through an email announcement that also provided instructions on how to opt out of the study. Active student consent was not required for the study, and as a consequence no study students were lost due to the inability to secure consent for participation. Eligible students who actively opted out of the study after receiving the email announcement on the third day of the Fall 2016 term are considered attriters.

Sample Exclusions after Random Assignment

The final student study sample for the study was finalized after the conclusion of the Fall 2016 term at each institution. This is because one of the main eligibility criteria for selection into the study, matriculation at participating institutions, could not be established prior to the Fall 2016 term census, and other criteria that made students ineligible for participation were not always known ahead of time (e.g. the student was, at baseline, a returning student, NCAA athlete, had special-advising status, or was never a Pell-eligible or first generation student).

Since the conclusion of the Fall 2016 term, sites continued to identify non-eligible students to be removed from the study.^[6] Students are considered non-eligible for the study if the following criteria held true prior to the start of the evaluation study and did not factor into students’ group assignment:

• The student turned out to have baseline characteristics that render them non-eligible to participate in the study. This includes students who turn out to be neither Pell-eligible nor first-generation, to be NCAA athletes or participating in a special advising program similar to MAAPS, or to be transfer students^[7] (n=97). These students were considered “non-eligible by demographics” and were removed from the final sample.
• Prior to the start of the study (i.e. on or before day 3 of the Fall 2016 term when the letter informing students of their selection into the study was sent out), the student had not matriculated at the institution where they had originally enrolled and were originally selected for the study (n=360). These students were considered “non-eligible non-matriculators” and were removed from the study.

It is important to note that students who ended their enrollment or withdrew from the institution after the start of the study (i.e. after day 3 of the Fall 2016 term, when the letter informing students of their selection into the study was sent out) were retained in the final sample and considered attriters. Students who actively opted out of the study or passed away after the start of the study are also considered attriters.

Table 1 provides a breakdown of final samples at each site and across the study. In summary, the final study sample includes 10,489 students.

Table 1. MAAPS Study Samples by Participating Sites

	MAAPS-eligible at selection	Randomly selected & assigned			Included in the final sample
		C	T	Total	C	T	Total
ASU	3,845	519	518	1,037	507	504	1011
Georgia State	1,998	520	520	1,040	492	502	994
ISU	1,520	615	615	1,230	584	578	1162
KU	1,173	587	586	1,173	565	559	1124
MSU	1,830	465	465	930	456	456	912
Ohio State	2,615	512	512	1,024	494	499	993
Oregon State	920	460	460	920	437	430	867
Purdue	964	482	482	964	472	469	941
UC Riverside	3,534	556	556	1,112	544	544	1088
UCF	1,203	550	550	1,100	503	503	1006
UT Austin	416	208	208	416	196	195	391
Total	20,018	5,474	5,472	10,946	5,250	5,239	10,489[8]

Key Baseline & Outcome Measures

The six study outcome measures fall under the categories of academic achievement and persistence/credit accumulation. Four of the six outcome measures are included in the analyses presented in this report and the interim report it accompanies; two outcome measures pertaining to student persistence will be finalized at the conclusion of the study during the summer of 2019 and presented in the final project report. Due to the nature of the selected study outcomes, which are specific to students’ postsecondary experiences and performance, we rely on one baseline measure of high school achievement for all outcomes.

The outcome measures included in this report are derived from participating institutions’ official administrative data records through the last term for which data was collected (Spring 2018 term for this report). Designated and qualified staff members at each institution collect administrative data for the treatment and control group at the same time, using the same procedures, between five and nine weeks after the conclusion of each main academic term. Revisions and corrections to these files are made through the following main academic term as needed.

Outcome Measures – Academic Achievement

Credit Success Rate: The proportion of all credits the student attempted at their home institution since the start of the intervention that the student subsequently earned.^[9] Scores range from 0 to 1.

Cumulative GPA: The student’s cumulative GPA as determined by their home institution. Scores range from 0 to 4.3.

Outcome Measures – Persistence/Credit Accumulation

Credit Accumulation: The total number of credits the student has earned to date, as determined by their home institution.^[10] Continuous variable.

Continuous Enrollment: Whether the student was continuously enrolled at their home institution in non-optional academic terms between the Fall 2016 term and most recent term.^[11] A student is considered enrolled in a given term if they were enrolled at the time of the institution’s census for that term. Dichotomous variable with values of 0 or 1.

Baseline Measure – Academic Achievement & Persistence

High School Achievement & College Readiness: The student’s highest composite ACT score recorded by the participating institution where the student enrolled. For students who submitted SAT scores, concordance tables provided by the College Board were used to convert SAT composite scores to ACT composite scores.

Impact Analyses & Findings

This section provides details and results of the analyses used to assess the impact of the MAAPS intervention on participating students’ outcomes in the aggregate sample and at each participating institution after two academic years.

Year Two – Full Sample

Study and Analytic Samples

Table 2 presents descriptive data for the full MAAPS sample. The original full sample consists of 10,489 students. At the time of this report, a total of 447 students had opted out of the study or passed away (attriters), resulting in a 4.3 percent overall attrition rate and a final analytic sample of 10,042 students.

Table 2. Descriptives for Full MAAPS Sample

	Control Group n (%)	Treatment Group n (%)	Total N
Original study sample size	5,250 (50.1%)	5,239 (49.9%)	10,489
Current analytic sample size	5,093 (50.7%)	4,949 (49.3%)	10,042

Table 3 presents attrition information for each analytic sample reported in this section. All attrition rates are considered low according to the WWC standards,^[12] yielding a tolerable threat of bias under cautious assumptions regarding the exogenous nature of the attrition.

Table 3. Analytic Sample and Attrition Information for All Outcome Measures – Full Sample

Outcome Measure	Control Group		Treatment Group		Diff. Attrition (pp)	Overall Attrition
	# original sample	# analytic sample	# original sample	# analytic sample
Credit Success Rate	5,250	5,047	5,239	4,916	2.3	5.0%
Cumulative GPA	5,250	5,057	5,239	4,924	2.3	4.8%
Credit Accumulation	5,250	5,092	5,239	4,948	2.5	4.3%
Continuous Enrollment	5,250	5,093	5,239	4,949	2.5	4.3%

Analytic Approach

We employ linear regression analyses to assess the aggregate intent-to-treat effect of the MAAPS intervention for two academic years on the specified outcomes across the 11 participating institutions. Unless otherwise noted, we present four regression models for each analysis, with each model presenting a different or additional set of control variables. Model 1 does not include control variables, model 2 includes baseline demographic covariates only (high school achievement scores, low-income status at baseline, and the number of college-level credit hours transferred in before the start of the Fall 2016 term), model 3 includes institutional fixed effects only, and model 4 includes both baseline demographic covariates and institutional fixed effects.

A total of eight students have missing low-income status data, and a total of 308 students (151 treatment and 157 control) have missing high school achievement scores. We addressed missing baseline data in accordance with WWC standards,^[13] by replacing missing values with a constant of zero and adding a missing data indicator for the given baseline measure in the analysis.

Institutional fixed effects are included to account for idiosyncrasies across the 11 participating institutions around samples, implementation of the intervention, and policies regarding enrollment deadlines, credit accrual, and GPA calculations. We present the basic p-value for each analysis, and then present corrections to adjust for multiple outcomes within a given outcome domain when statistically significant.

The final model (4) is estimated as follows:

Y_ij = δ+ β*TREATMENT_i + αX_i + γ*INST_j + ε_ij

Where Y is an outcome for individual i at institution j, TREATMENT indicates whether the student was in the treatment or control group, X is a vector of control variables,^[14] and INST represents the institutional fixed effects.

Results

Academic Achievement Outcomes

Table 4 presents the impact analysis results estimating the intent-to-treat effect of MAAPS advisement on students’ credit success rate and cumulative GPA, based on model 4 of the regression analysis. Appendix B presents descriptive statistics for these two academic achievement outcomes and regression analysis results for each model.

Table 4. Intent-To-Treat Effect of MAAPS Advisement on Achievement Outcomes – Full Sample

Outcome measure	Control Group			Treatment Group			T - C diff.	Std. diff.	p
	n	Mean	(SD)	n	Adj. mean	(SD)
Credit Success Rate	5,047	0.88	(0.21)	4,916	0.88	(0.21)	-0.00	0.00	0.779
Cumulative GPA	5,057	2.83	(0.85)	4,924	2.84	(0.86)	0.01	0.00	0.732

The findings indicate that in the aggregate sample, assignment to the treatment group does not have a significant impact on either academic achievement outcome after two academic years.

Persistence/Credit Accumulation Outcomes

Table 5 presents the impact analysis results estimating the intent-to-treat effect of MAAPS advisement on students’ credit accumulation and enrollment, based on model 4 of the regression analysis. Appendix B presents descriptive statistics for these two persistence/credit accumulation outcomes and regression analysis results for each model.

Table 5. Intent-To-Treat Effect of MAAPS Advisement on Persistence Outcomes – Full Sample

Outcome measure	Control Group			Treatment Group			T - C diff.	Std. diff.	p
	n	Mean	(SD)	n	Adj. mean	(SD)
Credit Accumulation	5,092	65.50	(29.22)	4,948	65.39	(29.31)	-0.11	-0.38	0.818
Continuous Enrollment	5,093	0.78	(0.42)	4,949	0.77	(0.42)	-0.01	-0.01	0.278

The findings indicate that in the aggregate sample, assignment to the treatment group does not have a significant impact on either persistence/credit accumulation outcome after two academic years.

Year Two – Additional Analyses of Full Sample

We conducted additional exploratory analyses to uncover potential impacts not captured through the aggregate analyses of the data. This section presents detailed information on those analyses and their results.

Analytic Approach – Conditional on First Year

We first assessed whether there are treatment effects on students’ outcomes in the second year of the intervention specifically (Summer 2017 through Spring 2018 terms). Because the first year of the intervention brought early implementation challenges (e.g. late hiring of MAAPS advisors due to bureaucratic hurdles, early pushback from some institution’s advising communities, adjustment to a new advising model…), it is possible that the intervention led to impacts in the second year that are masked by an absence of impacts in the first year (Fall 2016 through Spring 2017 terms).

The final model for exploring treatment effects on students’ outcomes in the second year of the intervention in specific is based on model 4 of the core full sample impact analyses presented in this report, and is estimated as follows:

Y_2ij = δ+ β*TREATMENT_i + xY_1ij + αX_i + γ*INST_j + ε_ij

Where Y is an outcome for individual i at institution j over the course of year 1 or of year 2 of the intervention, TREATMENT indicates whether the student was in the treatment or control group, X is a vector of control variables,^[15] and INST represents the institutional fixed effects.

Results – Conditional on First Year

Table 6. Intent-To-Treat Effect of MAAPS Advisement on Student Outcomes in Year Two – Full Sample

VARIABLES	Outcome – year 2
	Credit Success Rate	Cumulative GPA	Credit Accumulation
treatment	-0.00	0.00	-0.08
	(0.00)	(0.01)	(0.28)
Outcome - year 1	0.93***	0.93***	1.30***
	(0.01)	(0.01)	(0.04)
Constant	0.04***	0.15***	10.72***
	(0.01)	(0.05)	(1.63)
Observations	9,962	9,964	10,040
R-squared	0.88	0.90	0.80
Note: Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.10

The results indicate that in its second year alone (Summer 2017 through Spring 2018 terms), the MAAPS intervention had no significant impact on students’ credit success rates, credit accumulation, or cumulative GPA in the aggregate.

Analytic Approach – Student Subgroups of Interest

Next, we explored the impact of MAAPS advisement over two years of the intervention for three student subgroups of interest: Students who were deemed eligible for a Pell grant at the time of selection into the study, based on the FAFSA application (Pell-eligible); students who were deemed to be first-generation students at the time of selection, based on parental educational achievement included in their FAFSA application (first-generation); and students from underrepresented ethnic or racial minority groups at baseline.^[16]

The final model for exploring the impact of MAAPS advisement over two years of the intervention for three student subgroups of interest is based on model 4 of the core full sample impact analyses presented in this report, and is estimated as follows:

Y_isj = δ+ β*TREATMENT_i + αX_i + γ*INST_j + ε_ij

Where Y is an outcome for individual i from subgroup s at institution j, TREATMENT indicates whether the student was in the treatment or control group, X is a vector of control variables, and INST represents the institutional fixed effects.

Results – Student Subgroups of Interest

Table 7. Intent-To-Treat Effect of MAAPS Advisement on Student Outcomes – Pell-Eligible Student Subgroup

VARIABLES	Credit Success Rate	Cumulative GPA	Credit Accumulation	Continuous Enrollment
treatment	0.00	0.02	0.30	-0.01
	(0.01)	(0.03)	(0.49)	(0.01)
Constant	0.68***	1.34***	28.42***	0.43***
	(0.03)	(0.15)	(2.94)	(0.05)
Observations	8,013	8,025	8,074	8,076
R-squared	0.04	0.11	0.38	0.03

Table 8. Intent-To-Treat Effect of MAAPS Advisement on Student Outcomes – First-Generation Student Subgroup

VARIABLES	Credit Success Rate	Cumulative GPA	Credit Accumulation	Continuous Enrollment
treatment	-0.01	-0.02	-0.56	-0.02
	(0.01)	(0.03)	(0.70)	(0.01)
Constant	0.74***	1.62***	33.57***	0.51***
	(0.03)	(0.13)	(3.45)	(0.07)
Observations	5,273	5,283	5,315	5,316
R-squared	0.04	0.10	0.36	0.03

Table 9. Intent-To-Treat Effect of MAAPS Advisement on Student Outcomes –Under-Represented Ethnic and Racial Minority Student Subgroup

VARIABLES	Credit Success Rate	Cumulative GPA	Credit Accumulation	Continuous Enrollment
treatment	0.00	0.02	0.18	-0.00
	(0.01)	(0.03)	(0.68)	(0.02)
Constant	0.71***	1.45***	30.00***	0.48***
	(0.05)	(0.21)	(4.63)	(0.10)
Observations	3,689	3,691	3,710	3,711
R-squared	0.03	0.10	0.35	0.02
Note: Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.10

The results indicate that in its first two academic years, the MAAPS interventions had no significant impact on the four achievement and persistence outcomes measured for subgroups of students identified as Pell-eligible, first-generation, or from underrepresented ethnic/racial minority groups.

Year Two – Georgia State Subsample

The 11 participating institutions varied in their chosen model for MAAPS advisement and ability to implement different key components of the intervention with high levels of fidelity. Consequently, we conducted institutional analyses to determine whether the MAAPS intervention may have differential impacts at the 11 participating sites by year two. Impacts reached statistical significance in the Georgia State subsample only; they are presented below. Regression results for the remaining ten participating institutions are presented in Appendix D.

Georgia State Study and Analytic Samples

Table 10 presents descriptive data on the MAAPS sample at Georgia State University (Georgia State), the lead institution for the project. At the start of the study, the Georgia State subsample included 994 students. At the time of this report, a total of 30 students had opted out of the study or passed away (attriters), resulting in a 3 percent overall attrition rate and a final analytic sample of 964 students.

Table 10. Descriptives for Georgia State MAAPS Subsample

	Control Group n (%)	Treatment Group n (%)	Total N
Original study sample size	492 (49.5%)	502 (50.5%)	994
Current analytic sample size	476 (49.4%)	488 (50.6%)	964

Table 11 presents attrition information for each analytic sample reported in this section. All attrition rates are considered low according to the WWC standards, yielding a tolerable threat of bias under cautious assumptions regarding the exogenous nature of the attrition.

Table 11. Attrition Information and Rates for All Outcome Measures – Georgia State Subsample

Outcome Measure	Control Group		Treatment Group		Diff. Attrition (pp)	Overall Attrition
	# randomized	# analytic sample	# randomized	# analytic sample
Credit Success Rate	492	462	502	483	2.3	4.9%
Cumulative GPA	492	464	502	484	2.1	4.6%
Credit Accumulation	492	476	502	488	0.5	3.0%
Enrollment	492	476	502	488	0.5	3.0%

Analytic Approach

We employ linear regression analyses to assess the intent-to-treat effect of the MAAPS intervention on the specified student outcomes at each of the 11 participating institutions. We conducted two regression models for each institutional analysis; model 1 does not include control variables, and model 2 includes baseline demographic covariates (high school achievement scores, low-income status at baseline, and the number of college-level credit hours transferred in before the start of the Fall 2016 term). We addressed missing baseline data in accordance with WWC standards,^[18] by replacing missing values with a constant of zero and adding a missing data indicator for the given baseline measure in the analysis.

The final model (2) is estimated as follows:

Y_i = δ+ β*TREATMENT_i + αX_i + ε_i

Where Y is an outcome for individual i, TREATMENT indicates whether the student was in the treatment or control group, and X is a vector of control variables.

Results

Academic Achievement Outcomes

Table 12 presents the impact analysis results estimating the intent-to-treat effect of MAAPS advisement at Georgia State on students’ credit success rate and cumulative GPA, based on model 2 of the regression analysis. Appendix C presents descriptive statistics for these two academic achievement outcomes and regression analysis results for each model.

Table 12. Intent-To-Treat Effect of MAAPS Advisement on Achievement Outcomes – Georgia State Subsample

Outcome measure	Control Group			Treatment Group			T - C diff.	Std. diff.	p[19]
	n	Mean	(SD)	n	Adj. mean	(SD)
Credit Success Rate	462	0.87	(0.21)	483	0.91	(0.15)	0.04	0.04	0.002
Cumulative GPA	464	2.94	(0.87)	484	3.11	(0.69)	0.17	0.17	0.000

The findings indicate that assignment to the treatment group at Georgia State has a small but significant impact, after two academic years, on both academic achievement outcomes, with and without controls for high school achievement scores, low-income status, and the number of college-level credit hours transferred at baseline.^[20] More specifically, assignment to the treatment group leads to an average four percentage-point increase in a student’s credit success rate, and a 0.17-point increase in cumulative GPA.

Persistence/Credit Accumulation Outcomes

Table 13 presents the impact analysis results estimating the intent-to-treat effect of MAAPS advisement at Georgia State on students’ credit accumulation and enrollment in the last term of data collection for year two of the intervention, based on model 2 of the regression analysis. Appendix C presents descriptive statistics for these two persistence outcomes and regression analysis results for each model.

Table 13. Intent-To-Treat Effect of MAAPS Advisement on Persistence Outcomes – Georgia State Subsample

Outcome measure	Control Group			Treatment Group			T - C diff.	Std. diff.	p[21]
	n	Mean	(SD)	n	Adj. mean	(SD)
Credit Accumulation	476	52.17	(21.74)	488	54.36	(18.61)	2.19	2.40	0.061
Continuous Enrollment	476	0.73	(0.45)	488	0.77	(0.42)	0.04	0.04	0.158

The findings indicate that assignment to the treatment group at Georgia State has a significant impact, after two academic years, on students’ credit accumulation, with and without controls for high school achievement scores, low-income status, and the number of college-level credit hours transferred at baseline, but no impact on continuous enrollment. More specifically, assignment to the treatment group leads to an increase of 2.19 cumulative credits earned within the first two academic years at Georgia State.

Year Two – Additional Analyses of Georgia State Subsample

We conducted additional exploratory analyses to further our interpretation of the Georgia State results.

Analytic Approach – Impact of Higher Enrollment on Credits Accumulated

Firstly, because the intervention has a positive impact on enrollment, we assess whether treatment impacts on credits accumulated are a function of higher enrollment rates among the treatment group.

We restricted our analyses to 720 students who enrolled in the four main terms at Georgia State^[22] to examine whether the treatment effect on credit accumulation was driven by effects on enrollment. Table 36 presents regression results for the full Georgia State sample of 964 students (model 1) and for the restricted sample (model 2).

Results – Impact of Higher Enrollment on Credits Accumulated

Table 14. Intent-To-Treat Effect of MAAPS Advisement on Credit Accumulation in Year Two – Georgia State Subsample

VARIABLES	(1)	(2)
treatment	2.19*	-0.38
	(1.17)	(0.75)
Constant	25.60***	38.82***
	(4.54)	(2.89)
Observations	964	720
R-squared	0.21	0.38
Note: Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.10

The treatment effect on credits accumulated is no longer significant when we restrict our analyses to 720 students who enrolled in the four main terms at Georgia State, suggesting that treatment impact on credits accumulated is due to its impact on enrollment.

Analytic Approach – Conditional on First Year

Additionally, because impact analyses after the first year of the intervention also revealed significant impacts in the Georgia State subsample exclusively, we conducted additional analyses to examine whether there were added impacts in year two.

The final model for exploring treatment effects on students’ outcomes in the second year of the intervention in specific is based on model 2 of the institutional subsample impact analyses presented in this report, and is estimated as follows:

Y_2i = δ+ β*TREATMENT_i + xY_1i + αX_i + ε_ij

Where Y is an outcome for individual i over the course of year 1 or of year 2 of the intervention, TREATMENT indicates whether the student was in the treatment or control group, and X is a vector of control variables.

Results – Conditional on First Year

Table 15. Intent-To-Treat Effect of MAAPS Advisement on Credit Success Rate in Year Two – Georgia State Subsample

VARIABLES	(1)	(2)
treatment	0.04***	0.01*
	(0.01)	(0.00)
Credit success rate – Year 1		0.88***
		(0.02)
Constant	0.76***	0.05*
	(0.05)	(0.03)
Observations	945	945
R-squared	0.02	0.84

Table 16. Intent-To-Treat Effect of MAAPS Advisement on Cumulative GPA in Year Two – Georgia State Subsample

VARIABLES	(1)	(2)
treatment	0.17***	0.02
	(0.05)	(0.02)
Cumulative GPA – year 1		0.92***
		(0.02)
Constant	1.38***	-0.08
	(0.19)	(0.06)
Observations	948	947
R-squared	0.09	0.91

Table 17. Intent-To-Treat Effect of MAAPS Advisement on Credit Accumulation in Year Two – Georgia State Subsample

VARIABLES	(1)	(2)
treatment	2.19*	0.98
	(1.17)	(0.75)
Credit accumulation – year 1		1.31***
		(0.05)
Constant	25.60***	13.91***
	(4.54)	(3.08)
Observations	964	964
R-squared	0.21	0.67

The results suggest that impacts on students’ cumulative GPA and credit accumulation were observed in year one, and carried over into year two with no added impact in year two alone.^[23] However, a small but significant additional impact on students’ credit success rates was observed in year two, suggesting that MAAPS continued to help students more successfully complete attempted credits at the institution by one percentage point between the Summer 2017 and Spring 2018 terms.

Appendix A: Implementation Study Methodology

The MAAPS implementation study, also conducted by the independent evaluator, includes site visits to each institution, yearly phone interviews with advising lead staff at each site, and a yearly student survey.

The independent evaluator visited each participating site once during the first two years of the intervention (Fall 2016 through Spring 2018), during which they interviewed all MAAPS staff regarding the implementation of MAAPS at their site, and conducted focus groups with volunteer students in the treatment and control groups regarding their advising experiences at the institution. In addition, the evaluator conducted a follow-up site visit to Georgia State in March 2019 to better understand why MAAPS may have had positive impacts as early as the completion of the Fall 2016 term. Additional site visits to Purdue and Oregon State were conducted in April and May 2019 after preliminary analyses of year 3 data revealed interim positive impacts on cumulative GPA at Purdue and credit accumulation at Oregon State. These site visits are complemented with yearly phone interviews with lead advising staff to gather retrospective information on each implementation year as well as prospective information on implementation plans and anticipated challenges.

The MAAPS implementation study also includes a ten-minute student advising survey, administered to all MAAPS students (in both the treatment and control groups) once a year, to explore how treatment and control group students experience advising at their institution and whether their experiences are associated with their academic progress and achievement.

Except where otherwise noted, survey items were developed by Ithaka S+R with the support and input of MAAPS advising team members. The survey includes measures of constructs hypothesized to be influenced by MAAPS advisement or associated with student’s academic outcomes, including academic self-efficacy (14 items); academic optimism (6 items); institutional know-how (6 items); and grit (12 items).^[24] It also asks students about their advising experiences at their institution since the start of the MAAPS project (i.e. the start of their Fall 2016 term), including questions about the amount and type of advising students have received (e.g. whether the student has been contacted by an advisor to setup an individual in-person meeting), students’ impressions of their academic advisors and the services they have received (e.g. students’ agreement with statements such as “My advisors have been there for me when I’ve needed them” and “I receive conflicting academic advice from different advisors”); as well as their overall satisfaction with advisement.

The first student advising survey was administered during the spring term of the first academic year of the intervention (i.e. Spring 2017) to all students who remained in the study at that time (n= 10,089). A total of 1,143 students completed at least 40 percent of the survey (11.3% response rate), including 668 treatment group students (58% of survey responses). The second student advising survey was administered during the spring term of second year of the intervention (i.e. Spring 2018) to all students who remained in the study at the time (n=10,059). A total of 942 students across the 11 institutions participated in the second student advising survey (9.4% response rate), including 533 treatment group students (57% of survey responses). Both years, all eligible students were invited to enter a gift card drawing regardless of their participation in the survey.^[25]

Appendix B: Results Tables – Full Sample

Academic Achievement

Table 18. Descriptive Statistics for Academic Achievement Outcomes

	Control Group		Treatment Group		Total Sample
	Mean (SD)	n	Mean (SD)	n	Mean (SD)	n	Range
Credit Success Rate	0.88 (0.21)	5,047	0.88 (0.21)	4,916	0.88 (0.21)	9,963	0 - 1
Cumulative GPA	2.83 (0.85)	5,057	2.83 (0.86)	4,924	2.83 (0.86)	9,981	0 - 4.3

Table 19. Intent-To-Treat Effect of MAAPS Advisement on Credit Success Rate

VARIABLES	(1)	(2)	(3)	(4)
treatment	-0.00	-0.00	-0.00	-0.00
	(0.00)	(0.00)	(0.01)	(0.01)
Constant	0.88***	0.73***	0.88***	0.72***
	(0.00)	(0.01)	(0.00)	(0.03)
Observations	9,963	9,963	9,963	9,963
R-squared	0.00	0.04	0.00	0.04
Covariates	NO	YES	NO	YES
Institutional FE	NO	NO	YES	YES

Table 20. Intent-To-Treat Effect of MAAPS Advisement on Cumulative GPA

VARIABLES	(1)	(2)	(3)	(4)
treatment	0.00	0.01	-0.00	0.01
	(0.02)	(0.02)	(0.03)	(0.02)
Constant	2.83***	1.56***	2.83***	1.47***
	(0.01)	(0.06)	(0.01)	(0.13)
Observations	9,981	9,981	9,981	9,981
R-squared	0.00	0.11	0.00	0.11
Covariates	NO	YES	NO	YES
Institutional FE	NO	NO	YES	YES
Note: Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.10

Persistence/Credit Accumulation

Table 21. Descriptive Statistics for Persistence/Credit Accumulation Outcomes

	Control Group		Treatment Group		Total Sample
	Mean (SD)	n	Mean (SD)	n	Mean (SD)	n	Range
Credit Accumulation	65.50 (29.22)	5,092	65.12 (29.31)	4,948	65.31 (29.26)	10,040	0 - 238
Enrollment	0.78 (0.42)	5,093	0.77 (0.42)	4,949	0.77 (0.42)	10,042	0 - 1

Table 22. Intent-To-Treat Effect of MAAPS Advisement on Credit Accumulation

VARIABLES	(1)	(2)	(3)	(4)
treatment	-0.38	-0.28	-0.29	-0.11
	(0.58)	(0.46)	(0.73)	(0.45)
Constant	65.50***	44.49***	65.46***	30.80***
	(0.41)	(1.58)	(0.36)	(2.92)
Observations	10,040	10,040	10,040	10,040
R-squared	0.00	0.39	0.00	0.40
Covariates	NO	YES	NO	YES
Institutional FE	NO	NO	YES	YES

Table 23. Intent-To-Treat Effect of MAAPS Advisement on Continuous Enrollment

VARIABLES	(1)	(2)	(3)	(4)
treatment	-0.01	-0.01	-0.01	-0.01
	(0.01)	(0.01)	(0.01)	(0.01)
Constant	0.78***	0.51***	0.78***	0.46***
	(0.01)	(0.03)	(0.00)	(0.05)
Observations	10,042	10,042	10,042	10,042
R-squared	0.00	0.03	0.00	0.03
Covariates	NO	YES	NO	YES
Institutional FE	NO	NO	YES	YES
Note: Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.10

Appendix C: Results Tables – Georgia State Subsample

Academic Achievement

Table 24. Descriptive Statistics for Academic Achievement Outcomes – Georgia State Subsample

	Control Group		Treatment Group		Total Sample
	Mean (SD)	n	Mean (SD)	n	Mean (SD)	n	Range
Credit Success Rate	0.87 (0.21)	462	0.91 (0.15)	483	0.89 (0.18)	945	0 - 1
Cumulative GPA	2.94 (0.87)	464	3.11 (0.69)	484	3.03 (0.79)	948	0 - 4.3

Table 25. Intent-To-Treat Effect of MAAPS Advisement on Credit Success Rate – Georgia State Subsample

VARIABLES	(1)	(2)
treatment	0.04***	0.04***
	(0.01)	(0.01)
Constant	0.87***	0.76***
	(0.01)	(0.05)
Observations	945	945
R-squared	0.01	0.02
Baseline Covariates	NO	YES

Table 26. Intent-To-Treat Effect of MAAPS Advisement on Cumulative GPA – Georgia State Subsample

VARIABLES	(1)	(2)
treatment	0.18***	0.17***
	(0.05)	(0.05)
Constant	2.94***	1.38***
	(0.04)	(0.19)
Observations	948	948
R-squared	0.01	0.09
Baseline Covariates	NO	YES
Note: Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.10

Persistence/Credit Accumulation

Table 27. Descriptive Statistics for Persistence/Credit Accumulation Outcomes – Georgia State Subsample

	Control Group		Treatment Group		Total Sample
	Mean (SD)	n	Mean (SD)	n	Mean (SD)	n	Range
Credit Accumulation	52.17 (21.74)	476	54.57 (18.61)	488	53.39 (20.24)	964	0 - 128
Continuous Enrollment	0.73 (0.45)	476	0.77 (0.42)	488	0.75 (0.44)	964	0 - 1

Table 28. Intent-To-Treat Effect of MAAPS Advisement on Credit Accumulation – Georgia State Subsample

VARIABLES	(1)	(2)
treatment	2.39*	2.19*
	(1.30)	(1.17)
Constant	52.17***	25.60***
	(1.00)	(4.54)
Observations	964	964
R-squared	0.00	0.21
Baseline Covariates	NO	YES

Table 29. Intent-To-Treat Effect of MAAPS Advisement on Continuous Enrollment – Georgia State Subsample

VARIABLES	(1)	(2)
treatment	0.04	0.04
	(0.03)	(0.03)
Constant	0.73***	0.71***
	(0.02)	(0.11)
Observations	964	964
R-squared	0.00	0.00
Baseline Covariates	NO	YES
Note: Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.10

Appendix D: Results Tables – Institutional Subsamples

After adjusting for multiple outcomes within a given outcome domain, regression results for ten of the 11 participating institutions did not reach statistical significance (p<0.10). They are presented below.

Table 30. Intent-To-Treat Effect of MAAPS Advisement on Achievement and Persistence Outcomes – Institutional Subsample 1

VARIABLES	Credit Success Rate	Cumulative GPA	Credit Accumulation	Continuous Enrollment
treatment	-0.02	-0.07	-2.07	-0.03
	(0.01)	(0.05)	(1.26)	(0.03)
Constant	0.81***	2.06***	33.22***	0.57***
	(0.04)	(0.16)	(3.94)	(0.08)
Observations	992	992	996	996
R-squared	0.02	0.08	0.39	0.03
Baseline Covariates	YES	YES	YES	YES

Table 31. Intent-To-Treat Effect of MAAPS Advisement on Achievement and Persistence Outcomes – Institutional Subsample 2

VARIABLES	Credit Success Rate	Cumulative GPA	Credit Accumulation	Continuous Enrollment
treatment	-0.01	0.02	-0.09	-0.00
	(0.01)	(0.05)	(1.17)	(0.03)
Constant	0.66***	1.27***	30.95***	0.49***
	(0.04)	(0.17)	(3.61)	(0.08)
Observations	1,093	1,093	1,094	1,094
R-squared	0.05	0.11	0.39	0.04
Baseline Covariates	YES	YES	YES	YES
Note: Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.10

Table 32. Intent-To-Treat Effect of MAAPS Advisement on Achievement and Persistence Outcomes – Institutional Subsample 3

VARIABLES	Credit Success Rate	Cumulative GPA	Credit Accumulation	Continuous Enrollment
treatment	-0.00	-0.05	-1.14	-0.04
	(0.01)	(0.06)	(1.28)	(0.03)
Constant	0.57***	0.86***	13.16***	0.13
	(0.05)	(0.20)	(4.70)	(0.10)
Observations	1,052	1,065	1,084	1,084
R-squared	0.08	0.17	0.36	0.07
Baseline Covariates	YES	YES	YES	YES

Table 33. Intent-To-Treat Effect of MAAPS Advisement on Achievement and Persistence Outcomes – Institutional Subsample 4

VARIABLES	Credit Success Rate	Cumulative GPA	Credit Accumulation	Continuous Enrollment
treatment	0.00	-0.00	1.12	0.01
	(0.01)	(0.05)	(1.11)	(0.03)
Constant	0.68***	1.10***	30.36***	0.41***
	(0.05)	(0.19)	(4.15)	(0.10)
Observations	863	863	868	869
R-squared	0.06	0.17	0.36	0.03
Baseline Covariates	YES	YES	YES	YES

Table 34. Intent-To-Treat Effect of MAAPS Advisement on Achievement and Persistence Outcomes – Institutional Subsample 5

VARIABLES	Credit Success Rate	Cumulative GPA	Credit Accumulation	Continuous Enrollment
treatment	-0.02	-0.07	-1.72	-0.05
	(0.02)	(0.06)	(1.26)	(0.03)
Constant	0.60***	1.22***	15.21***	0.16**
	(0.06)	(0.20)	(4.25)	(0.09)
Observations	929	930	934	934
R-squared	0.14	0.22	0.53	0.14
Baseline Covariates	YES	YES	YES	YES

Table 35. Intent-To-Treat Effect of MAAPS Advisement on Achievement and Persistence Outcomes – Institutional Subsample 6

VARIABLES	Credit Success Rate	Cumulative GPA	Credit Accumulation	Continuous Enrollment
treatment	0.01	0.07	2.17	0.01
	(0.01)	(0.05)	(1.83)	(0.03)
Constant	0.70***	1.20***	37.98***	0.51***
	(0.05)	(0.19)	(6.31)	(0.11)
Observations	852	852	854	854
R-squared	0.06	0.16	0.51	0.05
Baseline Covariates	YES	YES	YES	YES

Table 36. Intent-To-Treat Effect of MAAPS Advisement on Achievement and Persistence Outcomes – Institutional Subsample 7

VARIABLES	Credit Success Rate	Cumulative GPA	Credit Accumulation	Continuous Enrollment
treatment	0.01	0.07	-0.42	-0.01
	(0.01)	(0.05)	(1.21)	(0.02)
Constant	0.89***	2.09***	46.39***	0.63***
	(0.04)	(0.17)	(4.66)	(0.09)
Observations	905	905	908	908
R-squared	0.03	0.07	0.37	0.04
Baseline Covariates	YES	YES	YES	YES

Table 37. Intent-To-Treat Effect of MAAPS Advisement on Achievement and Persistence Outcomes – Institutional Subsample 8

VARIABLES	Credit Success Rate	Cumulative GPA	Credit Accumulation	Continuous Enrollment
treatment	0.00	0.03	-0.50	-0.01
	(0.01)	(0.04)	(1.15)	(0.02)
Constant	0.98***	3.11***	34.26***	0.65***
	(0.07)	(0.28)	(8.39)	(0.15)
Observations	975	975	975	975
R-squared	0.01	0.04	0.50	0.01
Baseline Covariates	YES	YES	YES	YES

Table 38. Intent-To-Treat Effect of MAAPS Advisement on Achievement and Persistence Outcomes – Institutional Subsample 9

VARIABLES	Credit Success Rate	Cumulative GPA	Credit Accumulation	Continuous Enrollment
treatment	-0.01	-0.04	-0.05	0.01
	(0.01)	(0.04)	(1.55)	(0.02)
Constant	0.85***	1.85***	66.57***	0.67***
	(0.04)	(0.14)	(5.41)	(0.08)
Observations	987	988	993	995
R-squared	0.01	0.06	0.29	0.01
Baseline Covariates	YES	YES	YES	YES

Table 39. Intent-To-Treat Effect of MAAPS Advisement on Achievement and Persistence Outcomes – Institutional Subsample 10

VARIABLES	Credit Success Rate	Cumulative GPA	Credit Accumulation	Continuous Enrollment
treatment	-0.01	-0.04	-0.67	-0.02
	(0.02)	(0.07)	(1.65)	(0.03)
Constant	0.76***	1.82***	24.75***	0.84***
	(0.09)	(0.34)	(7.82)	(0.14)
Observations	370	370	370	370
R-squared	0.03	0.07	0.53	0.02
Baseline Covariates	YES	YES	YES	YES

Endnotes

1. For detailed results of the impact of MAAPS advisement after one academic year (Fall 2016 – Spring 2017), please refer to the First Year report: https://sr.ithaka.org/publications/monitoring-advising-analytics-to-promote-success/ ↑
2. Due to their smaller pool of eligible students and later enrollment calendar, one participating institution also included students who had shown serious intent to enroll by registering for an enrollment orientation session at the institution. ↑
3. The evaluator developed and shared standard guidelines for sites on how to assess student Pell-eligibility and first-generation status using FAFSA data. ↑
4. For example, NCAA athletes and TRIO students receive intensive one-on-one advising by specialized coaches that would prevent MAAPS advisors from reaching or adequately advising the student according to program criteria. Each institution identified programs that are incompatible with MAAPS advising and excluded students from those programs from their pool of eligible students. ↑
5. Two participating sites have multiple locations or campuses. For these sites, the desired number of participating students was identified for each location/campus, and student selection and assignment was conducted at the location/campus level using the same procedure. ↑
6. Because additional non-eligible students continue to be identified and subsequently removed from the sample, sample sizes may differ from those reported in previous reports for this study. ↑
7. Some institutions opted to include transfer students whose transfer status was previously not known in order to retain a large enough sample. The number of accepted credits these students bring in from their transfer institutions are accounted for in the analyses. ↑
8. Note: The final sample decreased from 10,499 in year one to 10,489 in year two as sites identified an additional 10 non-eligible participants who were neither Pell-eligible nor first-generation students at the time of selection, and were therefore dropped from the sample. ↑
9. Includes credits earned for remedial/developmental coursework. ↑
10. Includes credits for AP, dual-enrollment, and remedial/developmental coursework, as well as credits the student successfully earned at another institution that have been accepted at the student’s current institution. ↑
11. Non-optional terms include the fall and spring terms for semester system institutions, and the fall, winter, and spring terms for quarter system institutions. The last term of data collection for this report was spring 2018. ↑
12. The WWC Standards Handbook, version 4.0 (p.11); https://ies.ed.gov/ncee/wwc/Handbooks. ↑
13. The WWC Standards Handbook, version 4.0 (p. 40; https://ies.ed.gov/ncee/wwc/Handbooks). ↑
14. Controls also include a dummy variable capturing whether the student had graduated with a bachelor’s degree during a prior term. One student had graduated with a bachelor’s degree prior to the end of the second year of the intervention. ↑
15. Controls also include a dummy variable capturing whether the student had graduated with a bachelor’s degree during a prior term. One student had graduated with a bachelor’s degree prior to the end of the second year of the intervention. ↑
16. This includes students identified in administrative data provided by each institution, using the IPEDS race/ethnicity variable, as either “Hispanic,” “Black or African American,” or “American Indian or Alaska Native.” It excludes students identified as non-resident aliens or as of “two or more races.” ↑
17. The WWC Standards Handbook, version 4.0 (p.11); https://ies.ed.gov/ncee/wwc/Handbooks. ↑
18. The WWC Standards Handbook, version 4.0 (p. 40; https://ies.ed.gov/ncee/wwc/Handbooks). ↑
19. Results remain statistically significant at the p<0.10 level after correcting for multiple outcomes in the same domain. The Benjamini-Hochberg corrected p-values are 0.002 for credit success rate and cumulative GPA. ↑
20. Note: No students participating in the MAAPS study at the time of analysis had graduated with a bachelor’s degree from Georgia State in the term prior to data collection. ↑
21. Results do not remain statistically significant at the p<0.10 level after correcting for multiple outcomes in the same domain. The Benjamini-Hochberg corrected p-values are 0.122 for credit accumulation and 0.158 for continuous enrollment. ↑
22. The fall 2016, spring 2017, fall 2017, and spring 2018 terms. ↑
23. Note that in year one, impacts on cumulative GPA were observed for students in the bottom half of the GPA distribution, suggesting that MAAPS advisement can contribute to closing the income achievement gap at Georgia State. ↑
24. The academic self-efficacy scale is one of six scales from the College Learning Effectiveness Inventory (CLEI), and was administered and scored accordingly (for more information on the CLEI see http://www.k-state.edu/counseling/topics/cleistudent.html). The grit scale was derived from the work of Angela Duckworth et al, “Grit: Perseverance and Passion for Long-Term Goals,” Journal of Personality and Social Psychology, 9, 1087-1101 (2007). ↑
25. Due to state laws, some sites prohibited minors from entering the gift card drawing and one site prohibited the use of a gift card drawing as an incentive for research participation Students who were not eligible to enter the gift card drawing were not offered an alternative incentive for survey participation. ↑