Students are a notoriously difficult population to recruit for surveys. To combat this, Ithaka S+R has developed a number of strategies to encourage participation in our local surveys of undergraduate and graduate/professional students. Crafting effective invitation and reminder messages, determining when to send these messages, and ensuring that your communications are received and opened are all necessary steps in garnering maximum levels of participation from any population. For our local surveys of students specifically, we have found that the additional motivation of an incentive, such as a drawing for university apparel or a coupon for a free coffee on campus, is especially important for increasing levels of response.

Last fall at the Library Assessment Conference, Joyce Chapman and Emily Daly from Duke University shared new strategies that they recently developed to bolster response rates to a student survey on their campus. There was a ton of interest in and engagement with their session and I left the conference wanting to hear more about these strategies – especially their incorporation of “regret lottery” language. Joyce has been kind enough to share more about the regret lottery with us for the Ithaka S+R blog.

—Christine Wolff-Eisenberg

Background

In 2016, Duke University Libraries (DUL) fielded our biennial student survey. This is a brief survey released to a broad sample of undergraduates and graduate students that gathers feedback on DUL’s services, spaces, and materials, as well as how frequently students use the libraries and their prioritization of future services. The survey effort is led by the Assessment & User Experience (AUX) department within the Libraries. When fielding the 2013 survey, AUX had a difficult time eliciting a satisfactory response rate and was forced to run the survey twice to get approximately 500 respondents. In 2015, we implemented several new methods aimed at increasing our response rate:

  • We planned far in advance with the Office of Institutional Research in order to receive a larger sample of students
  • We provided an incentive to participants (a raffle for a single $75 Amazon gift card)
  • We implemented a “regret lottery” in our recruitment email, and tested it against our regular email recruitment strategy using A/B testing

Understanding regret

The concept of “regret” as we are using it here comes from decision theory. According to decision theory, the feeling of regret is “central to…the human psychology of risk aversion. Conscious anticipation of regret creates a feedback loop that elevates regret from the emotional realm…into the realm of the rational choice behavior.” In other words, anticipated feelings of regret can play a key role when people are making decisions, and the impulse to eliminate or minimize the future feeling of regret affects the decisions we make. Regret and its role in decision-making are studied by behavioral economist Dan Ariely and researchers at the Center for Advanced Hindsight (CAH) at Duke University, which studies, designs, tests, and implements behavioral interventions with the goal of improving people’s health, wealth, and happiness. The CAH has successfully used an approach they call the “regret lottery” to increase the effectiveness of incentives for preventative health care. In one planned clinical trial at a facility with low rates of compliance for health risk assessments, CAH implemented a regret lottery in which there would be a weekly drawing for a $100 prize, but an employee whose name was drawn would be eligible to receive the $100 only if they had already completed their health risk assessment. According to Ariely, regret lotteries work because “our happiness frequently depends not on where we are at the moment, but how easily we perceive we might be elsewhere, or in another, better situation.” Another way to understand it is to think about the lottery. In general, people who don’t buy lottery tickets also don’t feel a high level of regret when they hear on the radio that there was a $130M jackpot. They are removed from the situation: they came nowhere close to winning the money and so it isn’t easy to mentally place themselves in this other, better situation. People who bought a lottery ticket, however, feel much closer to the scenario of having potentially won, and therefore feel the regret more sharply. In the case of a regret lottery, you are causing participants to feel the proximity of “another, better situation” by entering them pre-emptively in the lottery; this encourages them to modify their behavior to avert the anticipated regret.

I was eating dinner with a friend who works as a researcher at the CAH one evening, and told her about the Libraries’ upcoming survey. I wanted to ask her whether from a behavioral economics perspective, an incentive with poor odds for a chance to win a single $75 gift card would get a better response rate than one with increased odds of a raffle with three winners of $25 gift cards (the answer is yes: our brains don’t weigh probability well, only the potential reward!) She went on to explain the concept of a regret lottery and how we might use it to increase our response rate even further. AUX was fascinated by the idea. Having no baseline data for what kind of response rate a traditional recruitment email would get with a $75 raffle incentive, however, meant we would have no idea how effective the regret lottery had been if implemented. We therefore decided to implement A/B testing with our recruitment email to test the success of both strategies.

The A/B test

In order to gather two sets of data on response rates, we split our 6,000 student sample randomly into two even groups. Group A would receive the Libraries’ traditional recruitment email. Group B would receive an email crafted with the assistance of the CAH to produce a regret lottery. Group A’s email asks students to participate in the Libraries’ survey in order to help the Libraries improve services, and as a bonus, informs students that if they participate, their name will be entered into a raffle for a $75 Amazon gift card. Group B’s email begins by informing students that they have already been entered in a raffle for a $75 Amazon gift card, but if their name is drawn, they could only claim their winnings if they had completed the Libraries’ survey. The relevant text from each email can be seen below (note that the emails continued with more information, we have truncated them for brevity).

A: Typical Library recruitment email

“I am writing to invite you to participate in a brief Duke University Libraries Survey.

The survey will give you an opportunity to share your thoughts about using Duke Libraries and provide suggestions for improving current library spaces, collections, and services. Your responses will help us enhance existing services and develop innovative programs.

To thank you for your participation, you will be entered in a raffle to win a $75 Amazon gift certificate…”

B: Regret Lottery

“Your name has been entered in a drawing for a $75 Amazon gift certificate. On February 15th, we will notify you if you have won the raffle, and if so, send you the $75 Amazon gift certificate.

Note that you will only be eligible to receive the $75 Amazon gift certificate if you complete the survey below. If you are the winner of the gift certificate but you have not completed the survey, you are not eligible to receive the $75 Amazon gift certificate.

The survey will give you an opportunity to share your thoughts about using Duke Libraries and provide suggestions for improving current library spaces, collections, and services. Your responses will help us enhance existing services and develop innovative programs…”

Findings

Within the first hour after the survey was released, we received more responses than the entire two runs of the 2013 survey. Five hundred and seventy two students responded to the survey in the first hour, 71% of whom were in group B (the regret lottery). By the end of one week, 1,452 students had responded to the survey and the regret lottery accounted for 66% of these participants. The regret lottery worked so well that we ended A/B testing after the initial recruitment email and implemented regret lottery wording in the reminder emails sent to members of both group A and B at the end of week one and week two.

 

Intrigued by the immense success of the regret lottery, we decided to delve deeper into the question of who has responded particularly well to regret. One of the first questions on the survey asks students to identify the primary library they use; students were then funneled to specific questions about that library. Analyzing response rates by primary library affiliation yielded fascinating results. We found that students whose primary libraries were the Business School Library and the Medical School Library responded almost exclusively to the regret lottery: 99% of respondents affiliated with these libraries came from group B. Students who selected Perkins & Bostock (the main campus library) responded evenly to email A and B – 51% to 49%. Students who primarily use Lilly Library (informally considered the undergraduate student library) and the Music Library responded in greater numbers to email A rather than the regret lottery. Sixty-one percent of Music and Lilly primary users responded to email A and 39% responded to email B. The Law Library did not participate in the survey, so students affiliated with the programs were not included in our sample.

 

Conclusion

Overall, DUL was extremely pleased with the response rates we achieved using combined tactics of an incentive, a regret lottery, and pre-planning with Institutional Research. We received a 43% response rate from the 6,000 student sample provided by Institutional Research, and 945 additional responses from an open URL available on our webpage. In addition to increasing our general response rate, we believe that the incentive and regret lottery also increased responses from a group the Libraries traditionally has trouble reaching: students who do not use our physical spaces. Nine percent of respondents did not use the Libraries’ physical spaces, which is more than we are usually able to reach for feedback.

While a regret lottery may not be a good fit for every institution, we found it to be an interesting experiment that produced favorable results. DUL will likely use a similar or modified recruitment strategy with our upcoming 2018 student survey.