Ithaka S+R is gearing up to field our seventh national US Faculty Survey on the research and teaching practices, perceptions, and needs of scholars at four-year colleges and universities. The questionnaire for this upcoming cycle has been designed to continue tracking critical trends in higher education from previous cycles while at the same time introducing new questions to address issues of current strategic importance.

Having gathered input on our draft questionnaire from our project advisors as well as colleagues internally, we are now in the process of testing our draft questionnaire by employing cognitive interviews for qualitative pre-testing. We do this to ensure that the questions we have written are understood clearly and consistently across respondents in a variety of different fields, institution types, and roles. The following post provides an overview of this process for others to use in their own survey testing, and we look forward to sharing the finalized questionnaire in just a few weeks.

What is cognitive interviewing?

Cognitive interviewing is the administration of draft survey questions while collecting additional verbal information about the survey responses, which is used to evaluate the quality of the response and/or to help determine whether the question is generating the information that its author intends.[1] It is a semi-structured qualitative assessment of the survey instrument.

Why is cognitive interviewing used?

Cognitive interviewing aids the author of the survey in understanding the cognitive processes that respondents use to answer survey questions. It can help to ensure that survey questions have been written clearly and will be interpreted consistently across respondents.

What kinds of participants should be selected for the cognitive interviews?

A small convenience sample that represents the characteristics of the survey population is generally used.

What are the techniques involved in cognitive interviewing?

Interviewers will often use think aloud techniques, in which the interviewer will present the respondent with a survey question and ask the respondent to verbalize their thoughts while answering the question. This assumes that the respondent can articulate their response process and that the process of reporting one’s thoughts while responding does not affect one’s thought process and, therefore, one’s decisions in responding to the survey question.

Interviewers may use verbal probes to understand the interpretation of certain elements of the questionnaire. There are a few different approaches with probing:

  • Concurrent probing: the interviewer presents the survey question; the respondent answers the survey question; the interviewer asks a probe question; the respondent answers the probe question
  • Retrospective probing: the interviewer presents all of the survey questions; the respondent answers all of the survey questions; the interviewer reviews elements of certain survey questions with the respondent after the survey has been completed
  • Proactive probing: the interviewer decides actively on probes in advance
  • Reactive probing: while the interviewer decides on some probes in advance, probes may also be based on respondent answers that suggest a problem

My preference is to begin the cognitive interview by explaining the cognitive interview process to the respondent, including how long the interview will take and what techniques will be used (e.g. “I’m going to ask you to first take the survey that we are testing and answer the questions as you would normally. Once you have finished the survey, I will ask you a series of questions about the survey.”) The interviewer should reassure the participant that there are no right or wrong answers to the survey questions and/or the probes. Finally, the interviewer should ask the participant if they have any questions before they begin.

What are some examples of probes used in cognitive interviews?

Prior to conducting the cognitive interviews, interviewees should identify questions that are potentially problematic in their questionnaire draft and develop probes for each of these survey questions.

  • Overall, how did you feel about the length of this survey?
  • Can you tell me in your own words what this question was asking?
  • How did you decide to answer this question the way you did?
  • Was there a particular experience or experiences that you reflected upon before/while answering the question?
  • What do you think we meant by this in this question?
  • Were there any specific types of this that came to mind when we asked these questions?

After using these probes to conduct the cognitive interviews, interviewees should analyze responses and revise their questionnaire draft accordingly.

While the judgment of the survey author and other subject matter experts is key to the development of a questionnaire, cognitive interviewing is an enormously important step for identifying and fixing problems with survey questions. It adds qualitative depth that helps the survey author understand questions from a variety of respondent perspectives and make adjustments as needed. It’s an essential part of the questionnaire development process that we revisit before fielding each national faculty survey cycle.  

 

[1] Paul C. Beatty and Gordon B. Willis, “Research Synthesis: The Practice of Cognitive Interviewing,” Public Opinion Quarterly, Volume 71, Issue 2, 1 January 2007, Pages 287–311, https://doi.org/10.1093/poq/nfm006