Over the last several months, we’ve shared Ithaka S+R strategies for ensuring inclusivity in writing and selecting demographic questions—key steps in the early stages of the data collection planning process. But inclusivity concerns don’t stop once data collection has begun, but continue when a survey is put into the field, an interview is conducted, or administrative data collection is facilitated. 

Ensuring proper care in collecting and reporting on identity-based data—or failing to do so—can have both short- and long-term consequences. Most immediately, the ways in which data are collected can help current research participants feel secure that their responses will not lead to any unanticipated negative consequences. And in the long-term, how data are reported can impact the willingness of prospective participants to engage in similar efforts in the future. Today we share three strategies that researchers at Ithaka S+R have embraced for responsibly and inclusively analyzing and reporting on data that deal with participant demographics.

Protect participant anonymity and confidentiality  

One major consideration in collecting and reporting demographic data is whether the data collection initiative is anonymous or confidential. Many surveys, for example, are either anonymous or confidential depending on whether identifying information such as names and email addresses are collected during the implementation of the survey. When these identifying variables are collected, the survey can be confidential, and when they are not, the survey is typically anonymous. 

When anonymity or confidentiality are promised, researchers are obligated to report data in such a way that respondents cannot be reidentified by the audience. This generally means only reporting aggregated data, assigning pseudonyms to quotes, and reporting subgroup data when there are at least a minimum number of respondents. In our research, we typically use a rule of thumb of 10 respondents for subgroup analyses. Thus, even when just reporting descriptives such as percentages or averages without running statistical inference tests (which generally require more participants), it is still important to keep in mind how big or small the subgroups of respondents are. 

If results are reported with fewer than 10 respondents in a given subgroup, readers familiar with the population could be able to guess how particular individuals responded to the survey. Even if their guesses are inaccurate, such guesses could have real impacts on survey respondents. For example, this is particularly important to consider when conducting employee engagement or staff climate surveys. If results are reported for subgroups with fewer than 10 respondents, readers may be able to or try to guess who responded in a particular way and may make assumptions about their colleagues. It is the researcher’s responsibility to make sure this does not happen especially if the survey is intended to be anonymous or confidential. 

Ensure sufficient statistical power

Another major consideration in analyzing subgroup data—that is, differences between demographic groups—is whether the statistical inferential test—such as t-tests, ANOVA, and regression analyses—will have enough power to detect real differences between groups. If not, it might appear that two or more groups are the same when they really are different; this is also known as Type II error in statistics or a “false negative” more broadly. Among the factors that influence statistical power are the number of participants in a given group. Researchers often use a rule of thumb that statistical analyses should only be performed when there are at least 30 participants in each subgroup. For example, there would need to be at least 30 women, 30 men, and 30 non-binary people who responded to the survey to include all three in a statistical test. 

There are several steps that researchers can take when there are not enough participants in a given subgroup to achieve enough statistical power. If it is important to the hypotheses of the study to examine subgroup responses, descriptive statistics (percentages, averages) can still be computed and reported. One caveat to this is that we generally still recommend that the subgroup has at least 10 respondents. Another possibility is to analyze differences between the groups that do meet the 30 participant threshold, either excluding the subgroup or grouping it with others. For example, given the overwhelming whiteness of the academic library field, in our surveys we often do not have enough respondents in each of our racial-ethnic subgroups to analyze differences across all groups. Given the significance of the Black Lives Matter movement to our recent national library director survey, we specifically ran analyses looking at differences between Black (including bi- or multi-racial people who selected Black as one of their identities) and non-Black directors (grouping all respondents who did not select they were Black) because both of these groups met the 30 participants threshold. Comparisons between white directors and directors of color were also reported.  

Translate between the research instrument and reporting with care

Data must not only be collected with care; reporting that information responsibly is just as important. Given that it is not always possible to report the findings of each subgroup, it is often necessary to do translational work between a survey and the shared results. This may include grouping two or more subgroups together or excluding groups entirely. 

When combining groups, it is necessary to consider what terms are accurate. For example, does a combined group include all people of color? Are Black and Indigenous respondents centered? In these scenarios, “people of color” and “BIPOC” would accurately describe each of these combined groups. If only a small number of subgroups are included, it is important to describe the subgroups with greater specificity (e.g. “American Indian and Alaskan Native and Asian or Asian American respondents”) when discussing findings. With a larger number of subgroups, naming all subgroups can become unwieldy, so more general terms can be useful. Regardless, keeping up to date with evolving terminology—in particular, the terminology preferred by members of the group in question—is essential.

Even when subgroups are excluded from analyses or rolled up into larger groups, including an appendix in a long-form report that displays the number and percentage of respondents from each subgroup can provide readers with a more granular sense of the makeup of the sample. In other forms of reporting such as executive summaries or memos and presentations where such a table may not be practical, calling out the most common subgroups—for example, stating that the majority of respondents were women—can be useful. Since aggregate data are not tied to other survey responses, sharing it does not violate the anonymity or confidentiality of respondents. Rather, it demonstrates the diversity of respondents. 

Even when tracking the diversity of a population is not central to the purpose of a study, sharing aggregate demographic data can be useful for others interpreting survey results. For example, if most respondents of a given survey are white, providing this information to readers can help explain particular results. Further, if comparisons are made grouping together all people of color, but most non-white respondents are from one particular racial group (e.g. Hispanic, Latino, Latina, or Latinx), readers can interpret any group differences to be largely due to differences between these groups of respondents (in this example, white and Hispanic, Latino, Latina, or Latinx respondents). This information can also be used in conjunction with population parameters to determine the representativeness of the sample. If the sample is not representative, the results may overrepresent the perspectives of particular subgroups of people.

Concluding thoughts

Ensuring the inclusiveness of demographics questions does not end at the creation of a data collection protocol. Instead, it is important to consider inclusivity across the entire research process from developing hypotheses to reporting results. Data analysis and reporting practices benefit tremendously from proactively planning on confidentiality and anonymity, analytical approaches, and subgroup reporting. 

 

This blog post is the third and final post in a series on our demographic questions related to individual identity. The first and second blog posts in this series included recommendations for writing and revising demographic questions and choosing when to ask or not ask demographics questions. We welcome your comments, questions, and reactions to these strategies and our sample questions in the comment box below or via email at jennifer.frederick@ithaka.org.