In October 2018 we had the opportunity to engage with attendees to at the Digital Library Federation (DLF) conference about fostering strategic approaches to digital library assessment. During the session we briefly presented a framework for thinking through various user research avenues and related methodologies (suggesting that the user survey isn’t the only useful tool libraries could employ even if we use them a lot!) and then invited participants to consider what they wanted to work on in these areas. As session participants described their projects, we offered guidance on developing or improving the research approaches attendees were  considering. This ideation and coaching session enabled attendees to consider how user research design should always have an underlying strategy to work effectively. We were heartened by the engagement of those participating in the session and the number of follow-up conversations we had throughout the conference.

Based on our experience at DLF we share here a three-stage framework for strategically designing user research to align with library priorities. We also provide some concrete examples of how these considerations shaped the decision making behind the design of specific user research projects. In doing so we hope to spark further dialogue on how strategic approaches can be built into user research.

1. GOALS: Derive Your Inquiry Goal(s) from Library Strategic Priorities

There are a variety of approaches to user research and the choice of which inquiry approach(es) to take hinges on strategic needs and priorities. For example, you might ask:

  • Do you need to evaluate a pre-existing service towards its improvement or articulate that service’s value to the provost?
  • Do you want to make a few improvements to a library space or re-imagine the concept of the library as place?
  • What new service would meet a given population’s unmet service needs?
  • Who else do you need to engage with to make these decisions?

The above questions demonstrate how the inquiry mode determines what strategic goal(s) can be accomplished through research. For example, an assessment approach to inquiry involves evaluating the efficacy of specific tools or services towards identifying opportunities for their improvement or seeking to measure the impact of services on user outcomes. In contrast to the evaluative approach taken by assessment models, inquiry can also be structured as a more exploratory endeavor when less is known about a phenomenon or it is important to understand cultural underpinnings (e.g .ethnography); as an experiment when the need is to compare or test (e.g. A/B testing); or, more explicitly as enacting design processes when the goal is to create or improve a specific space, tool, or service (e.g., participatory design exercises).

There are strategic benefits and trade-offs to any inquiry approach. It is also important to recognize that, in library contexts, certain approaches to inquiry are more common than others due to the nature of the field as applied and service-oriented. Assessment and various user experience approaches have stronger representation (including robust conference cultures), exploratory research can be valuable but is much rarer, and some forms of experiments are virtually non-existent due to the potential harm of denying certain essential library services to a test control cohort. Within library assessment research there is an uptick in interest in the kind of assessment that measures the broader impact of library services, which reflects administrative priorities for using this kind of data to communicate the value of the library to institutional stakeholders.

2. METHODS: Select the Appropriate Method(s) for Your Inquiry

Selecting the appropriate method(s) is essential to bolstering the approach to inquiry. Questions here include:

  • Do you need to capture someone’s perceptions of a service or how they actually are using it?
  • Do you need to know how often they are using that service or why?
  • Would it be helpful to observe patrons using the service in real time, test out a simulation of that service, or only consider their perceptions?
  • How much data do you really need to collect to answer your questions?

When thinking through these kinds of methodological questions, we find it helpful to focus on four dimensions: 1) attitudinal/behavioral, 2) quantitative/qualitative, 3) context of use, and 4) the relationship of the researcher with participants. The first three of these dimensions are extrapolated from this helpful schema and the fourth emanates from participatory-oriented approaches to research, with Indigenous methods being particularly informative for conceptualizing this dimension.

The fourth dimension arguably supersedes the first three dimensions because from it cascades all other methodological choices. Hegemonic Western research typically structures research methods around the premise that distance between the researcher and research participant is optimal because it leads to “objective” insights. When designing this kind of research, the researcher is not necessarily expected to develop (and may even be critiqued for) ongoing relationships with their participants, incorporate their feedback, or credit them as co-creators.  

Post-modern and non-Western approaches to research, such as found in Indigenous Studies and some branches of feminist research, typically challenge the concept of objectivity and the value of insight derived from manufactured distance between researcher and research participant(s). These approaches place greater emphasis on building long-term, mutually beneficial relationships that attend to how participants’ insights are incorporated and credited, with the end result being that far more agency is given to the participants over their contributions. It is always important to consider how agency can be given to participants but also important to recognize that there is no one size fits all approach to how that can be done. The Ithaka S+R Indigenous Studies project provides one example of how researchers can build greater attention to relationship building and participant agency into all aspects of user research methodology, including the structure of the interviews, review of transcripts, and deriving insight from the findings, among others. Some of the researchers’ initial reflections on their experiences participating in the project and initial findings can be found here.

For many projects it is necessary to combine approaches from the other three dimensions in order to ensure that the results are sufficiently robust to provide illumination to your inquiry. Here are a few examples from Ithaka S+R’s work. For the business project, researchers are asking business faculty behavioral and attitudinal questions to gain a contextualized understanding of teaching practices and support needs in this field. In a project with Montgomery College, students were engaged in a variety of contexts so that the study could capture what they currently do in the library and their unique perspectives on the future of library services and spaces. And finally, the IMLS-funded Community College Academic and Student Services Ecosystem (CCASSE) project will incorporate qualitative and quantitative approaches to develop the fullest possible picture of the academic support system landscape at community colleges.

3. ACTIONS: Communicate Results and Commit to Implementation

Having results from your goals-aligned user research is not enough to ensure that they will be used strategically. It is essential that these results be communicated effectively and garner commitment to acting on them. Questions to consider include:  

  • Who would benefit from learning about your research and its findings?
  • Who should be be convened to identify and develop next steps?
  • What processes will ensure that the next steps can be effectively identified and acted upon?
  • Who has the authority to act on the results?

There are a variety of ways that results can be effectively communicated and acted upon as part of a research process and it is essential that these processes are built into the overall plan for the research to ensure the research is effective. For example, for the IMLS-supported Community College Libraries and Academic Support for Student Success (CCLASSS) project that Ithaka S+R is involved with, the participating community colleges are not only contributing data but will also participate in the design, evaluation, and iteration of new services as part of the project. After participating in the Ithaka S+R Asian Studies project, Lafayette College used the findings on their scholars’ research support needs to design a new library event series for scholars to share their work and foster connections between researchers.

At Illinois, a locally-directed project to conduct a meta-analysis of a decade of user surveys, informed by data from focus groups,analysis of search logs, and review of virtual reference transcripts, resulted in a set of principles for the library’s discovery service. The discovery team was involved in conceptualizing the research project and reviewing the results after the analysis was completed. The resulting principles have been referenced in user interface design and system development discussions and decision making.

Final Thoughts

We recognize that libraries will need to engage these three stages iteratively, not linearly, but believe that abstracting the process into phases can help clarify the different considerations that relate to different aspects of user research. We look forward to hearing how others are working to ensure that their user research approach fits their inquiry goals and leads to actionable results. We wonder about how the goals, methods, and actions involved with user research overlap or vary based on the aspect of the library is under inquiry and who is leading the investigation. In particular, it is still unclear to what extent unique approaches are necessary when researching “digital libraries” and/or digital components of a library, and how others in the digital library community are approaching these issues. As “digital” has become nearly indistinguishable from other library offerings, it is important to continue to consider how and whether pre-existing library user research methods can be mapped onto current library contexts.