Planning, Partnering, and Piloting

Service Concept Testing

As part of a multi-year student service innovation project, co-led by Northern Virginia Community College and Ithaka S+R, we developed and implemented a new mixed-methods assessment approach: service concept testing.^[1] With participation from six additional community college partners and support from the Institute of Museum and Library Services, we designed, evaluated, and piloted a variety of service prototypes. In this playbook, we describe the services generated and piloted as a result of these collaborations and provide lessons for other institutions interested in developing and testing new services in their context.

Service concept testing is a mixed-methods market research process, guided by a participatory decision making framework, and entails gathering data on user needs, generating possible offerings, and testing those offerings. In our implementation of the methodology, we began with a qualitative discovery phase examining existing data sources and interviewing students. This was then followed by a service concept development phase, which involved collaborative brainstorming for specific service ideas that addressed the challenges and needs uncovered in the first phase. Lastly, we assessed these concepts by surveying the student community to gauge the services’ potential value. Representatives from each of the seven partner community colleges, referred to as “partner college” or “project lead” for the remainder of this playbook, held key roles in each step of service concept testing.^[2]

Students’ self-described goals and challenges, as well as experiential evidence from the partner colleges, significantly guided our creation of eight “service concepts”—hypothetical prototypes of services that colleges might offer to students for specific purposes relevant to the goals, challenges, and/or needs expressed by students themselves. Based on the results from the assessment phase, the partner colleges were able to make data-informed judgements on services to implement at their library based on both aggregate (cross-institutional) and local (institution-specific) results.^[3]

Toward generating this playbook, we held semi-structured interviews with project leads in January 2020 following each college’s first semester or quarter of piloting a new or adapted service resulting from this project. In these interviews, we asked about the colleges’ service development processes, implementation strategies, and outcomes. We also asked the project leads to share lessons learned from the pilot implementations and their recommendations for other institutions considering using a similar process to develop services on their campuses. Although these services were implemented within the library, any unit on a college campus can utilize the service concept testing methodology as well as the lessons learned from this piloting process. In this playbook, we synthesize and summarize both the services themselves as well as key takeaways from the process of implementing those services, offering strategic insights gleaned throughout the initial implementation.

Service Concepts in Action

Partner colleges often implemented one or two of the eight service concepts, selecting which services to employ based on a combination of (1) students’ valuation of the service concept, (2) their colleges’ strategic initiatives, and (3) available resources. The following section describes how the service concepts—which served as prototypes of services rather than services themselves—were translated into actual services. Some of the resulting services derived from a single service concept are quite similar to one another, while some are very different, signaling the extent to which college strategic initiatives and available resources—along with mixed-methods research findings—drive the service implementation process.

Table 1. Partner colleges that piloted a service based on each concept

Loaning Tech: access to technology, either to borrow for use outside of class and home, or on-site with expert training and assistance

Through our survey, we asked students how valuable they would find borrowing a variety of technological resources, such as laptops, scientific calculators, Wi-Fi hotpots, black/white or color printers, and more. Overall, 42 percent of students across the partner colleges rated this service as extremely valuable, and of those students, eight in ten rated Wi-Fi hotspots, black/white or color printers, laptops, and multimedia editing computers or software as particularly useful to borrow or use. Roughly half of the partner colleges were already loaning many of these specific resources prior to service concept testing. Based on this new evidence they decided to adapt and expand their existing services:

• Queensborough Community College (QCC) extended the loaning period of laptops from two-hour time slots to a week-long period. With the help of the Information Technology (IT) department and Vice President of Finance, the project lead was able to refurbish and recycle laptops returned to IT by academic faculty—acquiring 24 laptops newly available to loan.
• Northern Virginia Community College (NOVA) librarians collaborated with their Financial Stability Office to lend iPads with keyboards and wireless mice and laptops for an entire quarter to students with demonstrated need.
• Pierce College District (PCD) plans to expand their loaning tech service, which currently provides Chromebooks, laptops, and hot spots for students to borrow.  Additional computers have been added to the quarterly loan program and alternative hot spot vendors are being considered. They are also providing information about low-cost technology programs to students who apply for quarterly loans.
• Monroe Community College (MCC) is now seeking additional funding for more laptops and other types of technological resources to loan.

Personal Librarian: a professional employed by the college available to help students find and use all kinds of content sources needed for coursework, including books and journals, in paper, and on the internet

Across the seven partner colleges, students rated this service concept as the third most desired, with about four in ten rating this service extremely valuable. Four partner colleges decided to adapt this service:

• LaGuardia Community College (LAGCC) has academic subject librarian liaisons working with Academic Affairs and IT to provide specialized outreach and support, enabling students to contact their subject liaison with questions and requests via an online platform, as well as allowing librarians to reach out to students. As LAGCC’s Academic Affairs office stratifies students based on their support needs, the service is initially targeting students who need the lowest level of support.
• PCD identified an opportunity for integrating a personal librarian service into specific general education English course tracks where there was a completion gap between the first two levels of the course. They are currently planning how to best scaffold information competency instruction alongside personal contact points.
• Borough of Manhattan Community College (BMCC) also has librarian liaisons performing specialized outreach and support to students, collaborating with Student Affairs to provide this service to male students who fall below a certain threshold on their high school records.
• NOVA identified two groups to target this service: students within the honors program and those within a specialized transfer program. Librarians provide research consultations and on-the-ground support within classes to honors students as well as similar support services outside of the classroom for students within the specialized transfer program.

Child Care: an array of services at a college for parents or caregivers of children while they are attending class, which might include designated spaces for families to study together, both regular and emergency childcare programs, and tutoring and other afterschool services for children

About half of student parents or caregivers across the seven partner colleges rated child care services as extremely valuable. Two partner colleges are piloting related services:

• PCD created child activity kits for children of different ages, filled with games, toys, and educational books at different reading levels to occupy children with as their parent or caregiver studies.

Children activity bags, Pierce College District. Photo by Kathryn Scott

• MCC dedicated spaces at each campus library for family-friendly study rooms. These rooms were remodeled with new, brightly colored and comfortable furniture, desks, and educational toys and books for children. The college is also collaborating with their local YMCA to provide free drop-in childcare this semester.

Family- Friendly Study Room, Monroe Community College Brighton Campus, LeRoy V Good Library

Knowledge Base: a single point of contact for whenever a student needs help navigating any part of college including advising, registering for classes, applying for financial aid, securing personal counseling, and obtaining tutoring or other coursework assistance—offering expertise in connecting students with the right college employee for assistance

Knowledge base was the most highly valued service concept, with 44 percent of students rating this service as extremely valuable. Despite the popularity of this service among respondents, getting this service off the ground requires substantial startup resources, including funding, time, and staffing. As a result, just two colleges are currently piloting a knowledge base service:

• Bronx Community College (BCC) hired a non-teaching adjunct to collect data queries received at their information desks to improve the training of relevant personnel as well as their directory of campus and community services.^[4] They are aiming to collaborate with other service providers on campus to further streamline their knowledge base service as it continues.
• Prior to this initiative, LAGCC already had a robust, online knowledge base, branded as Ask LaGuardia, monitored and managed by a librarian.^[5] Ask LaGuardia provides prewritten answers to common questions students ask—such as how to change majors, how to set up an advising appointment, how to register for classes, etc. For their new pilot, LAGCC invested in promoting Ask LaGuardia to new students.

Screenshot of AskLaGuardia, an existing knowledge base service

Community Advocacy: an opportunity to help students better develop their capacity as a member of their community and society through one-time presentations, workshops, ongoing discussion groups, informal meetups or online resources by faculty, members of community groups, industry experts, or fellow students

Although only two in ten students across the partner colleges rated community advocacy as extremely valuable, one college decided to pilot this service:

• PCD implemented scheduled conversations and panel discussions on the pressing challenges students face, including food insecurity, on the last Wednesday of every month, open to students, faculty, and other individuals in the community.

Community advocacy panelists on food insecurity, Pierce College District. Photo by Elissa Blankenship

Strategic Insights

The pilot services outlined above are very much tied to the strategic priorities of the partner colleges, with varying levels of coordination and support from leadership and peer colleagues. Lack of time, funding, and human resources, as well as difficulties with student outreach efforts, presented challenges to the project leads who nevertheless found unique ways to utilize existing relationships and resources to execute their pilots. In the following section, we offer a series of high-level strategic insights from the experiences of and reflections from the project leads.

Align service implementation with strategic initiatives and existing service models

Grounding service conceptualization, development, and implementation within strategic initiatives and existing service models can be an effective way to gain approval from leadership and help meet user needs.

Naturally, project leads incorporated the strategic initiatives of their college as well as existing service models provided by their own library, college, or other institutions when initially brainstorming the list of service concepts for subsequent testing via survey. Grounding services within their colleges’ strategic initiatives made it so that none of the services were provided—or even tested—arbitrarily. Additionally, mapping services to existing models either at their own college or at another institution provided a foundation to reinvent services and continuously meet user needs.

Approximately half of the project leads chose to implement service concepts that most aligned with the strategic initiatives of their college, even if students did not rate those service concepts most valuable. For two project leads, aligning with their college’s strategic priorities made it easier to get leadership approval and assistance. Additionally, four of the partner colleges already had been loaning laptops and other devices to students, but were able to expand and adapt this existing service based on new evidence, keeping their existing service sustainable.

Provide meaningful engagement opportunities for faculty and staff early on in service development

Incorporating faculty and staff perspectives into service development can help secure their support and streamline the provision of new services during implementation.

Some partner colleges experienced resistance to implementing new services from their library faculty and staff, who expressed concerns about potentially winding down other services traditionally offered by the library in favor of new services. Faculty and staff were also worried about their own capacity and the amount of work adding new services would entail. Generating early awareness of service development and subsequent plans for implementation can potentially lead to useful conversations with faculty and staff to address their concerns prior to piloting.

Project leads whose library faculty and staff were able to provide input earlier in the process often had a relatively easier time implementing their pilot(s). Since it will fall to these individuals to provide new services to students, soliciting meaningful engagement helps to secure their approval and streamline implementation. For instance, one college that piloted a version of the personal librarian service concept was able to target their service towards ameliorating completion gaps within their general education courses because early on a librarian began researching external examples of the librarian liaison model and opportunities to target completion gaps in tandem with the assessment phase of the project. Through this librarian’s enthusiasm and research, this college was able to target specific courses when they piloted this new service.

Use multiple data sources as leverage—even if some aren’t specific to your college

Build your case over time for providing new services using multiple empirical and experiential data sources available internally or externally.

It is important to know students both anecdotally and by generating and examining empirical data. Through student interviews at the outset of service concept testing, colleges were able to understand the student experience more deeply and develop a suite of possible services corresponding to students’ expressed needs and challenges. Incorporating the student perspective throughout service development is essential to providing assistance that will best suit their unique needs and challenges.

Additionally, students may express their needs directly to service providers in a way that interviews or surveys may not capture; they may, for example, see or hear about a service offered at another college and bring that idea to a librarian or faculty member. At one of the partner colleges, a student suggested their library obtain a laptop vending machine—something the student had seen at another college. Although the partner college did not buy a vending machine, this suggestion provided essential qualitative evidence for later provision of a loaning technology service for laptops.

The ability to survey students and rate potential services against each other not only provided for easier decision-making, but the ability to demonstrate that students needed these new services. Although each partner college has its own institutional effectiveness and assessment processes and offices, many of the project leads found data collected through service concept testing to compliment those data and bolster their decision-making process. Many of the project leads were able to triangulate data from service concept testing with data from outside the project to demonstrate need to their administrators and boards.

Scope small, for a couple of semesters or quarters, and scale from there

Targeting subgroups within the student body, and piloting services beyond a semester or quarter, can be an efficient strategy to serve students while addressing potential high demand and limited resources.

Initially piloting a new service at full scale to all students may not be feasible due to limited resources—financial, personnel, or otherwise. If anticipated demand is high, the thought of implementation may be daunting and can deter execution. Two project leads in particular were concerned about implementing highly rated services—potentially overwhelming their already limited resources. However, as one project lead mentioned, “you should never not do something because you’re worried about demand [being high].”

One college combatted concerns over high demand with a “soft launch” of their loaning laptops pilot. This involved announcing the service informally to students and faculty, but not embarking on a widespread outreach effort, ensuring that those who needed to borrow laptops were able to. A soft launch may also help demonstrate need for new technological resources to loan; usage data from a small-scale, initial implementation can provide additional leverage to obtain new devices, increasing the scope of these programs over time based on level of demand.

Offering targeted services to specific student subgroups can be another helpful way to address these concerns, while also reducing equity gaps. Across colleges, we found that students who are historically underserved or disadvantaged—students of color, student parents, and low-income students, among others—often report encountering greater challenges and demand for support services.^[6] After testing service concepts via survey, partner colleges were able to disaggregate their data and determine which students saw the most value in, and need for, different services.

Many project leads chose to initially target their pilot services towards subgroups of students and are now able to adjust outreach and implementation strategies to scale up their programs. For example, one college targeted male students with high school grades below a certain threshold, another targeted honors students and those in a specialized transfer programs, and one targeted subgroups of students based on their level of needed support.

Further, at the outset of implementation, it’s important to ensure that the pilot is planned for more than one semester, quarter, or year, and that a strategy has been created in advance for pivoting and adapting the service if it is not successful within the first few semesters or quarters. Piloting for the long term, while collecting data and assessing use and efficiency along the way, is critical for determining value, what to change, and how best to scale and reach more students. For some services and institutions, it can also take substantial time to build awareness and partnerships to enable the success of the service. One college mentioned that despite low usage within the first semester of the pilot, they are intending to pilot and adapt their service for multiple years in order to further develop and scale the service to serve students more effectively.

Collaborate with administrators and other departments

Utilize existing tools, resources, and relationships with other departments and external vendors, especially when resources are limited.

Collaborating with other service departments can be an effective way to pool resources, avoid duplicating services, and amplify outreach to serve more students. Early buy-in from leadership can also be an effective way to seek new partnerships. Administrators may be able to see new opportunities for collaboration, find resources elsewhere at the college in ways that others cannot, and promote the service to other departments on the library’s behalf. Generating awareness of new services to student and academic administrators, provosts, deans, or those in similar roles can be a great support when gathering resources for new services.

One partner college was able to use their support from administrators effectively to generate collaborative relationships with other departments: they worked with the library and IT to refurbish recycled laptops, resulting in 24 new laptops for their loaning tech program. Another partner college was able to work with their Financial Stability Office to locate students with demonstrated need.

Consider the start-up and long-term costs of implementing a new service

Occasionally, larger start-up costs are preferable especially if the costs of sustaining the service are low. If possible, consider external sources of funding to bolster your budget.

Considering the start-up and long-term investment necessary to sustain a service is key. Project leads often faced challenges, including insufficient staffing, time, and funding. Scoping small, utilizing existing tools and resources, as well as collaborating with other departments can be an efficient way to utilize already limited resources. Considering other institutional funding opportunities or external funding, such as grants or sponsorships, can also be an effective way to generate large start-up costs. Although the grant from this service innovation project helped to cover many of the partner colleges’ personnel costs, they had to find additional resources within their own budgets to pilot their services; some are applying for, or are currently waiting to hear back from, external grants to adapt their pilot services and implement additional ones.

A larger start-up cost can be preferable if the long-term cost of implementation is minimal. One of the partner colleges anticipated large initial costs of designing and furnishing their new family-friendly library spaces, but expected the long-term financial, time, and staffing costs of sustaining these spaces to be minimal. Another college hired an adjunct to expand their existing knowledge base and to help train existing faculty and staff on answering college navigational questions. This saved staff time in the long-term as they did not have to find needed information on their own. Additionally, one of the partner colleges devoted more time and resources to promoting their already existing knowledge base, so that staff, rather than answering recurring questions, can devote more time to students with more complicated questions and needs.

Reach out to academic faculty when marketing; direct-to-student marketing may not always be the most effective choice

Student outreach may present unique challenges depending on campus culture. Reaching out to faculty and informing them of new services can generate greater awareness and use.

Marketing new or adapted services is essential, especially if the service is newly introduced and conceptually unfamiliar to students. However, students may react differently to various communication strategies, and outreach to specific subgroups may present unique challenges. One project lead said that the library and other departments are having difficulties engaging students via email; while the number of students opening emails is high, the number that engages with those messages by then using a related service is low. Another partner college had difficulties gathering contact information across different campuses, hindering their direct outreach efforts.

It may be helpful to reach out to academic faculty, inform them of new services, and ask for their assistance with outreach. Faculty are in a unique position to identify students in need of added support and can assist in marketing library services. For example, one college who did not initially market to faculty found that faculty wanted more notice of new services at their library to be able to inform their students. Another partner college is winding down their direct-to-student email outreach and will instead inform faculty of their program in the hopes that this outreach will result in greater engagement. Overall, direct-to-faculty marketing can potentially reduce the amount of time, money, and personnel spent on outreach, and can ensure that information is circulated to students effectively.

Define metrics of service success before implementation

Collecting usage data is important when assessing the success of service implementation, but other metrics can also help assess success.

When we queried the partner colleges on how they were going to demonstrate impact or measure success of their new services, many said they would rely on usage statistics generated throughout the duration of the pilot. One project lead is planning to issue a student satisfaction survey a year after their pilot launches. Another college is working with their institutional research (IR) office who are collecting similar student success data.

It is important to be strategic about data collection at the outset of piloting to inform next steps, potentially including scaling up or winding down the new initiative. Other data metrics that can be used to measure the success of a service or program may include anecdotal experiences, expenditures over time, money saved over time, and student post-service outcomes. To collect these data, it may be useful to partner or collaborate with IR or IT to leverage their expertise in this area, especially when determining post-service outcomes.

Guiding Questions

If you are working on implementing a new service at your institution, the following questions, paired with the strategic insights above, can be used to begin brainstorming for service development and piloting plans. These questions can be used for reflection individually or collectively as well as within or across departments.

1. What are your institution’s strategic priorities? How can new services bridge the gap between unmet user needs and these priorities?
2. What data exist and where are they located to supplement existing data on your student body and support needs?
3. Are there any subgroups of the student body that can be targeted within the development and/or implementation of services to address equity gaps?
4. What similar services exist at other departments? Are there opportunities to utilize existing relationships and collaborate to more effectively support students?
5. Who has the power to advocate for new services at your campus? What administrators can be made aware of service implementation plans and potentially facilitate cross-departmental collaboration?
6. What resources are currently available for new service implementation and what still needs to be obtained? What does the scope of human resources look like in the immediate future for commencing implementation?
7. How can new services fit within your department’s current workload? What other services – if any – can be scaled down or removed to provide new services?
8. What outreach strategies have or have not been effective in the past? What can be changed to ensure that students are aware of new services and can effectively utilize them?
9. What metrics will help to assess the success of newly piloted support services? What resources are needed to collect and analyze these data, as well as respond to the results?

Endnotes

1. For more in-depth information on service concept testing, see Christine Wolff-Eisenberg and Roger Schonfeld, “What Do Our Users Need? An Evidence-Based Approach for Designing New Services,” Ithaka S+R, 6 November 2019, https://doi.org/10.18665/sr.312092. ↑
2. We would like to thank the project leads at the participating colleges: Jean Amaral, Borough of Manhattan Community College; Braddlee, Northern Virginia Community College; Christie Flynn; Pierce College District; Jeanne Galvin, Queensborough Community College; Katie Ghidiu, Monroe Community College; Michael Miller, Bronx Community College; and Steven Ovadia, LaGuardia Community College. ↑
3. For more information on the aggregate results from the survey assessment phase, see Melissa Blankstein, Christine Wolff-Eisenberg, and Braddlee, “Student Needs Are Academic Needs,” Ithaka S+R, 30 September 2019, https://doi.org/10.18665/sr.311913. ↑
4. Because the IMLS grant for the service innovation project aided with personnel costs, some colleges were able to hire additional staff to help execute their pilot services. ↑
5. For more information on this existing service, see: https://laguardia.intelliresponse.com/. ↑
6. Melissa Blankstein, Christine Wolff-Eisenberg, and Braddlee, “Student Needs Are Academic Needs,” Ithaka S+R, 30 September 2019, https://doi.org/10.18665/sr.311913. ↑