Purpose & Goals

Managing large-scale change is difficult for any organization, but can be particularly challenging for busy and resource-strapped libraries. In our case, the lack of a structured approach to assessing library-wide operations meant that decisions were often driven by anecdotal information and hinged on selective requirements rather than a full accounting of stakeholder needs. As it became clear that a migration may be needed, this project sought to both determine our current Integrated Library System (ILS) needs and build a more robust framework for ongoing assessment and consensus-making. Importantly, our goal was to develop an assessment strategy that would carry us through not just the review and selection process, but could continue post-migration to help support implementation, training, and ongoing system improvements for both staff users and library patrons. This paper will discuss the scaffolded assessment methods and tools that we employed and how they factored into our overall review process and ultimately sparked a culture shift in our ILS review and management processes.

Design & Methodology

The ILS Review Task Force scaffolded a variety of assessments of the ILS over several years, including: surveys, focus groups, workflow analyses, and ethnographic-style interviews. In Phase One, all library staff were allowed to complete a nine-question Qualtrics survey, consisting of both multiple-response and free-response questions. The aim was to identify and garner feedback about the digital tools staff use in the course of their work. In Phase Two, testing consisted of two stages: a patron-facing workshop and workflow-analysis interviews. During the workshop, eighteen staff members created workflows for typical information-seeking tasks using the Libraries’ discovery tools. Participants, broken into three groups, evaluated each process, highlighting potential user difficulties. Before the workshop, participants used worksheets to reflect on witnessed patron issues. In Phase 3, staff participants created workflows using Excel, outling three processes centered on digital resources and three related to analog resources. Participants outlined the actions taken and the ILS module(s) they used. In cases where the ILS was not used, an explanation and the identification of alternative tools were requested. This information was coded and stored in an Airtable database. Finally, the researchers conducted two group interviews during which participants explained their workflows and tool preferences. Interview questions were distributed before the sessions and, to fill any gaps from the sessions themselves, afterward as a questionnaire. The investigation concluded with a feedback survey and analysis of the findings. After these assessments, an RFP committee was formed to take the functional requirements and findings from the analysis and create a detailed list of RFP criteria. Next, standardized usability and functionality tests were created to evaluate system performance and provide a standardized rating for each criteria. This ensured that each system was evaluated fairly and various library functions and staff needs were weighted equally in the scoring and selection process.

Findings

Early findings demonstrated that the library system was both outdated and missing key functions to support a modern, e-focused collection – a situation exacerbated by stark increase in e-resource usage during COVID. The workflow analysis, in particular, highlighted that the emphasis on e-resources dramatically shifted where work was taking place, moving labor from the later collection lifecycle events (i.e. shelving, circulating, weeding) to early lifecycle processes such as acquisition and licensing. Critically viewing the possibilities in new systems under the lens of the needs identified helped staff to investigate potential new systems with more intentionality. Additionally, staff reporting to the Collections and Discovery department were asked to provide reports on how the workflows they had outlined might be changed or improved by migrating to a new system. This analysis proved critical to changing perspectives on the burden of migration – with almost all staff reporting that a new system would improve their work efficiency. Standardizing the way staff assessed the viability of new systems during the RFP process was an important development because the same tests for each criterion were applied to all systems reviewed. Feedback and results were less mired in individual preference and were focused on performance with realistic, typical tasks undertaken by staff. While the ultimate conclusions were heavily dependent on the costs analysis, the technical reviews were critical for establishing the justification for the system selection to the library.

Action & Impact

The scaffolded approach we took produced a lot of data points for our analysis, but also exposed numerous problems with our training, communication and management practices surrounding the ILS. Bringing in ILS vendors to showcase their products afterwards brought these problems into sharper relief, demonstrating what was possible in next-gen systems and generating buy-in and excitement for a potential migration. After initiating the RFP process, this systematic approach was continued during our ILS review and selection process, with multiple opportunities for staff to engage and independently test and rate system performance using standard evaluation criteria. This more objective, consensus-driven process helped mitigate potential bias toward certain functions or user groups within the library, and led to a decision that reflected the broad needs of the library as best as possible. Although ILS migrations can be quite challenging and stressful, if staff had to be put through such pain, it made sense to take advantage of the opportunity to deeply examine our needs and re-think our workflows. Not only did this help ease the implementation process, it also helped library staff know what to expect and how to take full advantage of the system to ensure workflows could not only be adapted to the new ILS, but were actually improved and more sustainable as a result. Moreover, our process sought to include more voices and perspectives into our ILS discussions, including end-user perspectives that were often secondary to internal collection management needs. This is beginning to re-shape the culture around our technical systems, helping the library as a whole understand the role and impact of library technology, and creating a foundation for continued assessment and improvement of library operations more broadly.

Practical Implications & Value

From an organizational decision-making perspective, utilizing process scaffolding helps ensure that prior, related assessment data is woven into current and future library evaluations. The scaffolding approach encourages an assessment-minded culture by establishing a baseline of data-informed needs that staff can build upon. The process has helped our library shift decision making further away from ‘feelings’-based or anecdotal data to systematically-collected assessment data that builds onto itself to form a robust collection of user-informed data. From a methodological perspective, our paper contributes to the limited body of work on assessing for the internal and external needs of ILS. It uniquely incorporates voices from across staff levels and specialties rather than relying on the voices of a few to make an impactful decision like selecting and migrating to a new ILS. When it comes time to choose an ILS, we will have justifications that everyone understands—justifications that are not based in one person’s needs or wants. It was not biased to one group. From a leadership perspective, our methods proved successful in generating substantial—if not total—buy-in for migrating to a new ILS. Consensus building in academic library is extremely difficult when faced with a wide-reaching, impactful decision. Our paper demonstrates that utilizing this assessment methodology was the key to building consensus and collective knowledge building in place of anecdotal decision-making. It brought employees from across the library together to enhance our users’ experience—a process that everyone was willing to get behind. Leadership brought people in as participants in this entire conversation and not as token representatives. Treating people as experts and sources of information—with respect and as the experts that they are—generated a lot of positivity and buy in that will carry them through the pain of the migration process.

View Slides (PDF)