Purpose & Goals
The Sperber Health Sciences Librarians have regularly offered an intensive workshop on the topic of systematic review searching skills throughout the years. This workshop has taken several forms in terms of length and mode of delivery; at the time of this project, the workshop was delivered virtually as four 1-hour synchronous sessions offered on consecutive days in a single week. To accompany the synchronous workshop, asynchronous modules covering the same content were also available to learners. The workshop teaches researchers, usually graduate students from various health sciences faculties, how to design, execute, and report a robust, comprehensive search strategy appropriate for publication in a systematic or scoping review. In spite of offering the workshop on a near monthly basis, reaching approximately 900 learners per year, and having the online asynchronous modules, librarians still conducted a large number of one-on-one consultations with researchers completing systematic review searches. These consultations often covered nearly identical content to what was offered in the workshop, making them repetitive and time consuming as a single consultation. Additionally, these consultations seemed unproductive since there was not always sufficient time to include the design of the search strategy for the review during a 1-hr appointment, leading to multiple consultations for a single project. The goal of this assessment project was to three-fold: 1. Determine how well students were learning the foundation skills required to conduct the initial steps of a systematic search as part of a larger systematic or scoping review 2. Determine the delivery preference for learners 3. Understand the knowledge gap between the theory of advanced searching and the application of this information to complete a systematic search
Design & Methodology
This was a collaborative project of the recently formed Assessment and Insight Team (AIT) and the Health Sciences Librarians at the University of Alberta Library. The project was led by two members of the AIT who worked with two of the Sperber librarians to create an assessment strategy. This project took seven months in total to implement and complete. A literature review was conducted to determine the best approaches for conducting an assessment of learning while still making the workload manageable, and therefore the project feasible. The assessment strategy was survey based and comprised of three parts: 1. A post-test style survey was used to assess the learning of researchers who had taken the workshop in the three months prior to beginning this assessment project (August-October 2022). This survey included "test your knowledge" style questions as well as questions about demographics and mode of delivery preferences 2. In class problem-based learning style quizzes were delivered at the end of the daily workshop sessions and data was collected to determine which concepts learners were struggling with and which were clear. These quizzes took approximately 5 minutes for learners to complete and were promoted as a good way for learners to test their own learning. 3. A follow up survey was distributed to attendees (Jan-March 2023). The survey asked questions about demographics, mode of delivery preference, and "confidence" with skills taught at the workshop. These "confidence questions" were adapted from Fresno Tests and "researcher-readiness" assessment literature Surveys were distributed using the survey software, Qualtrics. Daily quizzes were implemented using Google Forms. Additional assessment was completed for the in-person version of the workshop that existed in the years previous. However, this assessment was limited mostly to "satisfaction" with the workshop rather than if the attendees learned core skills.
Findings
From the daily quizzes, we found that the responses demonstrated a fairly strong understanding of the concepts. There were two questions with a notably high number of incorrect answers: one question related to the inclusion of subject headings (MeSH) in the search strategy and the other was about selecting the most appropriate review methodology for a research question. This was not unexpected as Sperber Librarians frequently discussed both of these concepts during one-on-one research consultations. From the surveys, we found that most attendees were graduate or post-doc students, or research associates/staff from various Health Sciences Faculties. Learners preferred virtual synchronous instruction as the mode of delivery. Additionally, shorter (1-hr) sessions spread out over a single week were noted to be preferable as these were easier to fit into learners' schedules. Many respondents mentioned that they preferred to have learning materials (such as slides and handouts) made available ahead of time and that the workshop be recorded. Interestingly, when asked if they had reviewed the asynchronous modules available to accompany the workshop, nearly half said they had not reviewed them and the other half said they did not know these modules existed. The majority of survey respondents responded positively to questions about confidence with skills taught during the workshop.
Action & Impact
Based on the data collected for this project, the following recommendations have been implemented: - Maintain virtual synchronous workshops as the mode of delivery, but reduce from four days to three. - Content has been realigned; extraneous content on the systematic review method, rather than the search, was removed. - Slides and handouts are sent to attendees ahead of the sessions. One session series per term is recorded and these recordings are made available to learners after attending. Further, the lack of knowledge or use of the asynchronous modules has led to a complete revitalization of this content. Many of the skills taught at the workshop benefit from supporting materials that provide additional explanation or demonstration. There are also many links to various handbooks, reporting guidelines, and other documents that support systematic searching that could be made more readily accessible if included in an asynchronous learning module. Additionally, an asynchronous module can be used by learners who are unable to attend the workshop for scheduling reasons, and also used by librarians as a teaching tool during their one-on-one research consultations. Because of the importance of the content in these modules Health Sciences Librarians decided to revamp the content in the asynchronous modules to make sure it is up to date and more accessible. To complete this work, the Health Sciences Librarians have worked with a new career librarian who is part of the "Library Resident" initiative at University of Alberta.
Practical Implications & Value
This is an example of a collaborative assessment project between a functional "assessment team" and a specialized library unit. In this case, members of the AIT provided insight on best practices and options for data collection, survey design, and data analysis, while health librarians provided detailed knowledge about the various aspects of the workshop including content, methods of delivery past and present, and potential limitations for implementing an assessment strategy. Both of these parts were necessary to develop an effective and meaningful assessment strategy. Data from this assessment project has been used to make evidence-based changes to the workshop and make it as useful as possible for both learners and librarians. Further the data from this project has been used to initiate additional collaborative projects within the library.
View Slides (PDF)