Study: Blacklight Search UX
Introduction
An evaluation of the default Blacklight search interface, branded for CRKN's Canadiana collection, was conducted in late 2024. The aim was to identify usability issues and enhance the user experience as CRKN transitions from the existing Canadiana Access Platform (CAP) to a new Blacklight-based system. This report summarizes the findings from usability testing involving diverse target groups, including genealogical researchers, GLAM professionals, and post-secondary students.
Object of Evaluation
The object of evaluation was the default Blacklight search interface, with CRKN-Canadiana branding applied, carried out in late 2024 by the CRKN team. See the figures below.




Evaluation Method
This test was conducted as an unmoderated think aloud usability test with the online usability testing software, Maze. As this was an unmoderated usability test, the users were not observed live while they carried out the tasks. Instead, their interactions with the website and their verbal comments were recorded for later analysis. Users were recruited from our voluntary pool of existing Canadiana users, as well as a group of seminar students from a university level history course.
Each participant individually carried out tasks on the Canadiana-Blacklight website during their usability test session. At the end of their usability test session, the participants answered a System Usability Score (SUS) questionnaire of pre-defined questions and had the opportunity to submit any comments in a final open-ended question. Subsequently, the captured video recordings were analyzed by the author. The evaluation is based on an analysis of these videos from 24 unmoderated usability test sessions. The total time for each usability test session including answering questions was less than 30 minutes.
Data Captured
Task 1
Instructions: Search for the term, Maple Leaf
Expected answer: Users were expected to type the search term, Maple Leaf, as prompted in the search box and press the magnifying glass icon-button to initiate the search.

Average Time on Task
16 seconds

Task Success Rate
97%
Task 2
Instructions: Show 100 items per page in the search
Expected answer: The users were expected to move below the search bar and locate the drop-down component, which indicates the current number of items displayed in the search results. They were then expected to click this component, which would open the full list of values: 10, 20, 50, and 100 items. Then, they were expected to click the last item, 100, which would reload the search page with 100 results in the first page of the search listing.

Average Time on Task
31 seconds

Task Success Rate
91%
Task 3
Instructions: Sort the search by newest to oldest
Expected answer: Users were expected to move towards the right side of the page, below the search bar, to find the search sort options. They were then expected to click the drop-down menu which indicated the current search sorting selected, “Sort by relevance.” Then, a full list of values including “Sort by relevance” and “Sort by newest to oldest” would be displayed to the user. Then, they were to click the “Sort by newest to oldest” option which would reload the page in this order.

Average Time on Task
10 seconds

Task Success Rate
100%
Task 4
Instructions: Show only items written in the Ojibwa language
Expected answer: The participants were expected to navigate to the facet listings in the left-hand menu on the search interface, titled “Limit your search.” They were then expected to locate the “Language” facet. Then, they were to click the facet title, which would open an expansion panel listing the first twenty values alphabetically, and a “more” indicator. Users were then to notice “Ojibwa” was not in the first twenty values and click “more.” Then, a pop-up window was expected to appear, where the users could see an alphabetical, paginated list of values for the Language facet. The pop-up model also included options for sorting the listing alphabetically, or numerically, by most popular values. Finally, the full-value listing pop-up had “previous” and “next” buttons for navigating through the search results. As “Ojibwa” would be somewhere in the middle of the list of languages available to choose from, users were expected to click the “next” button three times, until they saw “Ojibwa” listed in the values. Then, users were expected to click “Ojibwa” which would close the pop-up and filter the search results on “Language” -> “Ojibwa.”

Average Time on Task
36 seconds

Task Success Rate
100%
Task 5
Instructions: Browse all the items about Arctic Regions
Expected answer: The users were expected to navigate to the “Subjects” facet in the left-side “Limit your search” menu in the search interface. Then, they were expected to click the “Subjects” heading, to expand the list of values, this time, sorted numerically by most popular subject. Users were expected to scan the listing and see that “Arctic Regions” was not in the top 20 values shown in accordance with the sort order. Then, users were expected to click “More,” which would open the full listing of values for “Subject,” and click “next” once to find the “Arctic Regions” subject listed. Finally, they were expected to click the “Arctic Regions” value, which closed the pop-up and filtered the search results by “Subjects” -> “Arctic Regions.”

Average Time on Task
102 seconds

Task Success Rate
50%
Task 6
Instructions: Find all the items by the Royal Society of Canada
Expected answer: The users were expected to navigate to the “Limit your search” left hand side menu, and find the “Creator” facet heading, and click the heading to expand the list of the top twenty values sorted numerically by popularity. This time, users were expected to see the “Royal Society of Canada” in the top twenty values. Users were expected to click the “Royal Society of Canada” value, which would then filter the search results on “Creator” -> “Royal Society of Canada.”

Average Time on Task
44 seconds

Task Success Rate
75%
Task 7
Instructions: View the items published from 1800-1850
Expected answer: Participants were expected to navigate to the “Limit your search” menu and find the “Date Range” facet. They were then expected to click the “Date Range” facet heading, to expand the default listing, which lists increments of 200 years from year 0 to the current year, as well as two inputs for entering a custom “from” and “to” range, and finally, an “Apply” button to initiate the custom-date-range filtered search. Noticing that the required date range is not a 200-year interval, users were expected to enter “1800” in the “from” input, and “1850” in the “to” input, then press “Apply” to initiate the filtered search. Then, the search results filtered on the “Date Range” -> “1800-1850” were expected to be displayed.

Average Time on Task
16 seconds

Task Success Rate
96%
Task 8
Instructions: Remove all applied filters
Expected answer: Users were expected to move the cursor below the search box, to the section that lists all the applied facet values. Then, they were expected to notice a button which said, “Reset Search” and to click it. This would refresh the page to the default empty search, with no filters applied.

Average Time on Task
22 seconds

Task Success Rate
96%
Post-Session Interview
SUS Scoring
System Usability Scale (SUS) is a standard way to assess a products overall user-friendliness. Higher scores mean better usability. A score above 70 is a good, while one that is over 85 is excellent.

SUS Score
81.5 (good)
User Comments
"Add a description (or little help explanation pop-up) of what main page search is searching when one selects Title vs Subject vs All Text vs etc."
"I want more detail on the resources that could be searched."
"Add a filter by type of source (such as peer reviewed journal.)"
"It would be very good if it included more archives, museum catalogues, and library resources."
"More content. This is a great service. It helps me greatly with subjects that interest me. The more resources that could be added the happier I would be. Keep up the great work!"
"If digitized microfilm files could have more than just image 1, image 2, etc. It would be so nice if for each new file to have something that indicates 'new file, or file page.'"
Interface Analysis
Search Box
Recommendations
Ensure "all fields" is the default option for the search field drop down selector.
Ensure that when the user presses enter, a search is initiated.
Search Results Count Selector
Recommendations
Add a loading indicator on the search page that displays when the page sends a request to the back end for search results. This could be triggered by any of the sorting options, the display count, or when entering new search terms or facets.
Search Results Sorting Selector
Recommendations
Ensure "oldest to newest" is an option listed in the sorting options.
Selected Filters Section
Recommendations
It may be beneficial to have a “clear selected values” functionality within each facet's expansion panel to remove all selected values for a facet in one button press.
Facet Listing
Quotes
“I don't know how to find this, oh my gosh.”
Recommendations
Make facet lists easier to scan by adding dividers, adding spacing between values, or reducing the number of values listed if possible (without triggering the more interface, see the next component section for more information.)
Ensure your facet labels are easy to understand and differentiate. For example, CRKN can reflect on their use of the term “collections” for describing the format-based grouping (serials, monographs, etc.) as this is not immediately intuitive to all users. Possible alternatives could be “format” or “material.”
Facet Pop-ups
Quotes
“I feel like I am missing how to do this properly.”
“Why is it so hard to figure out?”
“Having to hit next is annoying.”
“Clicking, clicking, clicking, clicking! This is crazy – that needs work.”
“This is tedious!”
“I am never going to find it this way [by clicking next], this is annoying, and I am done.”
“Why not just give me a full list at this point?”
Recommendations
Redesign needed to make the facet pop-ups usable for longer listings of values.
A temporary solution is to set the default before there being a “more” function to the count of your largest facet.
Do not rely on the search facets for displaying all the metadata values you want your users to be made aware of. Use facets sparingly and to support common filter experiences for users, such as date and language. For supporting exploratory experiences or topical listings, instead design an index with alphabetical navigation as a separate experience. Label the page “Explore our collection by topic” or similar.
To make it easier to explore from the search results, a value for a field of an item can be turned into a clickable link. When users click on this link, they will be directed to a search page that displays all items matching that specific field value. For example, if an item lists their subject, that could be a clickable link, which gives a search result matching all items with that subject.
A good example of a more user-friendly facet interface, which solves both the pain points experienced by the users is the Inuit Circumpolar Council Canada Archives. (ICCCA)
The ‘More’ button loads more values directly into the facet listing
The ‘Explore’ button loads a pop-up with a search box and alphabetical navigation


Date Range Facet
Quotes
“Oh - custom year range, at the bottom!”
“I think I could use that for genealogy for sure. This is fun.”
Recommendations
Place the custom date range inputs directly under the Date Range facet header.
Search Results Listing
Quotes
“It must be bilingual because the subject is bilingual.”
“It is telling me that there are 363,844 results in here, and I imagine these are individual items.”
“Serials – oh It is a journal.”
Recommendations
Ensure that the distinction between a series and an individual item is clear to users, so they understand that they will encounter different page designs based on the item type.
Closing Statement
In conclusion, the usability evaluation of the Blacklight search interface identified important areas for improvement. While users generally found the interface intuitive, challenges such as missing loading indicators and issues with facet pop-ups were evident. By addressing these concerns and implementing usability enhancements, we can significantly improve the search experience and ensure a smoother transition for users familiar with the existing CAP system. Additionally, adopting an open-source and community-driven approach will enhance usability not just for Canadiana users, but also for other institutions utilizing Blacklight-powered search systems.
Last updated