Page 1

TheBRAIN Usability Study PART A: Background

Objectives: The Library@Mohawk’s redesigned website, TheBRAIN, was launched in September 2007. The results reported in this document reflect the first round of usability testing conducted on the redesigned website, in an effort to simplify and make the new site as intuitive to students as possible.

Participant Recruitment: The volunteer recruitment campaign was directed at all Mohawk campuses (Fennell, IAHS, STARRT, and Brantford). Initially, the desired number of participants was 15, spread throughout the four campuses. Following recruitment, however, this was reduced to 5. The usability study was advertised from January 21, 2008 to February 1, 2008 via: - Mocomotion notice - Flash display on Library homepage - Posters on “Campus Happenings” boards (posted by Student Life) - Poster blitz (posted by Student Life) - Tent cards displayed in all libraries (on computers and work tables) - Posters on circulation desks Students could register either in print or online. In total, approximately 110 students registered to participate. Students chosen to participate would receive a $10 gift card to Cineplex theatres, and the chance to win a 4GB iPod Nano. Refreshments were provided during the interviews. The first screening of the 110 volunteers removed any student that had received formal instruction concerning how to use TheBRAIN. The first round of invitations were sent out to students from this screen group that represented a variety of programs, age groups, campuses and expressed having little to no experience using TheBRAIN. After a small response from this group, invitations were sent out the remainder of students from the screened group. In total, six students responded and agreed to participate in the study. Of the six, one was a no-show.


The Five Participants: Identifier #1 #2 #3 #4 #5

Campus Fennell STARRT Fennell IAHS IAHS

Program Eng.Tech Eng.Tech Human Services Health Sci Health Sci (MedRadSci)

Semester 4 2 1 2 7

Gender M M F F F

The Interview Process: Two individuals were present at all interviews, a facilitator and a recorder. Interviews were conducted in a librarian’s office (Fennell), a group study room (IAHS) and the general computer area (STARRT). A welcome message and list of important information relevant to the study was read to each participant before beginning the study. The participant was then given a print out detailing the 10 tasks they were to complete using TheBRAIN. While they worked, participants were asked to perform a “stream of consciousness” and narrate their own actions, detailing the thoughts and expectations behind their behaviour. If this failed, the facilitator would prompt for elaboration. After completing the tasks, participants answered 12 interview questions regarding their experiences using the website.


PART B: Task Responses 1. How would you find out if the Library has a book on global warming? Everyone chose the link to the catalogue from the center of the screen. The decision came quickly and naturally to the participants. Once in the catalogue, three participants (#1, #2, #5) did a keyword search for “global warming” and two (#3, #4) chose a subject search for “global warming”. When deciding #3 stated she was unsure what the difference between the two options was. Of the five participants, three (#2, #3, #5) chose to limit their search by both location (their home campus) and by language (English). The decision to limit by location reduced the number of relevant results for both #2 and #5, as they limited to STARRT and the IAHS respectively. The Fennell collection best supports this question, and there appeared to be a lack of awareness concerning the different collection focuses between campuses. When few relevant results were found after the initial search, neither #2 nor #5 thought to broaden their search to include all the Mohawk libraries. Instead, #5 settled for a book related to global warming and #2 tried to search the entire Bibliocentre catalogue using the link “All College Libraries”. It appeared that #2 was unaware that “All College Libraries” meant all the college libraries in Ontario, rather than all the Mohawk College Libraries. It may not be obvious that the Bibliocentre catalogue is a shared resource that allows the user to search non-local libraries. It took the participants a maximum of 2 minutes to complete the task.

Notes & Recommendations: 1. Uncertainty surrounding the use of keywords and subject headings. - Clarify use of search options - Change “Subject” to “Subject Heading” or some other term to demonstrate that it means more than just “subject of the book” - Action: CW to take to OPAC group 2. Location limit used inappropriately. - Make limits appear optional - Note collection focus next to campus - Create more obvious book delivery option (instead of “Hold”) - Action: apply results to Evergreen development 3. English limit used unnecessarily. - Make limit appear optional - Set English as default language, or put at top of drop down list. - Action: apply results to Evergreen development


2. How would you find copies of the following articles: a) Moreno, A. & Grande, S. (2005). Knowledge management and life long education in science. Data Science Journal 4(28), 119-126. b) Schaum, K. (2000). Reimbursement Q&A. Frequently asked questions about incontinent skin care products. Advances in Skin & Wound Care 13(6), 264-265.

Four out of five participants (#1, #2, #3, #4) had never searched for articles using TheBRAIN. Participant #5 was self taught, and could use the eJournal Portal proficiently. The following observations reflect the actions and comments of participants #1 through #4. The problems encountered can be divided into two major categories: i)

Understanding article citation

Participants (notably #1 and #2) had difficulty distinguishing the article title, and the journal title, and the roles of each. #1 and #3 tried to use the “Article Quick Find” by searching for the title of the article, but were unsuccessful. Neither was aware why the search failed. It is unclear what the “Article Quick Find” searches, and how it returns results. Participants who attempted to use the Article Quick Find tried it once, and when it failed to return the desired result, they checked their spelling, then gave up and moved on. The intended function of the Article Quick Find was unclear to students, and it may have misled participants into believing they were conducting a more thorough search then was actually be executed. ii) Differentiating locating a known article versus finding an article on a topic The major source of confusion when completing this task came from the list of options presented to the participants after clicking “Magazines & Journals”. The options mix together the resources needed to find both known articles and articles on topics. Participants #1, #2, and #4 selected “Databases by Title”. Once presented with the list of databases, they quickly return back to the previous page. When they saw the list of databases, they were unaware what they were looking at and how they could be used to answer this question. This may have been another case of “too many ‘titles’” (article title, journal title, database title). Once in the eJournal Portal, participants #1, #3, #4, and #5 had little difficulty searching for the journal title, and making use of the coverage notes. When presented with two choices, #4 initially chose the CINAHL option because she was familiar with the name. After exploring the results and finding that the collection did not include the correct volume, she backtracked and successfully chose the correct provider. Other reflections: #1 was the only participant to search the catalogue for the journal article. This may have been due in part to the fact that he had not exited the catalogue following the completion of the first 4

task. After unsuccessfully searching the catalogue using the title of the article, #1 returned to the homepage. It is interesting to note that the catalogue is the only place that uses the term “periodicals”. The other participants neglected to search the catalogue and any print journal the library might subscribe to. This question took too long to complete and the students were generally frustrated at their lack of progress. #2 attempted to search for approximately 5 minutes before giving up.

Notes & Recommendations: 4. Trouble distinguishing article title, journal title and database title. - Plain language key to deciphering citations - Remove “title” from “Databases by Title” – change to “Databases (List)” - Action: Quick fixes approved 5. Use of article resources unclear (eJournal Portal vs. Databases by Title) - Divide resources by use (e.g. Know what article you’re looking for? – eJournal Portal. Looking for articles on a topic? – Databases by Course) (Approved) - Remove “Best Bet” (Approved) - Narrow down options (Approved, requiring more discussion) 6. Print journals in catalogue lost - Add note mentioning the collection is split (Approved) - Add print titles to eJournal Portal with note they are only in print (and link to catalogue) - Action: CW to check with Serial Solutions re: adding print to EJP. 7. Process is too long with too many clicks - Clear, brief instructions at each step to reduce missteps - Action: See #5 - Action: In EJP, change “Title begins with” to “Journal Name” - Action: Simplify second “log-in” page. Include only two options with pictures: “From Home” and “On Campus”. Remove wordy instructions. 8. Temptation to select known database in eJournal Portal, rather than one with the correct coverage. - Omit provider’s name (only include coverage notes and link) - Action: More research/discussion required


3. How would you find a newspaper article on the bird flu?

All participants used the “Newspapers” link from the centre of the homepage. After examining the potential options, #1, #3 and #5 specifically chose to search only the Hamilton Spectator. Reasons for this choice included its familiarity, localness, and the fact it was recognizable as a newspaper, whereas some of the other options were not clearly newspapers. #2 and #4, however, did not choose the Spectator, as it was “too local”. It is interesting to note that all participants mentioned the Hamilton Spectator in some respect. After rejecting the Spectator, #2 chose the National Newspaper Index (NNI), believing it would have a wider scope. While searching, however, it appeared that he was unaware that the results he got were not available in full text. He ended his search believing that the citation and abstract were sufficient. #4 chose to search the Canadian Reference Centre (CRC), and found an article from Rolling Stones Magazine. She was content with this result, despite the fact it was not a newspaper article. All participants keyword searched “bird flu”.

Notes & Recommendations: 9. Trouble distinguishing newspaper databases and individual papers - Separate newspaper databases from individual papers (Approved) - Include brief explanatory note (Approved, requiring further discussion) 10. Overwhelmed by newspaper options - List newspaper databases by amount of full text (Approved) - Specifically note Canadian content (More discussion) - Make online tool to index Mohawk’s newspapers (Country, Newspapers, Coverage, Location in Library) (More discussion)


4. How would you find an eBook on chemotherapy? Not a single participant successfully located an eBook. Everyone easily found the list of eBooks/eBook databases (3 via the homepage, 2 via the right-hand navigation menu), but once they were presented with the options they had no idea how to proceed. #1, #3 and #5 searched Access Science, but did not understand the format of the retrieved results (encyclopedia articles) and failed to find an actual eBook. #1 readily admitted that the order of the eBook list influenced his choice. Participant #2 was clearly unsure where he should be looking, and quickly searched Funk and Wagnall’s New World Encyclopedia and Gale Virtual Reference Library for chemotherapy, finding only encyclopedia articles, which he was not satisfied with. At this point, however, he had dedicated as much time to the search as he was prepared to give. Participant #4, being in the Health Sciences, chose the more familiar “Books@OVID”, but once inside chose to search the EBM Reviews in DARE, feeling there should be something there about chemotherapy. She was satisfied with the resulting article. When selecting from the list of eBooks, participant #3 stated she was avoiding any collection given the subject “all” because it was too broad and less likely to contain something of interest to her. She also noted that the subject descriptions got shorter and less descriptive as she looked down the list.

Notes & Recommendations: 11. Unable to identify an eBook - Language barrier – change to online books? Specifically mention full text. (Action: Address issue through education. No change to BRAIN) - Online library – “read books from home” (More discussion required) 12. Overwhelmed by list of eBook options - Remove collections with limited access to eBooks (Access Science, Funk and Wagnall’s) (Approved) - Include more descriptive subject notes (replace “all” with general topic listing) (Approved, requires content work) - Note size of collection on main screen (e.g. NetLibrary (10,000 titles); Books@OVID (8 titles)) (Approved) - Put general collections at top of list (NetLibrary, Safari) (Approved)


5. How would you find a guide on TheBRAIN about how to prepare a bibliography in APA style? Participants #1, #2, #3 and #5 took less than a minute to successfully complete this task. #4 began by going into the RSI blog and found a PowerPoint presentation on APA, which she thought would do, but then later saw the link to the APA guides on the homepage. There was a striking amount of variation evident in the paths the five participants took to find this information. Two went to the “How To” tab and selected “Cite Your Sources”, one went through the link posted in Library News, one used the “Cite Your Sources” link in the right hand navigation bar, and one used the link in the centre of the homepage. Once at the “Cite Your Sources” page, #3 noted the HTML option was not clear (she first thought the only options were Word and PDF). She preferred the HTML version with the internal links to help navigate the very long page.

Notes & Recommendations: 13. HTML guide not clear - Label HTML link (e.g. Guidelines, Tips, & Chart – [HTML] [Word] [PDF]) - Action: No. Leave as is. 14. Multiple routes used to access guides - Maintain multiple routes


6. Find a list of websites recommended by the library for Environmental Science students. All participants found the link to Recommended Websites from the main homepage, but #1, #2 and #3 were confused by the second page of options. They expected to be taken to the listing of recommended websites, rather than another list of options. There was also a change in terminology used on the second page (“Recommended Websites” changes to “research resources” with no mention of websites). The appearance of the second page led participants to say there were too many clicks involved in getting to the actual BRAINlinks page. They expected a direct link. Only #1, #3, #4 and #5 made it to BRAINlinks. #2 detoured to Faculty Space, and then gave up. Once in BRAINlinks, the participants noted the change is layout. #3 specifically mentioned the need for consistent navigation between all pages (BRAINlinks lacks the tabs, right navigation menu, and breadcrumbs). Participants (#1, #3 notably) also disliked how BRAINlinks opened in a new window (a pop up, by some standards). Participants #1, #3, #4 and #5 found navigating BRAINlinks intuitive. #1 began to search for “environmental science”, but abandoned this approach when he saw the subject browse on the right side. The other participants (#3, #4, and #5) automatically chose the subject browse. Subject browse was the preferred route in this case, as they were not told to find any specific resource, only those suggested to Environmental Science students. #1 also noted when viewing the Environmental Science sites that the list did not appear to be of “websites” as he thought of them. He saw them more as online readings and tools.

Notes & Recommendations: 15. Inconsistent terminology - Change “Recommended Resources” to “Recommended Websites” (Approved) 16. Too many clicks / confusing second page - Link directly to BRAINlinks from homepage (Approved) 17. Inconsistent page layout and functionality - Carry navigation structure to BRAINlinks - Open BRAINlinks in the same window - Action: Forward to Robert and Julia


7. How would you find information on booking meeting/study rooms in the library? With no relevant links in center menus of the main page, participants #1, #2, #3 and #4 began by examining the “About the Library” tab. Participant #5 quickly scanned the Quick Links menu before looking at the “About the Library” tab. Once in the tab, however, most participants had trouble deciding which of the presented categories would include meeting/study rooms. Participants #2 and #4 found the “Services” link and the link to Meeting rooms quickly, but the other participants flipped back and forth between the top tabs before settling on “About the Library” because the others did not fit at all. Rather than choosing “About the Library” because it was the logical choice, participants appeared to settle on this tab because the other options were worse. Within “About the Library”, #1 went to the FAQ, and found the question concerning meeting rooms, and followed the link contained in the response. He remarked that this was a roundabout path to the answer, but it was the best he could do. #1 would have preferred a more linear path to the answer. Participant #3 oscillated between “How to” and “About the Library”, before accidentally noticing the link to “Meeting Rooms” on the right-hand navigation bar. There was some confusion involved in this discovery. At the time, #3 believed she was in the “About the Library” section and was thrown by the fact that the right-hand navigation bar had exploded the “Services” options. She felt that the incorrect menu was being displayed on the right hand side, but it was a fortuitous mistake that worked to her benefit. After searching the “Quick Links” menu, participant #5 scanned all the tabs looking for the phrases “meeting rooms” or “study rooms”. She eventually settled on the “About the Library” tab, and decided that the information would be in either the “Services” or “Locations”. After much hesitation, she selected “Locations” and although this was the incorrect choice, she did not go back and check “Services”. From the participants that located the correct information on TheBRAIN, there was an overwhelming reaction of “is this it?” There was not the amount of information expected, and what they did find was very general, wordy, and not practically useful. Participants were expecting information concerning how to book, locations, hours, and the capacity to book online.

Notes & Recommendations: 18. Information buried in website / too many clicks to find information - Pull link out and display on main page - Create Services “highlights” box on main page (Approved, but requires more discussion) 19. Insufficient information provided - Include meeting room map, individual room capacities/features, how to book instructions


- Action: Short term – enhance current note; Long term – create interactive content 20. No online booking available - Create online room booking form (Forward to Robert and Julia)


8. If you needed help getting started with your research, how would you find an online tutorial? The participants encountered many obstacles when trying to complete this task. The first, and most significant, was the general confusion understanding what was meant by a tutorial. If questioned on the term “online tutorial” during the testing, it was rephrased as “an online guide to help you get started with your research”. Even with clarification, participants #2 and #3 thought the RSI blog was the closest fit to this idea, and would have used the RSI presentations to find out how to begin their research. Participant #3 only found the actual tutorials after the fact when she returned to the homepage. Participant #5 did not understand the idea of “online tutorial”. Instead of looking for a guide, she went directly to the online tools (in the “Find Information” menu) and said these would help her get her research started. Using an online guide or tutorial was not an instinctive action for her during research. Whether this is the result of personal preference or lack of experience is unknown. Participant #1 chose “How to” and “Get Help”, but failed to notice the “Research Survival Guide”. Little time was spent reading through the options displayed on this screen. Eventually he noticed the link to “Self-Serve – Guides and Tutorials” on the main page. He noted that by this point, he would have used Google. Participant #4 was the only participant to find the tutorials on the first try. Although she was able to find the Research Survival Guide, she was confused by the titles of the individual sections. Based on the titles (“What is your mission? Scout out the scene”, etc…) she did not know what the contents included.

Notes & Recommendations: 21. Link is not prominent (buried in text) - Separate tutorials and guides into own section visually - Reduce amount of text - Use clear language - Action: More discussion required 22. Guides and Tutorials are hidden below other forms of service - Make self service step #1 – bring up to the top and increase prominence - Put guides and tutorials before in-person help - Action: Agreed, but more discussion required 23. No help at point of tool - Add links to specific guides from tools (i.e. – link to “beginning research” tutorial from the catalogue, link to “finding specific information” tutorial from the databases page)


- Action: More discussion required 24. Students not accustomed to “self-serve� service model - Promote the use of self serve tools (offer it as an option when conducting in person reference, show during in class instruction) (PF & MM)


9. If you had a quick question, how would you find recommended online encyclopaedias or dictionaries? Participant #3 was the only test subject to use the Quick Links menu and locate a dictionary. All others searched the site briefly then said if the answer was not obvious, they would use a third party internet site, like, Wikipedia or Google. Participants felt there was too much time and effort needed to find these resources for a “quick question”. Participant #5 said as she searched she was looking for the words “encyclopedia” or “dictionary”. The term “Quick Reference” may be creating a language barrier between TheBRAIN and student users.

Notes & Recommendations: 25. Effort needed to find resources not consistent with question type - Put main Quick Reference items on homepage - Action: Put items in BRAINlinks – more discussion required 26. Participants favour popular sites (Google, Wikipedia,, etc) - Include search bars from popular sites on TheBRAIN (No) - Do we want people to go through our site, rather than directly to the site itself? (No) 27. Library language confusing to students - Change “Quick Reference” label - Action: Move Quick Reference and contents to BRAINlinks – consult LIST


10. Where can you find the answers to common problems encountered when attempting to access online resources from home? This task was quickly completed by all five participants with little trouble. Four participants (#1, #2, #4, #5) went to the “How To” tab and selected “Access From Home”. Participant #3 chose to look in the FAQ under “About the Library”, where she found and followed a link to the correct page. Participant #1 had several reactions to the “Access From Home” page. He expected the page would be a log in page. He also noted that it sent a bad message that “Troubleshooting” was front and centre (it leads you to think the service is problematic).

Notes & Recommendations: 28. Troubleshooting message is negative - Remove troubleshooting embedded links to make the “Steps for accessing the databases from off-campus” instructions more prominent. - Move troubleshooting sub-menu links down to a separate section. - Action: Quick fixes approved - Action: Evaluate troubleshooting messages for relevancy


PART C: Post-task Interview Responses Overall Impressions: The participants all agreed that the site functioned well and fit the needs of Mohawk students. Minor complaints included: • • •

Slow (may be due in part to the Internet connection, a slower machine, and running Camtasia) Multiple log-ins required (BAD) Too wordy

There were no complaints or suggestions regarding site content, only structure and organization. When asked what they saw the main functions of TheBRAIN, participants said: • Research (for books and articles) • Citation help • Research tutorials and guides • Hours and Contact Information

TheBRAIN Design and Organization Most participants demonstrated the same navigation preferences when using TheBRAIN: look in the centre of the main page, and then flip through the drop down tabs. Participant #4 noted aloud that she liked the roll-over effect. It was important to participants that each page be visually and functionally consistent. The majority of TheBRAIN functioned in this manner, except BRAINlinks that lacked the navigation options. It was also questioned why BRAINlinks and the catalogue opened separately in new windows. Some participants considered the new windows to be pop-ups. Participant #1 commented that as he went through the tasks, he felt as though he were being forced to narrow his search before he was comfortable doing so. When searching for journal articles, for example, he did not want to select a database by course; he wanted to search a bigger, general collection. A fully functional federated search or comprehensive “Article Quick Find” would suit his needs. There were many comments and observations made concerning the right-hand navigation bar. • It initially appears redundant (displays the same options as the top tab menus). There were mixed feelings about the redundancy (it was there to use, but could make remembering how to locate a resource tricky if you noticed a random link along the way) • There are too many options • Some participants did not realize the content was page/level dependant. They expected continuity between pages. 16

• •

Participants who used this option tended to do so when they have exhausted the other options, or there was too much text in the centre window. The content was logical, but participants were surprised when they successfully found something there.

Other comments regarding navigation and page structure: • Secondary pages are confusing and too busy When asked how a website designed especially for them would present the available resources, responses included: • Intuitive pages (show me what I need) • Resources by Subject/Program • Information presented in labelled folders with lists • Federated searches • No surveys or extraneous clutter on site

Notes & Recommendations: 29. Content is overwhelming and unpredictable in right-hand navigation bar - Use standard menu on each page, but explode subsection relevant to each page. - Action: Approved, with research/development by Robert 30. Lack of fully functional general search options - Develop federated search functionality - Action: More research and discussion required 31. BRAINlinks does not follow navigation format - Alter BRAINlinks to match conventional format - Action: Robert to investigate 32. Secondary pages are busy and take too long to read - Simplify and streamline pages - Action: More discussion and investigation needed 33. Wanted personalized content - Develop subject guides or create subject-centric section of website (alternative presentation) - Action: More discussion required


Language: All participants agreed that the language used on TheBRAIN was easy to understand. The main criticism involved the amount of text presented on each page, and the time required to read through all the presented links. The format of the text (paragraphs of links, rather than lists) may have contributed to this observation. When participants knew what they were looking for, they found the amount of test frustrating to scan through, and when they did not know what they were looking for they found the amount of text overwhelming. There were also two cases where participants noticed inconsistent naming (Recommended Websites changes to Recommended Resources, and the RSI blog changes from “Library Research Class Notes” to “Research Skills Instruction Classes”. There was mixed feelings about the use of “creative” names, like QuickLinks, Quick Reference, BRAINlinks, Cite Your Sources, and the tutorial “Research Survival Guide”. Some used them with no difficulty, but others were unable to decipher what the name was actually trying to say. “Research Survival Guide” and “Quick Reference” are two examples where the purpose of the link did not engage students or strongly convey its use.

Notes & Recommendations: 34. Secondary pages are text heavy and overwhelming - Organize links into lists and smaller sections - Action: More discussion required 35. Inconsistent naming - Create consistent names throughout website - Action: More discussion required 36. Creative names obstructing purpose of resource - Consider replacing with straightforward names - Action: More discussion required

Additional Suggested Changes (from May 1, 2008 LLT) 37. Create Blog/Podcast Tab on top navigation bar.


Usability Study (Test)  

Usability study on the library website.

Read more
Read more
Similar to
Popular now
Just for you