1
Carolyn Gearig, Julianne Walkiewicz, Daniel Schorin, Ethan Zemmin and Marissa Gomez
Assignment #8 Usability Test
Introduction The Michigan Daily is a student run campus newspaper at the University of Michigan that is published in print every day class is in session during the fall and winter terms, and once a week during the spring and summer terms. The Michigan Daily’s website, which can be found at michigandaily.com, is updated around the clock with coverage of a wide range of topics such as academics, administration, athletics, and the city of Ann Arbor. The Michigan Daily also publishes photo and video content, stories on arts and entertainment, and editorials, columns and cartoons expressing the opinions of Daily staffers, the Daily as a whole and any members of the community who wish to contribute guest columns. The audience for The Michigan Daily consists primarily of students, faculty and alumni of the university. According to Google Analytics data we were able to obtain from The Michigan Daily, the audience between October 12 and November 11, 2015 was just over 30% between the ages of 18 and 24. About 23% of readers are between the ages of 25 and 34. Just under half of users, 47%, are above the age of 35. This indicates that the audience consists of some students, but a majority of users are not students and likely are alumni, fans of Michigan sports, Ann
Arbor residents, parents of students and more. The Michigan Daily’s staff consists of about 250 students with varying commitments to the organization. Some editors, such as the Editor-in-Chief and the Managing News Editor, devote over 40 hours a week to the organization, while other students write one story every week or month. Content produced by the Daily is divided into sections: news, copy editing, arts and entertainment, The Statement, video, photo, web development, design, sports, social media and opinion. Each section has one or two managing editors who oversee the section’s content, and most sections have one or more senior editors. The newest section is the web team, and the social media section is just over two years old. The Daily’s web team designed, developed and
2
launched the new website, which debuted in August. This shows that the Daily has more capability than ever before to develop and maintain a high-quality online presence with a quality user experience. Because of this, right now is an optimal time for the Daily to begin prioritizing user experience of the site. We hope that the results of our usability test can have real-world impact on the Daily’s operations. Methods As we developed the methods for our usability test, we considered our previous work with The Michigan Daily’s audience from our interviews and surveys. Our test subjects were three students in SI 422, all of whom are familiar with user experience research. The introductory script and pre-test questionnaire served as a method for us to gain more information on the demographics of our participants and as a chance for them to state what affordances they would like to see on the site and how it might be structured. Questions covered participants’ expectations: we asked them how often they read news sources and specifically, how often they read The Michigan Daily. We also asked how often they use the website, what they want to see on a news website homepage and story page, and what the names of the different news categories meant to them. All of our participants rarely or never have read The Michigan Daily and were not familiar with the site. Therefore, asking our subjects to answer these questions acted as an introduction to the site and what it contains, and gave us context to understand their behavior throughout the usability test. We created thirteen different tasks for our participants that encompassed normal and specific functions that users would need to be able to perform while using The Michigan Daily. Since the website is content driven and has limited interactions for users to engage with, most of the tasks revolved around finding and understanding content. Easy navigation is critical to a functioning news website, so we asked users to find content through their own preferred methods, which included the navigation bar and the search bar. Other functions consisted of commenting on an article, reporting a problem or issue, and finding how to send a letter to the editor. Additionally a task to describe the category “The Statement” after going on the page was also assigned, as this particular section of the site has proved to be confusing to our audience and The Michigan Daily readers. We sequenced tasks so that they became increasingly specific and
3
varied, driving our users to explore new and different parts of the website. During our tests, the moderator read out each task aloud and permitted the test subjects to ask clarifying questions some of our questions were relatively vague, and participants had the option to interpret them in their own way. In cases of a task to search for something specific, the participant was able to read the query directly off of our script. Logging forms were structured to imitate the exemplary forms from lecture. We used a table with four columns: time, screen, type, and notes. One group member was assigned to track how long tasks took and overall start and end time. The logger noted which screens the participant used for each task and any relevant notes, comments, or vocal thoughts. In addition to these observations, the logger indicated what “type” of observation each one was; the three types included comment by user, error, and critical incident. If a task was successfully completed, this was recorded, as well as if the participant was unsuccessful and had to give up. Our post-test questionnaire strived to elicit feedback about the participants’ experiences using the website. We asked about favorite and least favorite features, most useful and least useful features, and what was difficult and easy. We also asked how The Michigan Daily compares to other news sources. This question was particularly important as our participants rarely or never read the Daily, and their impressions of the site could only be compared to other outside news sources. In addition to asking participants to assess their own performance on the site, we inquired their suggestions for changes for the website and things that should stay the same.
Results The objective of our usability test was to develop a significant and vertical understanding of the strengths and weaknesses of The Michigan Daily’s website from a wide demographic group. With many different users logging in to the website everyday, we aimed to discover what errors were made, why they were made, and how we can ameliorate those mistakes in order to create a clean, aesthetic, and functioning website. The underlying theme of our usability results can be identified more clearly with the analysis of the 3 stages of the test, pre-test questions, the usability logging results, and post-test questions.
4
Our pre-test questions emphasized the understanding of who our test subjects were: what their background is, age, major, reading habits, and preferences regarding news websites. We used these questions to develop a more accurate analysis of their logging forms. Our first user, Joanne Fabrics, is a 20 year old BSI major who has a thorough background and interest in technology. Joanne has never read The Michigan Daily and has never visited the website, making her an extremely useful participant. Joanne identified the importance of a proper social media connection with the website, a functional comments section, and related article links at the end of each regular article. Joanne’s sentiments regarding the navigation bar topics are as follows: news - ‘updates in the world’, sports - ‘most recent wins related to Michigan’, opinion - “Area to search for personal questions”, arts - “latest/popular music”, statement - “Don’t really know”, multimedia - “celebrity media”. Our next user, Leah Wilkinson, is also a 20 year old female studying information science. She reads news sources monthly, but rarely reads The Michigan Daily. Her preferences for news articles are picture oriented articles that are displayed in chronological order on the homepage with the most recent articles closer to the top. For the Daily’s navigation bar, Leah’s thoughts are as follows: news: “recent events”, opinion: “student’s thoughts”, arts: “STAMPS and art museum”, statement: “UM’s mission”, multimedia: “video or audio”. The last user we tested is Alexander Georges, a 20 year old male who is an information science major. He reads news sources weekly, but has never visited The Michigan Daily website. He prefers news articles to feature picture oriented stories that are expanded upon in succinct details. Alexander prefers to use the news as an entertainment source. His sentiments on the Daily’s navigation are: news - “anything that is not on the other categories. . . Also means the big headlines”, sports - “sports games/stats, whether with schools, colleges, professional, etc,” opinion - “comments on things like politics, products, life,” arts - “new art techniques, new artists, new discoveries from famous artwork, ” statement - “I’m not too sure here… maybe like opinions?”, and multimedia - “videos, photos.” The pre-test questions allowed us valuable insight into the thoughts and habits of our testers entering the usability test. There are also many patterns that illuminate the strengths and flaws of the website. For instance, almost all of our testers noted confusion about the Statement portion of the website. Since they were not familiar
5
with this section, they are less likely to visit the section, further preventing user experience and deterring them from returning to the site. Additionally, some of our users noted a preference for Michigan related information, while others wanted more national coverage and entertainment. The next stage of our usability test is our logging results. On average, it took our participants 14 minutes to finish the test. Our first participants, Joanne, breezed through the first portion of the test, quickly finding the most important article on the home page, the most read article, and the most recent sports article. However, on the fourth task, Joanne was unable to find a recent multimedia video. Unable to use the media navigation section, she searched ‘video’, and ended up finding a year old video. After this task, she successfully completed tasks 5-8, in which she found the Daily’s social media sites, and found articles by author and by title. During the 9th task, Joanne struggled to find an article written on a specific date. She used the search function, but was unable to sort based on date published. Joanne went on to finish the rest of the test without difficulty. During task 10, in which she was asked to identify what the statement page meant to her, she stated that ““I guess it’s like when [the authors] have a personal opinion and they just want to share it.” Overall, Joanne reported a good overall experience with the site, except the limited functionality of the search function. She made no critical errors throughout the test. Our next user, Leah, completed the test in 14 minutes and 17 seconds. Her first unsuccessful task was finding the Daily’s Facebook page. She repeatedly clicked on the Facebook widget buttons, which led to sharing the article, not the page itself. On task #7, Leah successfully completed the task of finding an article by author Lea Giotto, but made a critical incident when she accidentally navigated to the home page from the search bar and clicked an incorrect article. She quickly recovered and found the article. This highlights a small navigation flaw in the Daily’s construction. Like Joanne, Leah was also unsuccessful in task #9. She also used the search function, but made additional errors by failing to see the article’s date before clicking on it. Our last user, Alexander, completed the usability test in just over 13 minutes. Alexander did not successfully complete task #4 of finding a recent video. He did navigate to the multimedia section, got deterred in the photo section, and found an old video of his roommate.
6
He also did not complete task #5 of finding the Daily’s social media pages; despite finding the correct button, the link was broken on the website. Like the other testers, Alexander easily completed the rest of the test until task #9, of finding an article on a specific date. He used the search function, but due to the sites lack of chronological search listing, could not find an article on the specified date. He finished the rest of the test quickly and reported a good user experience, despite frustrating issues with basic site functions, such as the search and social media buttons. The post-test questions allowed us to zone in on specific areas and analyze the user’s preferences on specific functions of the website. Joanne noted the search function as her least favorite feature, but praised the easy-to-use widgets for social media and linked articles. She identified the report an issue function as the least important of the website. She reiterated her frustration with the lacking search function capability, and also advised the Daily to update the comment section to allow anonymity and disconnect with social media in that specific area. Leah commended the simplicity and minimalism of the website which allowed her to quickly finish tasks, but recommended re-organizing the navigation bar, citing her confusion about the ‘statement’ portion. Interestingly, she noted the search function as the most useful part of the page, specifically how it allowed her to quickly find articles based on author and title. Like Joanne, she was frustrated by her inability to find an article by date, which is a lack of capability of the website, not her own fault. She praised the layout of the website, indicating its cleanliness compared the other news sites, but said the Daily can improve by organizing its articles in a more pertinent manner. Alexander’s favorite part of the website was the statement page, which he didn’t know about initially, but was intrigued by the author’s opportunity to convey important personal sentiments. His least favorite feature was the multimedia page, which he says was an unnecessary addition to the website, because its content can be found in other areas. He likes the holistic nature of the homepage, where he can find all the news he wants to read very quickly. He noted his displeasure with the omnipotence of ads on the site, which detracted from the user experience. He thinks the navigation bar should be retained, except for the multimedia folder, which should be eliminated. Overall, the results from the tests were extremely useful in identifying the strengths and flaws of the website, as all user tests led us to a specific area. All results showed an emphasis on the importance of maintaining technological consistency,
7
specifically within the widget buttons, a topic that all 3 of our testers noted. Additionally, the flaws of the search button were thoroughly noted. The Daily’s strengths were constantly affirmed, in which the page’s aesthetics are very adept at allowing users to navigate the website quickly while finding important information quickly. We learned that all users criticized either the multimedia or statement sections of the navigation bar, so further attention should be made regarding the organization of the nav bar.
Conclusion Moving forward, there are a number of recommendations that we feel would greatly benefit The Michigan Daily based on our results. First, we recommend that improvements be made to the search bar. Many of our participants cited issues with the functionality of this feature, and it almost seemed to induce more confusion than anything else. Specifically, there were issues searching for articles based on an author and finding articles that were published on a certain date. We recommend that, since the search bar feature is very important to users, an extensive amount of effort is put into improving its functionality. Additionally, we recommend that The Michigan Daily more effectively conveys the purpose of the Statement, as our participants cited that it is an intriguing feature but they were unaware of its existence. We noted that, while the Statement is very beneficial, users are far less likely to click on something if they do not know what it is, which ultimately renders the feature slightly useless. Finally, we recommend that The Michigan Daily makes fully sure that they maintain consistency throughout their website, as inconsistencies can result in a loss of users. As for future research, we would be interested in delving more deeply into the best way to disseminate the Statement to the users of the site. We believe that we would be able to conduct a very productive study that would provide us with the methods leading to a large increase in the use of this feature. We would also look more into the correlation between social media and The Michigan Daily in an attempt to gain more users. Throughout the semester, many participants in our studies noted that articles posted on social media were the main reason as to why they were aware of the The Michigan Daily. This is a very important finding, and, with more time to
8
research, it would be very intriguing to figure out how The Michigan Daily can best utilize this information in order to gain users.
Reflections Overall, our usability tests went quite smoothly. We collected some valuable information and were able to gain great insight into what the website does well and how The Michigan Daily could work to improve its user experience. However, as anticipated, we did run into a few hiccups along the way. We had some difficulties logging information and tracking timing during the usability tests. Carolyn ran the first two tests and went through them a bit too quickly, straining Marissa and Julianne’s ability to record test results and timing in as detailed a manner as they would have liked. We could have benefited from doing a test run of the usability test before our first official test -- if we have done this, we may have been able to correct this problem immediately. However, by our third test, we had these problems figured out and did not have nearly as much trouble logging the test results. If we did this again, we would also try to significantly improve our communication during the test. This is something we did very well during our third trial: Carolyn asked Marissa and Julianne if they were ready before moving on to a new task, and overall, our team was more in sync and worked through problems more readily while the tasks were still ongoing. There were also some issues with the tasks themselves. One of our tasks asked users to find the Daily’s Facebook page, which is usually accessed from an icon on the navigation bar. However, the icon was turned into a sharing widget right before we began our usability tests. We were caught off guard when we got to this task in our first usability test. Again, if we had tested everything out ourselves, we could have avoided this problem or phrased the question in a different way to show that the icon was broken. A similar issue came up with another task: when we asked users to search for an article from a specific date, and they were not able to complete the task because the search function was not specific enough. Again, this brought up some important usability issues with the website, but it caught us off guard. We were not sure when to deem this task impossible, which made the usability tests a bit awkward. The physical setup of
9
our tests was not perfect, either. It was difficult to pay close attention to the person completing the tests, and most seemed overwhelmed by the fact that they were surrounded by five people. In the future, we would look into a better way to facilitate better communication, observation and tasks in the usability tests -- but these problems did not impede our ability to conduct high quality tests.