Page 1

4.30.19

Creative Loafing Usability Test Report Blake Sheffield, Katherine Turnipseed, Kevin Vicente-Gonzales


Creative Loafing Usability Test Report

2

Executive Summary

This report is based on the usability of the Creative Loafing Atlanta Website. Creative Loafing is an online newspaper to help people learn about upcoming music and entertainment events in the Atlanta metro area. This test was conducted on Monday, April 22, 2019 at the Kennesaw State Marietta Campus. We tested 5 different participants, and the duration of each test was around 10-15 minutes. We had a team of 3 testers: Blake Sheffield, Katherine Turnipseed, and Kevin Vicente-Gonzales. Our team initially chose the Creative Loafing website because of layout and navigation difficulties within the site. We wanted to specifically test the navigation and layout of the Creative Loafing website to find what was not working and recommend changes that could be made to improve the site. During the test, our team asked a series of 9 questions or scenarios for the participant to locate certain parts of the website, as well as a System Usability Scale (SUS) form to fill out once the test was completed. Generally, the tests found similar issues within the site. • The search feature was difficult to use and understand • The unique layout of the site made it challenging to navigate • The font and colors used made the articles difficult to read • Division of sections on website were not well organized • Dates for events were inaccurate and not updated in a timely manner • An “About Us” feature is missing in the website • A few links lead to an unsecure 3rd party website for coupons and deals that looks different from the native site and invites distrust in the site This report contains our methodology used, the testing goals and objectives, the metrics and information about our participants, tasks and scenarios, findings and test results, as well as recommendations.


Creative Loafing Usability Test Report

3

Table of Contents ..........................................................................................................................................................1 EXECUTIVE SUMMARY .......................................................................................................................2 TABLE OF CONTENTS ..........................................................................................................................3 TABLE OF FIGURES .............................................................................................................................4 INTRODUCTION .................................................................................................................................5 METHODOLOGY .................................................................................................................................5 TEST GOALS AND OBJECTIVES.............................................................................................................5 METRICS ............................................................................................................................................6 PROFESSION .........................................................................................................................................6 AGE ....................................................................................................................................................6 GENDER...............................................................................................................................................6 AREA OF RESIDENCY ...............................................................................................................................6 PARTICIPANTS ...................................................................................................................................6 TASKS AND SCENARIOS ......................................................................................................................7 TASKS .................................................................................................................................................7 TASK #1............................................................................................................................................................ 7 TASK #2............................................................................................................................................................ 7 TASK #3............................................................................................................................................................ 7 TASK #4............................................................................................................................................................ 8 TASK #5............................................................................................................................................................ 8 TASK #6............................................................................................................................................................ 8 TASK #7............................................................................................................................................................ 8 TASK #8............................................................................................................................................................ 8 SCENARIOS ...........................................................................................................................................8 SCENARIO #1..................................................................................................................................................... 8 FINDINGS AND TEST RESULTS .............................................................................................................9 TASK 1 OBSERVATIONS............................................................................................................................9


Creative Loafing Usability Test Report

4

TASK 2 OBSERVATIONS.......................................................................................................................... 10 TASK 3 OBSERVATIONS.......................................................................................................................... 11 TASK 4 OBSERVATIONS.......................................................................................................................... 11 TASK 5 OBSERVATIONS.......................................................................................................................... 11 TASK 6 OBSERVATIONS.......................................................................................................................... 11 TASK 7 OBSERVATIONS.......................................................................................................................... 12 TASK 8 OBSERVATIONS.......................................................................................................................... 12 SCENARIO 1 OBSERVATIONS ................................................................................................................... 13 SYSTEM USABILITY SCALE ....................................................................................................................... 13 RECOMMENDATIONS ....................................................................................................................... 14 SEVERITY CODES .................................................................................................................................. 14 FIXES................................................................................................................................................. 15

Table of Figures FIGURE 1 MAP OF ATLANTA AREA.......................................................................................................................................... 6 FIGURE 2 CREATIVE LOAFING HOMEPAGE ................................................................................................................................ 7 FIGURE 3 TEST TIMES FOR EACH PARTICIPANT .......................................................................................................................... 9 FIGURE 4: RIGHT-SIDED NAVIGATION PREFERENCE ................................................................................................................. 10 FIGURE 5 MOST OFTEN CLICKED FOR RECENT NEWS ............................................................................................................... 10 FIGURE 6 SEARCHING FOR EVENTS ....................................................................................................................................... 11 FIGURE 7 TIMES FINDING MUSIC FESTIVAL OR CONCERT .......................................................................................................... 12 FIGURE 8 SUS SCORES ....................................................................................................................................................... 13 FIGURE 9 CHART OF SEVERITY CODES ASSIGNED ..................................................................................................................... 15


Creative Loafing Usability Test Report

5

Introduction

Creative Loafing is a media and entertainment news company that has been around in Atlanta since 1972. They shifted from a strictly paper format to online also several years ago. Creative Loafing is a hub for finding events, restaurants, and live music in the Atlanta area.

Methodology Our team decided that it was most important to test the navigation and layout of the Creative Loafing site after walking through the site ourselves. We developed goals and objectives for our testing. Once we decided what we were going to test on the site, it was time to write a script for the questions we needed to ask the participants during the test. We also wrote a consent form for each participant to fill out, and a System Usability Scale form (SUS) for after the test was completed, to ensure we gathered as much data as possible. We set up the test in an empty classroom on the Kennesaw State Marietta campus. We had a tripod to hold the phone camera to record the participants from behind as they were completing the test. We also used the screen recording feature built into the Mac laptop we used for testing to record the sounds and navigation of the site. We recruited 5 participants in total. Three were recruited prior to the test, and two were selected the day of testing. Once the participant came into the room, they were greeted and seated in front of the laptop being used. We then read the script to them of instructions, which included their right to leave at any time. The instructions also informed them that the website was being tested, not them personally. We had them sign the consent form that stated they knew they were being recorded and they understood the statements in the instructions. Screen recording was started, and the test had begun. During the test, we asked the participants 9 questions/scenarios to lead them through the website. We asked them to think aloud so we could understand their thought process of site navigation.

Test Goals and Objectives • •

Determine the pros and cons of the layout for the Creative Loafing Website See how the navigation works for the site, and how it can be simplified


Creative Loafing Usability Test Report •

6

Find out what can be made easier when looking for specific events on the site

Metrics

In our testing, we recruited 5 different participants. The sections below detail the demographics of our participants.

Profession

All 5 of the participants in the test were students at Kennesaw State University. They all used websites regularly in their studies.

Age

Our participants were all students at Kennesaw State and were in the typical college student age range of 18-mid-twenties.

Gender

Out of our 5 participants in the tests, 3 were females and 2 were males. We did not expect the gender of our participants to affect the results of the tests.

Area of Residency

Because Creative Loafing is based in Atlanta, we wanted to make sure that our participants lived in the area and understood local culture and events. This was easy for us to find by using Kennesaw State University students.

Figure 1 Map of Atlanta Area

Participants

We recruited participants from Kennesaw State University that had never used or heard of Creative Loafing. This ended up being in our favor, because it was their first time navigating the site and everything was new to them.


Creative Loafing Usability Test Report

7

The information we gathered was all raw and from a brand-new perspective from each participant.

Tasks and Scenarios

Here we are describing the different tasks we had the participants complete and a scenario that we gave them. We also describe our reasoning behind these questions, and they will lead to our findings and recommendations for the site.

Figure 2 Creative Loafing Homepage

Tasks

The tasks were the questions we asked them to complete while finding specific parts of the site.

Task #1 We had the users simply look at the design of website’s homepage and give their opinion on how they thought it looked. We did this to help us understand their initial thoughts on the layout of the homepage.

Task #2 The next question was to ask the participant to find the most recent news. Since Creative Loafing is an online newspaper, we wanted to see how easy it would be for someone to quickly navigate to the most recent news.

Task #3 We asked the participants to find where to search for events. This was done because there are a few different search bars on the site as well as an entire


Creative Loafing Usability Test Report

8

page dedicated to searching for events. We wanted to see what was used most and if it was easy to find.

Task #4 We asked the participant to find an article on Bigfoot. This was done because there is an entire section dedicated to Bigfoot on the site. We wanted to know if a user is going to find that specific section in the navigation, or search for the information separately.

Task #5 The next task we had them complete was to find a restaurant recommendation based on location. We asked this because there is an entire section on food in the website, and since it is an Atlanta based newspaper, there are many food recommendations, but there should be options based on the location within Atlanta.

Task #6 We asked them to find the hours for a music festival or concert. Music is a part of the main navigation of the site and many people would come to the site simply to find out information about live music in the area.

Task #7 We asked the participant to find an event for the upcoming Friday. The menu bar has a few different ways to find events, but overall it is complicated to find. We wanted to know how easy it was for users to find an event for a specific date.

Task #8 The last task was for the user to find the Creative Loafing staff. We noticed in our initial review of the site that the website is missing an about page in the menu and wanted to see how users would go about finding this information.

Scenarios

We asked the users one scenario in which they would be led to find something on the website if they were in that particular scenario.

Scenario #1 We gave the users a scenario involving them wanting to find a deal for a music festival ticket on the sister site linked into Creative Loafing. This was done because the sister site is tricky to get to and looks like a completely different company. We wanted to know how users felt about the navigation of it and leaving the native site.


Creative Loafing Usability Test Report

9

Findings and Test Results

After the tests were completed, we went through the data we collected in the forms of videos, recordings, notes jotted during the test, as well as the System Usability Scale each participant completed.

Time of Each Test 12 10 8 6

4 2 0 Participant 1

Participant 2

Participant 3

Participant 4

Participant 5

Time of Each Test Figure 3 Test Times for Each Participant

Task 1 Observations

In the first task, we asked users to look around the homepage and tell us what they thought about the layout of the website. Almost all of the users noted that initially they did not know what the website was about. One noted that the site looked too “boxy� and not like a newspaper website at all. Most noted that the colors and fonts were not easy to read and not congruent throughout the website. They also immediately noticed that the right-sided navigation was different, and 1 participant liked that while the 3 did not, and 1 was indifferent.


Creative Loafing Usability Test Report

10

Right-Sided Navigation Indifferent, 1, 20%

Liked, 1, 20%

Did Not Like, 3, 60% Liked

Did Not Like

Indifferent

Figure 4: Right-Sided Navigation Preference

Task 2 Observations

When asked to find the most recent news on the website, all users clicked on the menu and then the “News and Features” part of the navigation. Then, most participants would click on the first subcategory instead of “Current Issue” which held the latest news and led the to the news home. It was noted that there were too many ways to find news, and that leads to confusion on the site. The menu navigation had too many options and the participants almost never clicked on the same thing twice.

Figure 5 Most Often Clicked for Recent News


Creative Loafing Usability Test Report

11

Task 3 Observations

In task 3, the participants were asked to find where to search for events. Most of the participants seemed to not see the “Events Search� subcategory on the website, and instead would click on the search bar. It was noted when searching for events that they were not updated and would show events from last year sometimes.

Finding Event Search Subcategory

Used Alternative Way to Find Events

Does See and Use Event Search

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

# Of Participants Figure 6 Searching for Events

Task 4 Observations

When asked to find an article on Bigfoot, all but one user immediately saw the name clearly in the subcategory under news and features. This showed us how easy it was to find things in the subcategories if they were worded and organized in the most simplistic way possible.

Task 5 Observations

We then asked the users to find a restaurant recommendation based on location. Since Atlanta is a big area, they should have been able to find a restaurant in their area within the food section of the website. However, even though the users could find recommendations, none were based or filtered by location.

Task 6 Observations

Finding the hours for a music festival seemed to be one of the most difficult tasks for the participants. Some of them gave up, and some of them just


Creative Loafing Usability Test Report

12

could find the hours for a concert. It should be easy to find the hours for a festival within the “Music” category on the website. They also would often click on the “After Hours” section, which would not lead them to find the hours for a music event in Atlanta. The titles were not specific enough. When events were listed in a chronological order on the site, they would not show times unless the individual event was clicked on. Participant 3 gave up looking after almost 4 minutes of looking.

Finding Music Festival or Concert 3.5 3 2.5 2 1.5

1 0.5 0 Participant 1

Participant 2

Participant 3

Participant 4

Participant 5

Time Taken In Minutes Figure 7 Times Finding Music Festival or Concert

Task 7 Observations

When asked to find an event for the upcoming Friday, most participants noticed that the events for the upcoming weekend page were one week behind. This was frustrating for some participants, because they would then have to use the search function, which yielded out-of-date results. Participant number 3 gave up after 3.5 minutes of searching and could only find an event for the Saturday after.

Task 8 Observations

All of the participants looked on the menu when asked to find the staff. Some noted that they would have appreciated an “About Us” within the menu, because that’s where they initially thought they would find the staff. After looking around, they all eventually found the staff by clicking back


Creative Loafing Usability Test Report

13

home and seeing “Contact” on the footer. This was a hassle for each participant, even though they all eventually found the staff.

Scenario 1 Observations

In our scenario for the participants to find the sister site to Creative Loafing within the subcategory, all eventually found it. However, one found it through seeing it as an ad on the website. Participant 4 noted that they would have no idea that it was related to the native site because it has a completely different look and logo. They noted that it should just be integrated within the actual Creative Loafing site.

System Usability Scale

The SUS has a list of 10 questions that the participant can answer 1-5 that go from “Strongly Disagree” to “Strongly Agree”. We gave this test out to each participant after they completed the test to have more quantitative data on what the participants thought of the website.

System Usability Scale (SUS) Score 80 70 60 50 40 30 20

10 0 Participant 1 Participant 1

Participant 2 Participant 2

Participant 3 Participant 3

Participant 4 Participant 4

Participant 5 Participant 5

Figure 8 SUS Scores

The usability can be tested by the final score of the test: • 0-50: not acceptable (Red) • 50-63: marginally acceptable (low end) (Purple) • 64-70: marginally acceptable (high end) (Blue) • 71-100: acceptable Two users found that the website was not acceptable, two found that it was marginally acceptable on the low end, and one other user found it to be


Creative Loafing Usability Test Report

14

marginally acceptable on the high end. However, none scored over 71, which is considered acceptable.

Recommendations

We assigned severity codes for each observation encountered in the tasks. The definitions for each severity code are as follows: • Catastrophic problem: “show stopper” problem, most important to fix • Major problem: Task can be completed with much difficulty • Minor problem: Task can be completed with some difficulty • Cosmetic problem: Minor appearance related issues

Severity Codes Task Task Task Task Task Task Task Task

1 2 3 4 5 6 7 8

Cosmetic Minor Minor Cosmetic Minor Catastrophic Major Cosmetic

Severity codes use scenarios • Participant three encountered a catastrophic problem when attempting to find music festival hours. • All participants encountered a cosmetic problem when asked to find an article on Bigfoot. • Participant three encountered a major problem when asked to find an event for the upcoming Friday. • All participants encountered a minor problem when attempting find a restaurant recommendation.


Creative Loafing Usability Test Report

15

Figure 9 Chart of Severity Codes Assigned

Fixes

Our biggest recommendation to improve usability is to remove bloat in the navigation menu. The navigation menu outside of “News and Features” is really just the event search feature, but more complicated. The subcategories within the navigation menu are not kept up to date, they are convoluted in genre assignment, and there are completely vestigial categories that could be removed. Most of the commentary below are specific examples of these points. Another recommendation, based on testing, is to move the navigation to the left side of the screen. Only one viewer found it interesting to have on the right, and most users were confused by it. Loaf Deals has an entirely different layout and aesthetic. This combined with the fact that https security is not present on the home page of the Loaf Deals website, damages credibility. Place an “About Us” section. If the site repeatedly lists articles by author name, it would be beneficial for users to be able to learn anything about the authors listed. The search feature was problematic for our users. There is a “Search” feature and an “Event Search” feature, which is confusing to users because you can search for events in both search functions. The only difference we noticed was that the Event Search feature has more filters to refine the search. This should be remedied for clarity. The CSS needs to be fixed as well- both search functions have different fonts, backgrounds, and the filter buttons are not organized. This makes the website appear less coherent.


Creative Loafing Usability Test Report

16

The News & Feature section has five news articles that are not organized chronologically in the sub-menu. These features are Sports and Justice, Bigfoot, Say No to Hate, March for Our Lives, and Letter to Dr. King. Most of these are from 2018. When users navigate to read the news and features, and they see the foremost article selections are up to a year old, it makes users confused and the News loses credibility. The News Home page, found through the News and Features menu, has a list of articles on the right that can be scrolled down, along with a large image on the left that the user can either click on to see the article, or click an arrow left or right to navigate articles. These two features dominate the page and perform the same function- they both serve to show an identical feed that users can click on and read. We would recommend altering the layout to allow the two feeds to perform separate tasks. A headline feature of high importance could take up the entire left category, for example, while smaller articles could be navigated to and seen on the right. In the “After Dark” subcategory of the website, there is an article titled “Welcome to After Dark” which serves as an explanation of the section. This article is third in the feed on the left and completely missing from the feed on the right. This is really confusing to users - not only is the introduction to the category buried in a feed, but these two feeds are different from each other and the News + Features feed, which shows the user two versions of the same article list. A significant number of features in the “After Dark” section are interviews and articles about DJ’s and music. These are not featured in the “Music” section at all. We would recommend a serious look at organization and hierarchy- there’s an interview with DJ group “Lords of Acid” in After Dark that is the first article you see and takes up the most visual space on the site. This article is over a month old, promoting a concert that occurred on March 26th, but it’s completely missing from the Music Homepage. The Arts and Ent. category of the site has two links highest in hierarchy: Edward McNally and Curt Holeman. After some exploration we discovered that these are the authors of articles within the site, and by clicking on the links you are filtering articles to view based on author. One problem with this is that Edward McNally’s link only links to a single article but requires a second click to actually view the article. Another problem is that each link sends the user to a separate layout- Curt Holeman’s navigation link sends you to the same layout as the rest of the site, but Edward McNally sends you to what appears to be a search result. Neither link actually tells you


Creative Loafing Usability Test Report anything about either author, or why their names are on the navigation menu.

17

Profile for katherine.turnipseed

Creative Loafing Usability Test Report  

Creative Loafing Usability Test Report  

Advertisement