Outdoor School Usability Study
FINAL REPORT Heidi Schindler, Hasani Burn, Ivan Koveshnikov & Nikki Pete MARCH 20, 2014
Table of Contents PAGE
EXECUTIVE SUMMARY . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
OVERVIEW Early Revision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
STUDY METHODS Background Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . User Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Recruiting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary of Research Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
04 04 04 05
DETAILED FINDINGS ISSUE 1: Sorting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ISSUE 2: Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ISSUE 3: Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ISSUE 4: Cancellation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ISSUE 5: Rentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
07 10 13 14 15
APPENDIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
REI Usability Testing Final Report | Heidi Schindler, Hasani Burn, Ivan Koveshnikov, Nikki Pete | UW HCDE UCD 517D |
Executive Summary This report provides an analysis and evaluation of the Outdoor School classes section of the REI website. Incorporating a classical usability testing methodology, this report will discuss the issues and recommendations based on the observation of eight participants. While there were relatively few issues regarding general navigation and participant understanding, participants had trouble with sorting classes, filtering search criteria for classes, finding specific class information, and easily canceling classes. This report will dive deep into these key issues and provide potential recommendations to resolve the issues.
REI Usability Testing Final Report | Heidi Schindler, Hasani Burn, Ivan Koveshnikov, Nikki Pete | UW HCDE UCD 517D |
Overview We conducted a usability study on users’ experience for the class sign-up process on REI’s online Outdoor School. The Outdoor School allows REI members and the general public to sign-up for outdoor activity focused classes or adventure vacations, while also providing additional information through activity-based videos and articles written by experts. This study will deliver an in depth view of users’ experience starting on the homepage and ending on the purchasing page. It provides insight and measurable action items within 1) finding a class, 2) decision making into the class chosen, 3) class signin process, and 4) purchasing a class.
Early Revisions The initial plan was to research the Outdoor School, general Navigation of the REI website and the REI Outlet store. After recommendations from the our instructor Michael Berg, we contacted our stakeholder Tom Lurino for guidance regarding REI’s main interests. We discovered that there was a significant need to focus on the Outdoor School section of the website. We refined our scope to dive deeper into that section.
Objectives Evaluation of the online Outdoor School class sign-up will benefit not only users with an interest in recreational education, but may also expand REI merchandising. These relationships can further develop more REI membership, which in turn can bolster customer loyalty through the purchasing of clothing and gear-- further enhancing REI’s brand within the community. The few keys in achieving this goal are 1) identifying and revealing, through an usability study, the pain points users experience throughout the entire class sign-up process, 2) communicating these findings back to REI, and 3) implementing changes through development and/or other processes needed in solving the identified pain points.
Research Questions 1. Can users easily navigate through the REI Outdoor Class website? 2. Can users easily identify classes of interest? 3. Can users filter classes by price range? 4. Can users find class information targeted for novices? 5. Do users understand where the chosen class is located? 6. Can users find directions to the chosen class? 7. Do users care about the instructor information for a class? 8. Do users care about the “affiliated partners”? 9. What roadblocks do user encounter signing up for classes? 10. Can users easily find how to cancel a class? 11. Can users cancel a class? 12. Do users seek additional information related to a class of interest, such as rental, gear, or prerequisite information?
REI Usability Testing Final Report | Heidi Schindler, Hasani Burn, Ivan Koveshnikov, Nikki Pete | UW HCDE UCD 517D |
Methods Background Research We communicated heavily with our REI stakeholder at the start of this project. We began with the intent to perform a comprehensive evaluation of multiple REI website sections (e.g. REI Outlet, REI navigation, and REI Outdoor School). However, due to time constraints and REI’s needs, we scoped our project to focus only on the stakeholder’s research interest: “What potential problems and opportunities can you find in signing up for an REI class?” We specifically focused on the Outdoor School activity class sign-up process. Our stakeholder commented that REI would benefit from researching and improving the Outdoor School because it attracted new customers and engaged existing customers. Customers are encouraged to learn a new activity through REI, which increases confidence and fosters a community of outdoor enthusiasts. This ultimately has the potential of inspiring people to buy more gear or clothing from REI, and to sign up as members-- thus generating revenue from classes and equipment. In addition, our stakeholder informed us that novices represented the majority of individuals who sign up for classes through the REI Outdoor School. We were informed that the majority of class registrations occur online. Two known pain points were finding and canceling classes.
User profiles Based upon our background research we constructed user profiles that would guide us in our research process. Initially we created three personas based upon the information shared by our project sponsor and our review of the REI Outdoor School. We initially wanted to represent different aspects of the Outdoor school which included signing up for adult classes, signing up for family-oriented classes, signing up for international activity tours, and interacting with expert-generated articles and videos. Eventually, we settled on two primary personas which were named Nelly Baker (“Looking for something new”) and Eric Anderson (“Enhance my skill sets”). These two personas corresponded with our newly scoped focus on signing up for adult classes. This better reflected out stakeholder’s needs and the patterns that stakeholders observed from their current Outdoor School customers. We eliminated the third persona because it was no longer relevant to our study. The key objective of this study was to understand every step in signing-up for and purchasing a class. For this reason, our two personas varied in age range and comfort level with activities. When seeking participants, we desired a diverse representation of exposure to the REI website and interest in outdoor activities.
Recruiting Our personas lead us to recruit individuals who were in interested in outdoor activities who were either experts or novices in the activity. We sought a mixture of genders and age ranges (23-60 years old). We
posted a recruiting ad on the Washington Trails Association Facebook page, a LinkedIn page, and through personal emails. Resulting participants were convenient samples of friends and associates. Participants did not completely match our personas but we felt that having a liberal recruiting strategy would allow us to recruit more individuals within our time constraints, thus allowing us to discover more problems and opportunities.
Summary of Research Methods Our main focus was on signing up for an Outdoor School class. The employment of a usability study, as opposed to other types of user research methods, was most appropriate because it allowed us to use scenario-based tasks that resulted in rich data within the set time constraints. By using a usability test, we were able to highlight the delights and pain points about the current website’s architecture and navigation. Our stakeholder’s concerns, our personas, and a website review guided us in our creation of our test plan and usability study.
Website Review Prior to creating our test plan all members of the team reviewed the website to find possible pain points that we wanted to investigate. Based upon this review we created our research questions and subsequent task elements.
Usability Test In our usability study, we gathered data regarding the overall ease of use and understanding of the REI Outdoor School section. We assessed not only the usability of the site itself, but also how well it served users with very little experience who may be looking for something new.
“Within-Subjects” Design We conducted a within-subjects study, where each participant worked through 15 tasks focused on the Outdoor School. Participants completed all tasks in sequential order to replicate real life scenarios. We conducted 8 individual 45 minute sessions. With each participant, we explained the purpose of the study, gathered background/demographic information, and conducted a post-test debriefing interview. During the remaining time, participants worked through tasks.
Scenarios and Measures Through our scenarios and tasks, we analyzed our users and how effectively they completed each task. During the tasks, we asked them to think- aloud to assess clarity of the presented information. As for measures, for most of the tasks we ensured that there was a way to complete these tasks, and in doing so participants effectively followed the same steps one would in a real-life scenario to use the REI Outdoor School section. For tasks that did not have a definite end, we ask participants what their expectations were if they were to complete the task.
Logistics Location The Outdoor School usability studies were conducted on the University of Washington campus either in the Odegaard Library or in Sieg Hall. Our stakeholder offered a conference room at the REI headquarters. However, conducting the studies on campus allowed for easier coordination between the researchers and participants.
Media Devices We wanted to record participants as they interacted with the REI website. We used the program GoToMeeting, a webinar software, to audio record participantsâ€™ voices and video record the website. This software was downloaded onto two laptops which were used by participants to complete the tasks.
Incentives Participants received a bag of either Ghirardelli chocolate squares or Seattle Chocolate truffles, for participating in the study.
Mock User Identity In order to protect the privacy of our participants during some tasks, we created fake personal information. This allowed our participants to attempt the sign-up process without entering their own contact information.
Limitations Completing the class sign-up When conducting the usability study, we were prevented from asking participants to go through the entire for-fee class sign-up process due to inability to obtain mock credit card information. We chose to use free classes to allow participants to complete the sign-up process. This would also allow participants to attempt the cancellation task. Free classes, however, were difficult to find.
Controlling a live website One major limitation of the study was having very little control of the daily changes within the Outdoor School website. Unlike working with a prototype, we found fluctuations in the content on the website (e.g. changes in available classes). However, we believe that these fluctuations would not change our end results.
Accessing a lab While the school offers LUTE as a resource to conduct and record usability studies, its availability is limited. Because of this limitation, we chose to use our laptops. This afforded us more flexibility when scheduling participants.
Detailed Findings The following findings highlight the problems participants experienced when completing the usability tasks. The business goals that were identified early in the study process were to: ● attract new customers ● engage existing customers by enticing them to try new things ● enhance customers’ confidence in learning new things ● foster a community around outdoor enthusiasts and teach best practices ● benefiting REI holistically through customer purchases of new gear or clothing and membership sign-up We have prioritized the following issues based upon their importance. Our recommendations for each issue are listed at the end of each section.
ISSUE 1: Sorting Participants faced challenges when sorting classes. Free Classes Six of the eight participants failed to find a free class using the sorting option currently provided on the classes. Once they were in the section shown below in Figure 1.1, all participants tried to sort utilizing the Price toggle. When the free class did not show, most participant tried the REI’s internal site search which provided only products, not classes. When asked what they would do next, most said they would quit, assume free classes did not exist, or do a Google search.
Inconsistent Listings There are inconsistencies in the toggle â€œPriceâ€? for sorting classes. As shown in Figure 1.1 there are no free classes available in the Puget Sound area. This occurs if you filter for classes using the using the left navigation panel. The two type of filtering options for location are shown below in Figure 1.2. Customers can filter using the left navigation panel or the drop down menus within the main panel.
If a customers use a drop down menu in the main panel, they would discover a free class, as shown below in Figure 1.3.
Closed Classes When asked to choose a class of interest, many participants found their first class of interest to be closed. This frustrated or confused most participants. There was no indication of what the user should do next in this situation. One participant commented that it was unnecessary to show closed classes in the sort column; only available or wait list classes should be shown.
Skill Level None of the participants were able to utilize the toggles to sort by class skill level. Participants would utilize the title to find a class for novices. Most participants determined their search was concluded once they found a class titled “introduction” but some of these classes were actually intermediate, as seen in Figure 1.5. But as you can see in Figure 1.6, the skill level is actually intermediate not beginner, and additionally, none of the participants found the skill level at this current location.
Overall Recommendations for Sorting ● Provide a separate section dedicated to free classes or sync various location paths to arrive at free classes when initially sorting ● Remove the closed classes or provide additional options for full/closed classes (next availability, etc.) ● Provide a separate column for skill level
ISSUE 2: Filtering Many participants struggled with the filtering options provided while struggling for classes of interest. Locating Class by City Many participants struggled with locating the city where they would take their class for several reasons. Initially, there is considerable redundancy in the main Outdoor School for how to locate your region. Figure 2.1 below shows the drop down menu that is recreated nine times on the main panel. When selecting a particular group of classes, these drop down menu provide the same information. On the left navigation panel, there is another listing of the same information, repeated twice, for a total of eleven instances of the same information. Four out of our eight had difficulty finding classes of interest in their region.
Filtering to Classes of Interest Once inside the actual filtering page, after selecting the region, participants struggled to locate classes of interest. Four out of eight participants had difficulty finding classes of interest in their region. One participant said her frustration would cause her to leave the website, stating â€œI would give upâ€? (Participant #1).
As the REI website has a considerable focus on membership and a user profile, a personalization and unified identity on REI are important to users. When unable to specify a city to locate, several users expressed that it was frustrating to have to re-select the region every time they search for classes.
“The computer already knows your location, why can't it predetermine your location and jump into the activities/classes?” - Participant 4 One participant was confused why the filtering options in the left navigation panel below were unchecked by default (Fig. 2.2).
Furthermore, once in the process of filtering and searching for classes of interest, many participants were confused by the options presented and their lack of clarity. The top-down navigation filtering was not intuitive, as seen in figure 2.3 below.
Not one of the participants could figure out what “Classes and Events” would filter the results by. In the future, if participants can not successfully find a class of interest at a convenient location in a quick, easy manner, they may abandon their search.
Overall Recommendations for Filtering ● Consider clarifying the top-down filters as seen in Figure 2.3 so that they are more intuitive and understandable ● Consider allowing the sign-in membership to track a user’s location if they would like, or prioritizing the nearby locations of that user in some way ● Consider simplifying the main Outdoor classes page to reduce redundancy. Consider showcasing more information about the classes in drop-down menus instead of the regions’ information which is heavily referenced in the same page.
ISSUE 3: Information Users wanted more information, and with clarity and consistency Participants wanted to visually see all of the information that was most pertinent to the class they were interested in signing up for. This included overview of learnings, skill level, and information about who would be teaching the class. Participants wanted to know what they will be learning from these classes in a clear and concise manner. Participants also identified inconsistencies within many of the classes’ descriptions of information, and expected learning outcomes. Some classes are lengthy explanations, as shown in Figure 3.1, while others are short bullet points , as shown in Figure 3.2.
Instructor information was nonexistent or participants could not find it readily. There was not a link in the class information in the classes page for most classes, as shown in figure 3.3. We heard from multiple participants that the instructor information would be pertinent to the beginning users, as well as those taking advanced courses.
Overall Recommendations for Information A syllabus style system of class information formatting, that includes the information about the class itself, the instructor and contact information, required materials (with links to the REI shop to purchase those items) along with what the user will be expected to learn after, including readiness for next level/course. We recommend this layout be consistent for all classes, and easy to read, like an academic class syllabus.
ISSUE 4: Cancellation Canceling a class was very difficult and confusing for participants Once participants had successfully signed up for a class (or had a moderator guide them through signup) they were instructed to attempt to cancel the class. Class cancellation is an important part of user autonomy in the event that they could not partake in the class and want to free up spots for other interested parties. During the test, 5 participants struggled and 1 completely failed to locate a way to cancel a class. Only 2 participants were successfully able to walk through the process of canceling a class. 6 of 8 participants expected a way to be able to cancel a class from within the website, electronically. This would allow them to take control of canceling classes in a similar way to how they signed up, and also showcase the classes they have signed up for. This was expected via the cart function or a similar
way, as the shopping experience extended to purchasing/signing up for classes. “Give me a screen where I can cancel things…” - Participant 2 While no participants discovered this issue as a class did not become unavailable during the study, www.rei.com/displaycart showcased the message seen in figure 4.1 below. This seems to indicate that viewing classes is a function of the website via the cart, yet it was not observed by any participant.
Overall Recommendations for Cancellation ● Allow participants to see classes they are registered for without leaving the REI website. ● Consider implementing an easier system to cancel classes online, in a more user-friendly way, such as via the Display Cart or from within the Class description page.
ISSUE 5: Rentals Every participant struggled to locate where they could get rental gear Rentals All participants struggled to find the Rentals section, and none found it quickly and efficiently. 3 participants attempted to look at the footer panel on the homepage and elsewhere through the dropdown menus, but were unsuccessful. After some deliberation, 5 participants eventually utilized the search option, and located Rentals like this. 2 participants commented they would use Google to search for the Rentals section.
Overall Recommendations for Rentals ● Make Rentals a more prominent section on the REI webpage via the headers or drop-down menus. ● Make each class tied to Rentals that may be appropriate to help identify what gear a user may need for a particular class.
Study Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary of Raw Data and Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Recruiting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Final Findings Presentation Deck . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Original Personas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . NDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
19 32 40 45 64 68
INTRODUCTION What (purpose) We will conduct a usability study on users’ experience for class sign-up on REI’s online Outdoor School. The Outdoor School allows REI members and general public to sign-up for outdoor activity focused classes or adventure vacations, while also providing additional information through activity-based articles written by experts. This study will get an in depth view of the entire users’ experience starting on the homepage to ending on the purchasing page. Providing insight and measurable actions items within; 1) finding a class 2) decision making into the class chosen 3) class sign-in process, and 4) purchasing a class.
Why (goals) Evaluation of the online Outdoor School class sign-up will benefit not only this experience, but also expands REI merchandising. These relationships can snowball into developing more REI Membership, which inturn can gain customer loyalty through the purchasing of clothing and gear. Helping enhance REI’s brand within the community. The few keys in achieving this goal are; 1) identify and reveal through an usability study the pain points users experience through the entire class sign-up process, 2) communicating these findings back to REI cooperate in a format that are actionable, and 3) implementing changes through dev and/or other processes needed in solving the identified pain points. RESEARCH QUESTIONS 1. Can users successfully find the class section, starting from the home screen? 2. Can users find cycling classes in the Puget Sound area? 3. Can users filter by bicycle classes by price range? 4. Can users find a cycling class for novices? 5. Do users understand where the chosen class is located? 6. Can users find directions to the chosen class? 7. Can users find out more about the instructor for this class? 8. Do users understand what class “affiliated partners” means? 9. Can users find the costs associated with the class? 10. Can users successfully sign-up for the class? a. (how do we do this? Should participant sign up for a free class? Should we create a fake account and use a personal credit card and then delete sign- up? 11. Can users cancel a class? 12. Can users find the rental equipment section, starting from the class section? a. (does not look like the REI site offers bike rentals—WE can have user shop instead…or the site offers bike tune-ups, which would be handy before a long bike ride!)
When (schedule) FEB 20
MAR 01 - 05
Draft of Study Plan for Review
Final Study Plan Submitted
Conduct Study Plan
Implement study plan with users matching primary persona
How (methods) In our usability study, we will gather data regarding the overall ease of use and understanding of the REI Outdoor School section. We are going to assess not only the usability of the site itself, but also how well it serves users with no experience who may be looking for something new, and additionally to better retain users who may regularly use this site or similar ones. “Within-Subjects” Design We will conduct a within-subjects study, where each participant will work through multiple tasks that we create that best suit the data we will need for our project. Participants will complete all tasks in sequential order to replicate real life scenarios. We will conduct 6-9 individuals in 45-60 minute sessions. With each participant, we will explain the test session, review background/demographic information, and conduct a post-test debriefing interview. During the remaining time, participants will work through tasks focused on the Outdoor School. Our Scenarios & Measures Through our scenarios and tasks, we will analyze our users and how effectively they can complete each task, and ask them to speak aloud to assess clarity of the presented information as well. Tasks will be presented further in this document in greater detail. As for measures, we will ensure that there is in fact a way to complete these tasks, and in doing so they will effectively be following the same steps one would in a real-life scenario to use the REI Outdoor School portal.
Who (personas) Recruitment for this usability test will pull key elements from within the two Primary persona named Nelly Baker who is “Looking for something new” and Eric Anderson “Enhance my skill sets.” Although these two personas vary in age range and comfort level with activities, our key objective of this study is to understand every step in signing-up and purchasing a class. So the overarching key differentiator between these two personas is the comfort level of online search and purchasing.
First Contact with Participants Do you enjoy taking classes to learn new things? Do you enjoy the outdoors? Well, we are looking for you. You probably know one of us, Heidi Schindler, Ivan Koveshnikov, Hasani Burns, or Nikki Pete, and we need you to conduct a Usability Study for our University of Washington masters class. What are we trying to solve? Well, you know all those online moments of, discovering information is too hard to find, or why canâ€™t I do this, type situations? Well, we want to solve those in our project, and to accomplish this, we need your insight. All we ask is for 45 minutes or less of your time. Where: XYZ When: Day, Month Date Time period How: Send Heidi Schindler an email at firstname.lastname@example.org to express your interest When Please not that this is NOT an open session. You much confirm your appointment with Heidi, Hasani, Ivan or Nikki. Look forward to hearing from you! Heidi Schindler, Hasani Burns, Ivan Koveshnikov , and Nikki Pete
Participant Screener Survey Screener survey: provide an online survey to conclude if ppt meet study plan criteria General 1. How many years have you lived in the Pacific Northwest? 2. Would you be interested in taking outdoor classes? a. If yes, what interests you about outdoor classes? b. If not, why? 3. Where have you looked for outdoor classes before? 4. Have you ever signed-up an outdoor class? a. If so, where? b. If not, can you imagine an activity that you would pursue classes for? 5. What is the maximum amount of money you would pay for the class you are seeking? REI Specific 1. Have you shopped at REI before? 2. How often do you go to REI?
3. Have you heard of the REI Outdoor School? a. If so, from where? 4. Have you ever signed-up for a REI class? a. If so, which class? i. How was your experience? ii. What do you remember most from signing up to the class ending? 5. Have you ever rented REI equipment before for classes or otherwise? a. If so, for what purpose? 6. What are your preferences for paying online? Credit Card, Paypal, other? Participants must meet the following criteria: Characteristics
Desired Number of Participants
Participant Type Pilot
Total number of participants
Persona type Nelly
Current method of transportation Car
Age 23-60 yro
Second Contact with Participants Thanks for interest! Before we go ahead and schedule an appointment. We would like you to take this initial survey. We have specific needs that must be met, and to make sure we use your precious time to the utmost efficiency, please take the survey. Take the Survey: XYZ.com Then: Once we are notified that you have completed the survey, we will reach out by sending you an email or if you prefer a phone call. Please note that this is NOT an open session. We must confirm the appointment with you. Thank you! Heidi Schindler, Hasani Burns, Ivan Koveshnikov , and Nikki Pete
Scheduling of Participants Accepted Participant Great you completed the survey! Thank you! Guess what! You are exactly who we are looking for, so let’s get that appointment scheduled. Please choose a time that works best for you by filling out the time below and sending this email back to us. Where: XYZ When: Day, Month Date Time period Time period Time period Time period We will be sending you a reminder email with directions and details. Thank you! Heidi Schindler, Hasani Burns, Ivan Koveshnikov , and Nikki Pete
DQ Participant Great you completed the survey! Thank you! For this specific study… I am not sure how to say “denied”. L
Thank you! Heidi Schindler, Hasani Burns, Ivan Koveshnikov , and Nikki Pete
On-call Participant Great you completed the survey! Thank you! Guess what! You are exactly who we are looking for, but right now all our appointment are scheduled, so we were wondering if you would be an on-call participant. This means that if we get a cancellation, which you would get a 24hr notice, you would fill the schedule hour. Please choose a time that works best for you below and the send this email back to us. Where: XYZ When: Day, Month Date Time period Time period Time period Time period We will inform you within 24hrs if we get a cancellation. Which we will include directions and details. Thank you! Heidi Schindler, Hasani Burns, Ivan Koveshnikov , and Nikki Pete
Reminder Email/Call This is a reminder that you signed up to be part of our Usability Test. Our project depends on you, so please come to your chosen appointment on-time. We have attached a PDF that specifies your appointment time, our cell phone numbers, emails, and directions to the building. Please do not hesitate to contact us if you have any questions. Where: XYZ When: Day, Month Date Time period Time period Time period Time period
If for an emergency reason, you need to cancel. Please give us a 24 hour notice by sending an email or giving us a phone call. Thank you! We look forward to working with you. Heidi Schindler, Hasani Burns, Ivan Koveshnikov , and Nikki Pete
USABILITY TESTING APPROACH Materials and Staff Recruiting notice Participant recruitment screener Non-disclosure agreement (includes consent form and intellectual property form) Set up REI account information for participants to enter into class form (privacy issue/participant comfort) Test script Lab/study room Recording devices/ software Waiting area for participants who come early Participant gratuity-- box of cookies
Documentation Tactics With regards to documentation, our team will work to utilize a team throughout the usability studies to ensure proper notes are taken, and that no user actions are missed. Currently we as a team communicate and share and review all documentation to ensure that we efficiently grasp the concepts and requirements of this project. Additionally, in compiling our scripts, and presenting our tasks and plan within this document we plan to keep record of our progress through our sharing of documents. We have taken the initiative to utilize a screener survey to get initial information about our participants that we will collect, and continue to collect data throughout and after the testing process before bringing our results to the surface in the final deliverable.
Testing In-detail 1. Welcome Script The script below was developed so that each participant is greeted in the same manner and same understanding. Giving an common introductory experience for each participant. Each participant will be given a task script. We will ask them to read the task aloud and then complete the task once they have stated that they understand the task.
Hi, __________. My name is ___________, and I’m going to be walking you through this session today. Do you need to use the restroom or need anything to drink? Before we begin, I have some information for you, and I’m going to read it to make sure that I cover everything. You probably already have a good idea of why we asked you here, but let me go over it again briefly. We are asking people to try using a Web site that we are researching so that we can see whether it works as intended. The session should take about 45 mins. Are you okay to stay a little bit over if needed? I would like to go over the NDA form with you. It simply states that what we are testing should remain confidential. Also we will be using audio and video recording to ensure accurate data collection. We will not share your identity with anyone outside of this research. However, the information obtained from the research is considered to be the intellectual property of our client. Your participation is voluntary meaning that you can stop the study if you no longer agree to participate. I’ll have you sign the consent here…Thank you. Before we begin the tasks I would like to ask a few questions: Intro Questions General 1. Are you new to the city or region? 2. Are you interested in outdoor classes? a. If yes, what interests you? b. If not, why? 3. Where have you looked for outdoor classes before? 4. Have you ever signed-up for an outdoor class? a. If so, where? 5. What is the maximum amount of money you would pay for the class you are seeking? REI Specific 1. When was the most recent time you shopped at the REI store or online before? 2. How often do you go to the REI store or online? 3. Have you ever signed-up for an REI course or online class? a. If so, which course? i. How was your experience?
ii. What is the most memorable experience of signing up for the REI class? 4. Have you ever rented REI equipment before for classes or otherwise? a. If so, for what purpose? 5. What are your preferences for paying online? Credit Card, Paypal, other? Script Great, let’s begin the tasks. First I want you to know that we are testing the site not you. You can’t do anything wrong here. I did not design this website so you do not have to worry about hurting my feelings. Don’t worry if you make mistakes. If you have any questions as we go along, just ask them. I may not be able to answer them right away since we are interested in how people do when they do not have someone sitting next to them to help. But if you still have any questions when we are done I’ll try to answer them then. If you need to take a break at any point, just let me know. Do you have any questions so far? As you use the site I’m going to ask you to think aloud: 3. Think Aloud Demo Script
We will present the participant with a pen and ask them to take it apart. We will ask them to verbalize physical and attitudinal reactions while they dismantle the pen. We will then ask the participant to similarly describe what they are thinking during the website tasks. 4. Tasks to be Achieved Participant will think aloud at each task and decision Moderator walk participant through each task Note taker will gather information into an excel sheet
Home Page Tour Script Goal: Let participant get used to the homepage before starting tasks. Also, let’s participant start the think aloud protocol. No need to spend a lot of time on this task. First, I’m going to ask you to at this page and tell me what you make of it: what strikes you about it, whose site you think it is, what you can do here, and what it is for. Just look around and do a little narrative. You can scroll if you want to, but don’t click on anything yet. Tasks Script Goal: Evaluate class sign up process
Task 1: Youâ€™ve decided to take a course through REI. Find a class of interest. Which activity are you looking for? Observe: Where does the user go to complete this task? Do they use the Learn tab or the class/events link at the top of page? Task 2: Now that you have found a course for your activity. Locate courses for this activity offered in your area. Task 3: Arrange the classes by cost and locate a free _______ course. Task 4:
Sign-up for the class using this information:
Morgan Edwards 5416 15th St Seattle, WA 98105 PHONE: 206 555 1212 email@example.com Ending Questions: How was that experience? Task 5: You want to rent equipment for this activity in the future and want to view rental options. Task 6: Return to the main activity course page. You would like to find a for-fee-course that has been created for novices. Task 7: Find the location of this course. Task 8: Find directions to this course. Ending Questions: How was that experience? What form of transportation would you use to get to this class? Would commuting prevent you or encourage you to take this class? Task 9: You would like to know more about the course instructor. Locate information about the instructor for this course. Ending Questions: How was that experience? Is the information where you expect it to be?
Find out more about the partners that are affiliated with course.
Ending Questions: Did you find this information useful? Task 11: Confirm the costs associated with the class . Task 12: Sign-up for the class using this information: Morgan Edwards 5416 15th St Seattle, WA 98105 PHONE: 206 555 1212 firstname.lastname@example.org Take participant through the first screen. Have them leave task before submitting. Task 13: You remembered that you plan to meet with your friend on day that the free course is being offered. Cancel the free course that you previously sign up for. Task 14: You would like to think about taking a course in the summer. Find available ____________ courses that are being offered in June. Ending Questions: How was that experience? Is the information where you expect it to be? Task 15: You want to the REI homepage; do that now. 5. Exit Interview Questions ● Would you recommend this site to a friend? Why or why not? -OR● Follow up questions regarding issues that presented themselves during the study -AND● Do you have any questions for me, now that we are done? 6. Gratuity Thank you for participating. That was exactly what we need. We have a gratuity here for you. You have been very helpful.
NEXT STEPS The following deliverables will be presented at the end of the study: · Final presentation of finding and recommendations · Executive summary and final report highlighting the study’s findings and resulting REI Usability Testing Final Report | Heidi Schindler, Hasani Burn, Ivan Koveshnikov, Nikki Pete | UW HCDE UCD 517D |
recommendations Below is the entire scope and schedule for this study plan. MAR 06
Draft of Study Results
Final Study Results
Final Study Results
Utilizing data documentation, provide a quick overview of raw data
Presentation of final finding
Final written report with any additional graphic format of data
Summary of Raw Data and Observations
This report provides an analysis and evaluation
Major Insights and Opportunities ●
● ● ● ● ●
Finding location: Location is easy to find. However, location name may too broad at times. One participant commented, “The computer already knows your location, why can't it predetermine your location and jump into the activities/classes.” Rentals: Rental page was difficult to find. Rental link should be located within course page or with other related equipment in the Shop section. Instructor: Participants would like to have access to instructor information within the course page. Free courses: Participants would like a way to easily find free courses on the website. It may help to provide a separate section for free classes to avoid having to filter based upon price to find. Cancel courses: Participants would like an option to cancel classes through the site. Entering personal information for course sign-up: Participants found that entering personal information an easy process
Summary of Participant Characteristics (8 participants) Most participants were female ranging in from 30-39 age. Almost all participants utilized the internet “greater than 6 times daily” and rating themselves at very proficient user. The majority of participants claimed Amazon as their most visited site. There was a split interest in outdoor activities ranging from somewhat to very interested, with the majority already have searched for an outdoor activity. Most participants stated, when asked if they had been to the online REI Outdoor School before, they had not. A majority of the participants are interested in group outdoor activities. Most of our participants fit the persona Nelly; we had two male participants that fit the persona of Eric in interests but not age range. Recruiting Screener Responses Gender Male
greater than 6 times daily
3-5 times daily
Level of perceived internet efficiency
Level of outdoor activity interest Somewhat interested
Frequency of online outdoor activity search Once per month
Twice per month
Level of solo outdoor activity interest Not interested
Level of group outdoor activity interest Very interested
Frequently used websites Ebay
Summary of Introductory Questions 1. Are you new to the city or region? Yes: 4
2. Are you interested in outdoor classes? Yes: 7
3. Where have you
Yes, Climbing Yes, University gym: 1 Mountainee
looked for outdoor classes before?
4. Have you ever signed-up for an outdoor class?
of Utah and a salt lake city outdoor organizatio n: 1
rs online and Backpackin g light (not local): 1
5. What is the maximum amount of money you would pay for the class you are seeking?
>$100 for $15 whole day
1. When was the most recent time you shopped at the REI store or online before? Never
$20 for a day and $40 for two $1000 days $40
Store, month ago; Site more frequentl Christmas y
2. How often do you go to the REI store or online?
Once (only to look for corpora te mission , not to shop) 3-4/year
3. Have you ever
every couple months dependi ng on season
$1,000 if the class was a week in the field of "hands on" training
3 weeks in store 1 day online
2 week instore Yes, & snowshoei online ng
2/monthly during the hiking season 3/monthly
every seaso n, 3 week 4/year instore
$50-$80 for once/wee k for a couple of months
$25 for one time class; $200 for a weekly class
Shopped once/ June 2013 Sept 2013
2x/month (she gets newslette r)
signed-up for an REI course or online class? 4. Have you ever rented REI equipment before for classes or otherwise? No: 6
5. What are your preference s for paying online? Credit Card, Paypal, Credit other? card:2
Debit card: 1
Summary of findings List of Tasks
1. Youâ€™ve decided to take a course through REI. Find a course of interest. Which activity are you looking for?
7/8 Passed 1/8 Found difficulty 0/8 Failed
2. Now that you have found a course for your activity. Locate courses for this activity offered in your area.
4/8 Passed 4/8 Found difficulty 0/8 Failed
3. Arrange the classes by cost and locate a free _(Course of interest)______ course.
1/8 Passed 1/8 Found difficulty 6/8 Failed
4. Sign-up for the course using this information:
5/8 Passed 1/8 Found difficulty 1/8 Failed
5.You want to rent equipment for this activity in the future and want to view rental options.
3/8 Passed 3/8 Found difficulty 2/8 Failed
6. Return to the main activity course page. You would like to find a for-fee-course that has been created for novices.
7/8 Passed 1/8 Found difficulty 0/8 Failed
7. Find the location of this course.
8/8 Passed 0/8 Found difficulty 0/8 Failed
8. Find directions to this course.
6/8 Passed 2/8 Found difficulty 0/8 Failed
9. You would like to know more about the course instructor. Locate information about the instructor for this course.
No pass/fail criteria as most classes did not provide this information, though most thought this was worthwhile information to have.
10. Find out more about the partners that are affiliated with course.
5/8 Passed 1/8 Found difficulty 2/8 Failed
11. Confirm the costs associated with the class.
8/8 Passed 0/8 Found difficulty 0/8 Failed
12. Sign-up for the class using this information:
8/8 Passed 0/8 Found difficulty 0/8 Failed
13. Cancel the free course that you previously sign up for.
2/8 Passed 5/8 Found difficulty 1/8 Failed
14. Find available ___(Course of interest)_________ courses that are being offered in June.
7/8 Passed 1/8 Found difficulty 0/8 Failed
15. You want to the REI homepage; do that now.
8/8 Passed 1/8 Found difficulty 0/8 Failed
Summary of end of session overall feedback Section 1. Summary of responses to the prompt: â€œWould you recommend this site to a friend?â€? All eight participants would recommend the REI website as an online venue for where they could sign up for outdoor activity classes. Most participants found the site easy to navigate even if they were not sure how to complete some of the tasks. However, two participants commented that there were too many steps to get to the class information needed.
Two participants commented that the site would be beneficial for novices, with one stating: “REI is how I learned about the outdoors”. Two participants praised the site for offering a multitude of classes. One participant elaborated, “REI is a great source to learn about the outdoors through classes and articles. Also they provide all the equipment needed to accomplish the outdoor activities”. Most participants had trouble locating free classes even after adjusting location and time ranges. When prompted for her experience, one participant expressed the desire for more free classes. Another participant exclaimed that she would recommend the site but with the caveat that very few free classes would be available, adding that she personally had not thought to look for free classes through REI previously. When prompted regarding commuting to REI classes, one participant stated that she would find carpooling to be a “sketchy” option but would find a shuttle to be safe. However, most participants found that commuting to a site would not encourage or discourage them from taking an REI course.
Section 2. Summary of responses to deep dive follow up questions (some information has been included in previous section) Two participants commented on being able to view closed classes and beginning the signup process for closed classes. One participant stated that it was unnecessary to show closed classes in the filter and that only available class should be viewable. Another participant found it very frustrating to be able to view closed classes that did not have a waiting list. One participant commented that the Filter should default to all activities, after experiencing the filter defaulting to “bicycle” activities whenever she looked for a new course. This participant also found it frustrating to be unable use the site to see which classes one is signed up for.
Section 3. Summary of insights from tasks Most participant found the cancellation policy information. However, five participants commented that they would prefer to have the ability to cancel through the site. Two participants were able to walk through the process of accessing their email and canceling through their email account. All participants had difficulty finding the rental page. Three participants first looked at the footer panel on the home page. Five participants utilized the REI site search option. Two participants commented that if they were at home, they would leave the site and search through Google.
All participants filtered and found the activity course level by reading the bulk text course description. When prompted to open the course for further details and confirm the course level. Participants continued to read the body text. Three participants later located the right panel skill level description. Many participants concluded that they would be unable to sort by skill level. All participants located the directions to their activity course. However, participants arrived at different types of directions--either a map or no map. One participant found directions without a map expressed that he would like to see a map. Another participant initially began entering her address into the "locate a store" field because she thought it was the directions page. One participant tried to select a bus option but could not find one. Most participants went to the "more information" to find detailed information regarding an instructor, but almost all classes provided no instructor information. When asked if this information would be important, five participants said that they would like to see the instructor’s credentials especially if they were spending the day with or going into wilderness with the instructor. Two participants stated that they would only seek instructor information if the course level was intermediate or advance. Most participants found the location easily, but had to input the location several times while searching the site. In addition, the location finder jumps from Puget Sound to Washington to a specific city. One participant had a great insight: “The computer already knows your location, why can't it predetermine your location and jump into the activities/classes.” While most participants located the affiliate partners link and website, these participants also found this information confusing and were not able to connect its relevance witch the course. In regard to affiliated partners, one participant found it to be "reassuring to know that the course is affiliated with professionals". One participant found it “annoying” that when attempting to return to the homepage, if she did not remove a partially signed up of class, the site would repeatedly return her that previous course page.
Worksheet: Writing a User Research Participant Screener Remember: Test your screening questions with team members and revise!
Who do you want to talk to?
What exact criteria will identify the people you want to talk to?
Mix of genders
None; at least one of each
What screening questions will you ask? (Questions shouldn’t reveal “right” answers.)
None; 23-60 y.o. Mix of ages
Mix of current commuting methods
At least one of each: Bus, Biking, Auto
Mix of current employment statuses
Full time with flexible schedule; full time with inflexible schedule with only weekends free; Part time (?); Student (?)
use of the internet for information gathering at least once per week
Who do you want to exclude?
What exact criteria will identify the people you want to exclude?
What screening questions will you ask? (Questions shouldn’t reveal “right” answers.)
REI and other sporting store employees
Recruiting Requirements http://1drv.ms/1dmyj8G
● Recruit 10 participants (1 pilot, 6 regular, 3 backup) ● Participant requirements: Mix of male and female participants ● Ages 23- 60 y.o. ● Mix of commuting methods __________________________________________ Testing occurs March 1st- 5th. Location: ______. Is participant available at ________time? March 1
March 2 Morn
March 2 Afternoon
March 2 Evening
March 3 Morn March 3 Afternoon March 3 Evening March 4 Afternoon March 4 Evening March 4 Morn March 5 Afternoon March 5 Evening Thank you for taking the time to fill out our survey. It will take you 5 minutes or less to complete this 11 question survey. After you have completed the survey, we will automatically be notified and will follow-up with you with either email or text message. Heidi Schindler, Ivan Koveshnikov, Hasani Burns, Nikki Pete
1. First, let’s make sure you are available for one of our testing dates. We will need 45 minutes of your time in person at UW’s Main Campus Seig Hall. Please choose your prefered two dates. a. Tuesday, February 26, between 6-8pm b. Wednesday, February 27, between 6-8pm
c. Friday, February 29, between 6-8pm d. Saturday, March 1, between 9am-6pm e. Sunday, March 2, between 9am-6pm 2. Gender a. Female ___ b. Male ___ 3. Age group a. 23-29 ___ b. 30-39 ___ c. 40-49 ___ d. 50-59 ___ e. Other, please enter your age ___ 4. Do you work for any of the following? a. Ebay b. Craigslist c. Columbia d. MYUW Website e. Northface f. REI g. Amazon h. Yahoo i. Zappos 5. How often do you use the internet for information gathering? a. I rarely or never use the internet ___ b. 1 -2 times daily___ c. 3-5 times daily ___ d. greater than 6 times daily ___ 6. Please select how proficient you feel using a computer to browse the internet? Very Proficient Somewhat Proficient Not Proficient 7. Please select how interested you are in recreational outdoor activities (ex: biking, hiking, kayaking, snowshoeing, rock-climbing, camping, etc) Very Interested Somewhat Interested Not Interested a. Have you ever used the internet to search for activities to do outside? Please describe. b. How often do you search online for outdoor activity sources?
i. ii. iii. iv. v.
Once per week ___ Twice a month ___ Once a month ___ Never ___ Other ___
8. How interested would you be in completing outdoor activities by yourself? Very Interested Somewhat Interested
9. How interested would you be in completing outdoor activities with other people? Very Interested Somewhat Interested Not Interested 10. Please circle/highlight which of these websites you visit frequently a. Ebay b. Craigslist c. Columbia d. MYUW Website e. Northface f. REI g. Amazon h. Yahoo i. Zappos 11. Please provide your contact information. This information will be utilized solely for this study, along with your name and information will stay completely confidential. a. Name b. Cell number: Can we call you on this number? Y/N Can we text message you on this number? Y/N c. Email Thank you for completing the survey. We have been notified that you have completed the survey and will be contacting you by your prefered method. If you have any questions, please do not hesitate to email Heidi Schindler at email@example.com. Heidi Schindler, Ivan Koveshnikov, Hasani Burns, Nikki Pete
Final Findings Presentation Deck
NDA REI, Inc. Research Project Participation Form (NDA) Thank you for deciding to volunteer in for a REI, Inc. research project. The research project will consist of one or more activities and/or sessions to learn about your interaction with technology, products, or services provided by REI, Inc. Please note that you have no obligation to participate and you may decide to terminate your participation at any time.
Materials and Feedback We ask for your permission to record, document, or transcribe your research session in any format and media (“materials”). You may also provide suggestions, comments or other feedback (“feedback”) to REI, Inc. with respect to the research project. Feedback is entirely voluntary. REI, Inc. shall be free to use, disclose, reproduce, license, or otherwise distribute, and exploit the Feedback and Materials. REI, Inc. shall own all information, software, documentation, and any materials in connections with the research project. Confidentiality The research project is confidential and you agree to : (a) never disclose it to anyone else (unless a court or government orders you to); and (b) keep it secret as you would your own confidential information. REI, Inc. does not consider all information about the research project to be confidential. If you legally received information from someone other than REI, Inc., and that person was entitled to share the information with you and did not obligate you to keep it a secret, you do not need to keep that information secret. The same holds true for information REI, Inc. disclosed to you or someone else without obligation to keep it secret or information that you develop independently. Your Authority to Participate You represent that you have the full right and authority to sign this form, and if you are a minor that you have the consent (as indicated below) of your legal guardian to sign and acknowledge this form, and you will not disclose to REI, Inc. any non-public information, whether yours or a third party’s without notifying REI, Inc. in advance. In addition, you agree to release REI, Inc. and its affiliates, and clients from any and all claims that you may have now or in the future related to your participation in the research project. Please confirm your acceptance by signing the bottom of this form. On behalf of REI, Inc., we thank you for your contribution and look forward to your research session.
____________________________________________________________________________ Name (print) ____________________________________________________________________________ Address ____________________________________________________________________________
Published on May 8, 2014