Issuu on Google+

Qualitative User Research Plan: AXIOM New User Experience Launch Prepared by Dianna Miller Senior Researcher, User Experience Design October 26, 2007

Rearden Commerce

1


Table of Contents INTRODUCTION ............................................................................. 3 Goals ................................................................................................ 3 Methodology .................................................................................... 3 Research Team ............................................................................... 4 Rearden Recruiting Support ....................................................... 4 Research Lead / Work Observation Moderator......................... 4 Work Observation Team ............................................................. 4 Extended Team ........................................................................... 4 THE VALUE OF QUALITATIVE RESEARCH ............................. 5 Contextual Inquiry ....................................................................... 5 Usability Testing .......................................................................... 5 Value To Rearden Product Development.................................. 5 Value To American Express AXIOM Customers ...................... 5 SCHEDULE ..................................................................................... 6 RECRUITING STRATEGY............................................................. 9 Recruiting Goals .......................................................................... 9 Geography ................................................................................... 9 End User Qualifications ............................................................ 10 Process ...................................................................................... 10 VISIT PROCEDURE ..................................................................... 11 Pre-brief (10 minutes) ............................................................... 11 Introductions (15 minutes) ........................................................ 11 Contextual Inquiry Session (90 Minutes) ................................. 11 Answering End User Questions (15-20 Minutes).................... 12 Debrief (30 minutes).................................................................. 12 FOCUS OF INQUIRY ................................................................... 12 General Focus Questions ......................................................... 12 Focus Questions by Store ........................................................ 13 FINAL DELIVERABLES .............................................................. 14

Rearden Commerce

2


Introduction This document is intended for the Rearden Commerce and American Express internal teams and is not intended for external distribution. We understand that usability of a product during the first few exposures affects both adoption and brand perception. To insure an on-going, positive user experience, this qualitative research study, conducted during the AXIOM New Experience Launch (Jan 10–Feb 18, 2008), will be aimed at checking AXIOM usability while learning about who our users are and what context-specific factors that affect their experience of the product. Participants will be recruited from American Express’ customer base. The general focus of the work observations will be: •

Usability issues that arise during normal use of the new AXIOM interface

End users’ work contexts in which AXIOM is used, daily task flows that include AXIOM booking

Corporate culture, social ecology: the roles, responsibilities, and daily interactions of end users

End users’ behaviors, motivations, needs, and pain points

GOALS We are requesting to visit AXIOM end users during the first two weeks after customer rollout in order to collect detailed information about the user experience in transition. Findings from the data will be applied to feature enhancements and refinements to the user interface in future AXIOM releases. To this end, we have three specific goals: •

Observe and record usability issues users experience with the introduction of the new interface.

Collect qualitative data about end users for user modeling (persona/scenario) refinement

Collect qualitative data for opportunity assessment across all AXIOM services

METHODOLOGY To accomplish our goals, we propose to conduct a series of contextual inquiries (work observations) with end users who have a need to book services offered by the Rearden Commerce Personal Assistant. These inquiries will be performed at the customer’s site and involve an apprenticeship session where the participant teaches the research team how he does his job while he is engaged in the act of doing it. The session will last between 1 and 2 hours to give us time to more deeply understand the nuances of the participant’s work. To see trends emerge, we will aim to visit 24 participants total (no less than 18). To provide the Rearden Products team with more complete picture of AXIOM end-user diversity, we propose to recruit outside the Bay Area and the Technology sector.

Rearden Commerce

3


RESEARCH TEAM REARDEN RECRUITING SUPPORT For questions on recruiting strategy, deadlines, and insure American Express guidelines for customer approvals are met, please contact: Key contact: Debra Marrazza Senior Activation Manager debra.marrazza@reardencommerce.com Backup contact: Meredith Mortimer Director of Customer Marketing meredith.mortimer@reardencommerce.com RESEARCH LEAD / WORK OBSERVATION MODERATOR To make customer or end user introductions to the Work Observation Team or for questions regarding scheduling and research protocol, please contact: Dianna Miller Senior Researcher, User Experience Design dianna.miller@reardencommerce.com phone 650-212-8418 WORK OBSERVATION TEAM This team will be made of the Work Observation Moderator, one designer and one product manager. The User Experience Design Team and Products Team may substitute their participants, but the moderator will remain the same across all visits to afford consistency and a global view. On some visits, Debra Marrazza or Meredith Mortimer will attend in place of a designer or PM to observe. Team members are present to observe and take notes (see Visit Procedure for more information). EXTENDED TEAM The following people will be kept informed about project updates. If you would like to be kept informed and you are not on this list, please e-mail Dianna Miller. At Rearden Commerce: Aaron Greco Adrian Wood Debra Marrazza Meredith Mortimer Jen Wolf Phil Dean

Rearden Commerce

At American Express: Jessica Humphry Robin Greenwood Brenda Brooks

4


The Value of Qualitative Research At Rearden Commerce, we are committed to building a product that is useful to people in the real environments they live and work in, one that meets both expressed and unexpressed needs. There are two central methods we use to accurately developing base criteria for what AXIOM should do and how it should be structured: CONTEXTUAL INQUIRY From Hugh Beyer and Karen Holtzblatt’s Contextual Design: Defining Customer-Centered Systems: “Contextual inquiry is a field data-gathering technique that studies a few carefully selected individuals in depth to arrive at a fuller understanding of the work practices across all customers. Through inquiry and interpretation, it reveals commonalities across a system’s [user] base” Originally a research method used by ethnographers, this field interviewing technique has been widely adopted by teams using a user-centered design (UCD) approach for software development. It is a discovery process than an evaluative process and is based on the idea that (1) understanding the context in which a product is used (the work being performed) is essential for elegant design, and (2) the user is a partner in the design process. The session focuses on the user showing and telling about what is important to him/her, rather than what the product developer believes is important. USABILITY TESTING While usability testing, as an evaluative process, provides critical user feedback to Rearden’s iterative development process, it is usually conducted in a lab where users perform predefined tasks. With this research project, we have the opportunity to combine usability testing methods with contextual inquiry protocols. This gives us the opportunity to evaluate AXIOM New Experience usability within the user’s real day-to-day context, focusing on tasks that are important to each user. The usability testing aspect of this research will focus on discovering usability issues that may directly or indirectly impact adoption. VALUE TO REARDEN PRODUCT DEVELOPMENT Designers and product managers benefit considerably from visiting and observing customers. It is a rare chance to “walk in the shoes” of someone whose day-to-day work experience is often markedly different from their own. The job of Rearden interaction designers and product managers is to anticipate what the user is thinking in the moment they approach the AXIOM interface and design for it. This one-to-one meeting often provides enlightening insight into workflows and motivations they could not completely construct from survey, support call, or focus group data alone. VALUE TO AMERICAN EXPRESS AXIOM CUSTOMERS This is a unique chance to demonstrate to customers that we are serious about understanding their needs. Previously participating employees have indicated they welcome the opportunity to be heard by and directly influence the people who are responsible for designing a tool they use regularly.

Rearden Commerce

5


Schedule The most important time to collect user impressions of the new interface is during and immediately after initial rollout when users are approaching the new interface for the first time or have had only a few exposures. To successfully recruit, schedule visits, and conduct the work observations, both Rearden Commerce and American Express teams understand the following dates must be met. US Regions named under travel waves are suggested, subject to change based on customer availability.

Week of

Mid-market customers

Large market customers

NOV 05 NOV 12

Nov 16, 2007 Mid-market customers identified by American Express

Responsible: Jessica, Robin, Brenda Approver: Debra, Dianna Support: Meredith Consult: Meredith, Debra Inform: Dianna

NOV 19 NOV 26

Nov 26, 2007 Large market customers identified by American Express

Responsible: Jessica Approver: Debra, Dianna Support: Meredith Consult: Meredith, Debra Inform: Dianna

DEC 03

Dec 03, 2007 Responsible: Mid-market Jessica customers approve Approver: the work observation AmEx Customer

Rearden Commerce

6


Week of

Mid-market customers visits and identify end-users

Large market customers Support: Debra, Meredith Consult: Dianna Inform: Dianna

DEC 10

Dec 14, 2007 Research Team completes scheduling with midmarket end users for visits Jan 10–Jan 31

Responsible: Dianna Approver: AmEx Customer AmEx GM

Dec 14, 2007 Large market customers approve the work observation visits and identify end-users

Support: AmEx GM, Debra, Meredith

DEC 24

HOLIDAYS

DEC 31

Support: Debra, Meredith

Inform: Dianna

Inform: Jessica, AmEx GM Dec 20, 2007 Research Team finishes booking travel to mid-market customers for Jan 10 –Jan 31

Approver: AmEx Customer

Consult: Dianna

Consult: AmEx GM

DEC 17

Responsible: Jessica

Jan 04, 2008 Research Team completes work observation scheduling with large market end users for Jan 31–Feb 18

Responsible: Dianna Approver: AmEx Customer AmEx GM Support: AmEx GM, Debra, Meredith Consult: AmEx GM Inform: Jessica, AmEx GM

Rearden Commerce

7


Week of

Mid-market customers

JAN 07

Large market customers Jan 07, 2008 Research Team books travel to large market customers for Jan 31 – Feb 18

JAN 14

Mid-Market Travel Wave 1 Work observations, Southeastern US Jan 14 –Jan 18

JAN 21

Mid-Market Travel Wave 2 Work observations, Midwestern US Jan 21– Jan 25

JAN 28

Data Analysis

FEB 04

Large Market Travel Wave 1 Work observations, Southeastern US Jan 14–Jan 18

FEB 11

Large Market Travel Wave 2 Work observations, Mountain/Southwestern US, Jan 21–Jan 25

FEB 18

Data Analysis

FEB 25

Data Analysis

MAR 03

Data Analysis

MAR 10

Delivery of Findings

Rearden Commerce

8


Recruiting Strategy Participants will be recruited through the American Express customer base. Participant companies must be AXIOM customers who are part of the New Experience launch, Phase 2 or Phase 3 rollout, and midmarket or large market customers RECRUITING GOALS The Rearden Research Team’s goal is to record details concerning use of AXIOM across a broad range of contexts. Because the data collection is qualitative, we will be focused on uncovering patterns of behavior, perception, and motivation that are not affected by context. Seeing users in a greater range of contexts allows these universal experiences to emerge more clearly. Recruiting “atypical” customers is also beneficial. Mid-market yield: •

12 end users

end users should be dispersed among a minimum of four companies

aim to recruit a 50/50 split between delegates and self-bookers

Large market yield: •

12 end users

end users should be dispersed among a minimum of four companies

aim to recruit 70% delegates Recruiting companies where Shipping and Conferencing (and other long tail stores) are turned on

Functional changes were made in the new interface to the Shipping and Conferencing stores. Because of this, we will test Shipping and Conferencing with end users. Additionally, we will test and inquire around use of other long tail services in relation to travel. Recruiting companies where these services are active will render findings of immediate need to Rearden development efforts. GEOGRAPHY The user’s workflow may vary in potentially important ways in different industry sectors and geographic locations. Here are the important geographic variables and how we intend to address them. SF Bay Area: We will visit companies in the Bay Area for convenience and expediency, however local recruiting should be focused on companies that are not in the tech sector (manufacturing is fine). One local company will be recruited locally to serve as a pilot visit to take place before to Jan 10. North American Large Cities: Because the Bay Area is a technology-saturated market, there is risk that visits we conduct here will be skewed towards participants who are more technically savvy and more steeped in tech culture. To avoid this, we intend to conduct visits in 2 travel waves per rollout phase to other North American markets (See schedule for travel waves).

Rearden Commerce

9


END USER QUALIFICATIONS •

All participants must book and manage services for themselves or others.

Participants must be willing to host us at their site for 2 hours for observation at their desk/work site. Part of this time will include an interview with the participant. (Participants may continue to work while we are there.)

At the time of our meeting, all participants should have booking to complete while we are present or have recently booked a travel or long-tail service.

PROCESS The Rearden Research Team assumes the following recruiting process: •

American Express will contact customers for approval and agreement to participate.

American Express or Rearden Activation (through Debra Marrazza) will make introductions between the Research Lead (Dianna Miller) and the customer’s Account Manager and customer’s internal point of contact or end user(s), which ever is most appropriate.

The Rearden Research Lead will be responsible for checking with the Account Manager to understand any relevant background on the customer.

The Rearden Research Lead will reach out by email or phone call to the customer point-ofcontact and end users to

o

introduce herself and team

o

schedule visits at the end users’ convenience

o

answer questions

o

set expectations about the visit.

In accordance with the Marketing Research Association’s Code of Marketing Research Standards (http://www.mra-net.org/), participants will be told at the recruitment stage: o

Recording of the session will take place will take place for the purposes of notetaking, analysis, and summary of findings only.

o

All customer and end user proprietary information will be kept strictly confidential.

o

Information will not be revealed that could be used to identify end users without proper authorization.

o

No raw data (video, audio, or notes) will be used for distribution beyond the Rearden Products Team.

o

No recorded material will be used by Rearden Commerce or American Express for marketing purposes or social research purposes beyond the scope of this study.

o

Video and audio used to summarize findings will not be distributed beyond the Rearden Products Team or American Express stakeholders of this study.

Rearden Commerce

10


Visit Procedure Below is the outline of the protocol for a single visit. Under most circumstance, three to four visits maximum can be scheduled per day (subject to travel time). PRE-BRIEF (10 MINUTES) Occurs before the team arrives at the customer’s site. •

The Research Lead will brief the other two members of the work observation team on any background collected during the recruitment process. This information will include: o

An overview of the company the participant works for.

o

Where the participant falls on the market / role matrix

o

What types of product(s) to company makes

o

Who the participant works with and in what capacity

Each team member reviews his/her note-taking or videotaping role for the session. (Team members are given a quick reference sheet that includes key questions and areas of focus.)

As a service to the end user, one team member will agree to field any how-to or support questions at the very end of the session.

INTRODUCTIONS (15 MINUTES) •

Introduce the work observation team to the end user and other representatives from the customer.

Move immediately to the participant’s place of work and get settled.

Verify that it is OK to record the session and turn on equipment.

Explain the contextual inquiry procedure. Cover the following: o

We are here to learn how you do your job. (Establish master-apprentice relationship)

o

We’ll observe and interrupt to ask a few questions. If something comes up and you need to attend to your job, just let us know.

o

We’ll be concentrating on how you currently book travel and services. We’d like to observe how you use the new AXIOM interface.

o

If it helps, imagine we’re new to the company, you’re training us to do a job similar to yours, and (where appropriate) how AXIOM is a part of what you do.

CONTEXTUAL INQUIRY SESSION (90 MINUTES) The participant should have relevant work to do on the day that we apprentice ourselves to him/her. •

The user researcher facilitates and takes light notes.

One work observation team member is designated to take detailed notes (including timestamps for key incidents).

The other member will be the designated videographer.

Rearden Commerce

11


ANSWERING END USER QUESTIONS (15-20 MINUTES) During the contextual inquiry session, the end user often poses questions to the facilitator. Because the objective of contextual inquiry is to observe the user in a circumstance as close to what they would experience if they were using AXIOM alone, we do not answer questions or prompt their action or responses during the session. However, we recognize that we are providing the end user with a unique opportunity to ask questions and learn about AXIOM “straight from the source.” In repayment for the time and feedback they have given us, the product manager or designer will answer questions, gives tips, or introduce features the user may not know about. If a bug or customer support issue is identified, the assigned team member will follow up with the Account Manager to insure it gets noted by the appropriate party and will be sent through proper channels. DEBRIEF (30 MINUTES) This is a required session that occurs after work observation team has left the customer’s site and usually takes place over lunch, en-route to the next customer’s site, or on the way back to the office/hotel/airport. While the information is fresh in our minds, the work observation team will •

cover task success rates, behaviors on-task, user perceptions, notable quotes

quickly record interesting incidents that stood out to team members

build consensus on behavioral variables to identify patterns or anomalies

note any fixes or market opportunity ideas that specific behavior suggests

agree to revisit all of these in an analysis or ideation session, so we will avoid any in-depth discussions of issues unless time permits.

Focus of Inquiry The following focus questions are used to frame the research inquiry (we do not directly pose these questions to participants). GENERAL FOCUS QUESTIONS Refinement of user modeling data: •

What are the user’s attributes?

Who does the user interact with at various stages?

What motivates the user? Needs, pain points, daily tasks, behaviors, regulations, social influences?

Rearden Commerce

12


Identify scenarios, opportunities: •

How and when do users choose and book services, and complete service transactions?

Where do users need or want to transact?

When does interaction happen offline (or off the desktop)?

Identify usability issues: •

How does the new interface affect user understanding of AXIOM?

Which area of the new interface are easier to understand/ navigate?

Are there areas that are more difficult to use or navigate? What factors are contributing?

What is user’s perception of time on task? Has it decreased, increased?

Identify concepts, integration opportunities: •

What are users’ mental models for each service? Have they changed with the new interface?

Do they notice stores or features for the first time?

What other tools do they use in addition to AXIOM in the service booking workflow? (Sites, applications, phone services, mobile, PDAs, etc.)

What are the assumptions about value proposition (by store)? Have they changed?

How do they perceive Rearden interface concepts?

FOCUS QUESTIONS BY STORE Rearden product managers have identified the following questions as key to store development over the next 18 months. (This is not a complete list.) Travel & Long Tail Services: •

To what degree do users understand travel industry terms & concepts?

Coordinated travel: what are patterns of behavior, booking flow when more than one traveler is booking or being booked?

Coordinated travel: what are in-transit patterns of behavior, preferences, needs, and pain points when two or more people travel together?

Search behavior: what factors are most important when booking hotels?

Service layering: how and when does it occur? What are the primary scenarios?

How do users apply and track preferences?

How/when do users make and track changes? When do they use a mobile device for this?

What are need and pain points around multi-city booking?

What is the perception of car services by city/region?

Rearden Commerce

13


Conferencing: •

Do users understand what web conferencing is?

What are the factors present in corporate culture that contribute to understanding and use?

What are the barriers to adoption? Are users concerned about cost? To what degree?

Package Shipping: •

Which shipping means, services do users turn to?

Do they understand industry terms & concepts?

Which features are primary?

What are the primary pain points?

How last minute is last minute?

Is compliance a value proposition? User privacy?

First market vs. second market inventory?

Do users want support of company inventory?

Are purchases for staff or clients?

Do they prefer physical delivery or e-tickets? What are the circumstances?

Are users interested in searching offline inventory?

What is the importance of finding a restaurant vs. inviting people?

What are primary user behaviors, booking patterns?

Are bookings for staff or clients?

What is the importance of recommendations? From whom?

Are users interested in searching offline inventory?

Is there a critical number of listings that affects trust or perception?

Events:

Dining:

Final Deliverables A summary of findings presentation and report of findings will be delivered to Rearden Commerce and American Express stakeholders the week of March 10.

Rearden Commerce

14


AXIOM research