Page 1

! ! ! !

!

School&of&Informatics,&IUPUI&

OneTravel&Team& I543&Usability&and&Evaluation&Method&&&& Final&Report&

Instructor:**Michael&Wilson* Team*Members:* Aeshvarya&Verma& Powit&Rungsangthiwakorn& HsiaoNWen&Wu& Reecha&Bharali& 12/10/2012! ! ! !


Content* List of Figures ................................................................................................................ I List of Tables ............................................................................................................... III 1.

Executive Summary ............................................................................................... 1

2.

Introduction to the Testing ..................................................................................... 2

3.

4.

5.

6.

2.1.

User Demographics ...................................................................................................2

2.2.

Testing Procedure .....................................................................................................2

2.3.

Task Conditions ........................................................................................................3

Overall Results ....................................................................................................... 5 3.1.

Success rates of across all tasks ................................................................................5

3.2.

Time on tasks for all tasks.........................................................................................7

3.3.

Post-task ratings across all tasks ...............................................................................8

3.4.

System Usability Scores (SUS).................................................................................9

Usability Testing Results ..................................................................................... 11 4.1.

Homepage ...............................................................................................................11

4.2.

Loading page ...........................................................................................................13

4.3.

Search result page ...................................................................................................14

Synopsis ............................................................................................................... 21 5.1.

Usability Problems ..................................................................................................21

5.2.

Recommendation ....................................................................................................22

Appendix .............................................................................................................. 24 6.1.

User Testing Package ..............................................................................................24

6.2.

Quantitative Data ....................................................................................................31

! ! ! ! ! ! ! I! !


List*of*Figures* Figure 1. Accumulated success rates across tasks. ........................................................ 6! Figure 2. Checkout button in OneTravel wesite............................................................ 6! Figure 3. Checkout path in Hipmunk website. .............................................................. 6! Figure 4. Average time on tasks among OneTravel and Hipmunk. .............................. 7! Figure 5. Overall perceived difficulty across tasks among OneTravel and Hipmunk .. 8! Figure 6. Overall post-task results for OneTravel. ........................................................ 9! Figure 7. SUS score of each user for OneTravel and Hipmunk .................................. 10!

II! !


!

List*of*Tables* Table 1.Success rates across all tasks and users in OneTravel and Hipmunk sites. . 5! Table 2.Overall score of post-task questionnaires in OneTravel. ............................. 9! Table 3.SUS scores of each participant. ................................................................. 10!

III! !


!

1. Executive Summary OneTravel is a travel reservation website. The primary purpose of the website is to provide a single point reservation service to different travel booking, for example flight booking, hotel booking and car rental. The customers can reserve these services individually (Flight Booking, Hotel Booking or Car Booking) and they can also book all 3 of these services together (Vacation Booking). The latter is the main area that is being evaluated. Expert Inspections were conducted on the website to preliminary identify the issues. Many of which was confirmed with real users testing the website. In addition, Hipmunk were tested side-by-side to illustrate the performance of OneTravel against the competing website. In the test, users are given basic task such as Finding Flight with a specific time, Finding Hotel with a specific location. Success rate, time taken to complete and other data, were collected. The result shown that OneTravel underperformed when compare to Hipmunk in term of success rate, time-on-task and perceived difficulties. Results from observation have confirmed that serious usability, uncovered during inspection, exist within the website. The findings show serious problems in 3 main basic functions, Finding information in the website, Comparing hotel and Comparing flight. Finding information in the website: 9 out of 10 users never used or notice the Vacation booking option. This clearly represents two important issues due to these two reasons. There are too many competing pieces of information and “Vacation� is not an adequate label. Comparing Hotels with location constraint: The current structure of the website make it difficult to identify the location of the hotel, let alone comparing them. Users had to check them one by one and use additional tools, such as Google Maps, to help them complete the task. Comparing Flight: All users have difficulty when choosing and comparing flight. OneTravel providing unrequested information, such as Near-by airport, prevents user from finding the desire information. In addition, booking flight that depart in a particular time, has a serious issue because of the structure and mechanism of the website.

1! !


!

2. Introduction to the Testing Expert Inspections and user testing were conducted on OneTravel.com. The main focus of this test is in Vacation booking. Careful and extensive analysis through usability testing has revealed problems with the websites navigation, layout, information architecture and visuals. The user testing was conducted to better understand the heuristics. This is both to verify the findings and possibly uncover new issues that the users actually face while using the website. Since OneTravel.com is a commercial website, Competitive Analysis was conducted to illustrate OneTravel performance against another competing website. At the time of analysis, Hipmunk is one of the best travel reservation website in the market and therefor selected for this analysis. With an emphasis on finding the usability issues associated with Onetravel.com tasks were designed to cover important aspects of the website through the results from the Expert Evaluation. These tasks are based on crucial functions such as finding information in the website, finding and comparing flight and finding and comparing hotel. Both the websites were critically analyzed and tested to make sure that all the tasks could be completely completed successfully within a respectable timeframe. Detailed test instruments and plan can be found in appendix section 6.1.

2.1. User Demographics 10 participants with following demographics were recruited. • 5 female and 5 male participants • English Fluency ranged from Fair to Fluent. • 6 hours and higher Internet usage. • Online Experience: 6 Competent and 4 Beginner 4 users. • No Experience with neither OneTravel.com nor Hipmunk.com • All have used a travel reservation website

2.2. Testing Procedure The study was conducted with one facilitator and one observer during the active user session. 5 tasks, in 2 scenarios, were given to the user to complete. In order to avoid order effect in the result, 5 users start the experiment with OneTravel.com and another 5 users start with Hipmunk. In this experiment both qualitative and quantitative data were collected. Quantitative data consist of, success rate, time-on-task, perceived difficulties and System Usability Score (SUS). Qualitative data consist of observation, think-a-loud protocol, post task interview and post-test interview. In the testing session, these steps are being followed: 1. User is given a pre-test questionnaire. 2. Facilitator begins the recording and start the introduction script 2! !


!

3. Facilitator begins asking warm up questions 4. Facilitator gives user the Task Sheet. 5. Facilitator read the Task to the User 6. User begins Task 1.1 for website A 7. After user finished the task, he/she is give a post-task questionnaire 8. Facilitator begin a short-interview about the task completed 9. Repeat step 5 to 8 for all tasks 10. After completing all the task for website A, user is give a SUS questionnaire 11. Repeat step 5 -10 for website B 12. Facilitator conduct Post-Test Interview 13. Stop the Recording and thank the participant.

2.3. Task Conditions Tasks were designed to foster different needs from 2 different scenarios, cost conscious customers and high preference users. There are many severe usability issue within these discover within these area of function. Scenario 1: First scenario represents a cost conscious type of trip searching. In this scenario, user wants to find the cheapest trip possible to New York. To achieve this user will have to find cheapest flight and hotel and try to compare them in multiple days. This is to test the workflow of the process as well as the main function of the website Task 1.1 Find the cheapest Flight and Hotel Task 1.2 Continue to make payment to the trip Task 1.3 Compare trip in different date Scenario 2: The second scenario is designed based on the basic needs of the users, the ability to select a convenient departure time of a flight and the ability to select hotels based on the location. Task 2.1 Find Flight that depart at a specific time Task 2.2 Find Hotel that locate in a specific location

3! !


!

2.4. Severity Rating The severity was calculated on a 4-point scale, 4 being a highly severe issues and 1 being a minor issue. The severity was determined by the error rates, possible recovery, recovery time and, most importantly, impact to the users. 1 = Cosmetic problem only: little or no impact to the user 2 = Minor usability problem: fixing this should be given low priority 3 = Major usability problem: important to fix, so should be given high priority 4 = Usability catastrophe: imperative to fix this before product can be released

4! !


!

3. Overall Results 3.1. Success rates of across all tasks For all task completed by the participant, we defined 1 for success and 0 for failure. Table 1 shows success rates across all tasks among users in two websites. Figure 1 depicts accumulated success among tasks for two website. From these results, we found that Hipmunk has higher success rate in task 1, 3 4 and 5. These data reveals that users fail complete most critical tasks in OneTravel. Figure 1 shows that task 1 of OneTravel has the lowest success rate. This is because when finding flight and hotel together, users have to use ‘vacation’ button. However, most of users do not know the meaning of this button. They regard this label indicated travel related with traveling and relax. OneTravel website only get higher rate in task 2 which is to find a way to checkout purchases. The phenomenon was due to the design layout for payment process. Since OneTravel uses conventional pattern such as a button for checkout and a form for filling out, users are easily to recognize the section that they could perform this task. On the other hand, Hipmunk uses a dialogue box and direct users outside of Hipmunk. Figure 2 shows checkout path in OneTravel and figure 3 shows checkout path in Hipmunk website.

Table 1.Success rates across all tasks and users in OneTravel and Hipmunk sites. (O for OneTravel, H for Hipmunk) Task1 User

Task2

Task3

Task4

Task5

Find Flight and Find the how to Find Flight w/ Compare Price Hotel pay specific time O H O H O H O H

Find Hotel w/ specific location O H

P1 P2 P3 P4

0 0 0 0

1 1 1 1

1 1 1 1

1 1 1 0

1 0 0 1

0 1 1 1

0 1 0 1

1 1 1 1

0 1 0 1

1 1 1 1

P5

0

1

1

1

1

1

1

1

1

1

P6 P7

1 1

1 1

1 1

1 1

1 1

1 1

1 1

1 1

1 1

1 1

P8

1

1

1

1

1

1

1

1

1

1

P9

1

1

1

1

1

1

1

1

1

1

P10

1

1

1

1

0

1

1

1

1

1

Success

5

10

10

9

7

9

8

10

8

10

5! !


!

Number!of!success

Success!Rate!for!Each!Task 12! 10! 8! 6! 4!

OneTravel!

2!

Hipmunk!

0! Find!Flight! Find!the! Compare! Find!Flight! Find!Hotel! and!Hotel! how!to!pay! Price! w/!specific! w/!specific! Bme! locaBon! Task

! Figure 1. Graph showing accumulated success rates across all tasks.

Figure 2. Checkout button in OneTravel website.

Figure 3. Checkout path in Hipmunk website. 6! !


!

3.2. Time on tasks for all tasks In each task, we measure time on task of all users. In order to have correct value, we subtracted loading time form measurements and then calculated average time on tasks. Figure 4 shows comparisons of time-on-task between OneTravel and Hipmunk. From overall results, we find that users spend more time on 4 tasks in OneTravel while compare to Hipmunk. But OneTravel had better performance in task 2. Users consume less time in OneTravel compares to Hipmunk when finding a way to pay. Because OneTravel provides a button at right hand side section, it is a short path for users to pay rather than go to the bottom of page and click next step. However, in Hipmunk, users had to find a way to pay in more effort because the interface is different to conventional way. Thus, they spent more time to figure it out in Hipmunk. Comparing task 3 with task 1, we find that users can learn from tasks. In these two tasks, they performed the search process both on flights and hotels. Moreover, they have to compare prices among two different departure dates. Ideally, it should spend more time on task when users have to perform same process for two times. With learning effect, they can be familiar with the interface and consume less time on second trial. From our results, the average time on task 1 of OneTravel is 8.5 minutes while task 4 is 6.0 minutes. It shows that users spend less time for performing search and compare. It is an evidence that they can learn from tasks.

Time!(min)

Onetravel!vs.!Hipmunk! Average!Time!on!Task 14.0! 12.0! 10.0! 8.0! 6.0! 4.0! 2.0! 0.0!

Find!Flight! and!Hotel!

Find!the! how!to!pay!

Compare! Prices!

Find!Flight! w/!speciďŹ c! Bme!

Find!Hotel! w/!speciďŹ c! locaBon!

Onetravel!

8.5!

0.9!

6.0!

5.7!

5.0!

Hipmunk!

6.1!

1.5!

5.6!

3.1!

2.9!

Figure 4. Average time-on tasks between OneTravel and Hipmunk.

7! !


!

3.3. Post-task ratings across all tasks After each task, we not only conducted interviews but also gave questionnaires to users. Users choose an answer from 1 to 7 in Likert scale. This questionnaire included three questions as following. Q1: How was the overall task? (1 very easy - 7 Very Difficult) Q2: I was satisfied with the ease of completing this task. (1 Strongly Disagree - 7 Strongly Agree) Q3: I was satisfied with the amount of time it took to complete the task (1 Strongly Disagree - 7 Strongly Agree)

In question 1, it asked how user felt about the task they just performed. This question could reveal their perceived difficulty in the task. We compared OneTravel results to Hipmunk. Results are shown in figure 5. We find that there is not much difference between two websites in task 1. Average scores are 4.7 in OneTravel and 4.6 in Hipmunk which represent neutral difficulty. However, in task 2, since the interface Hipmunk is quite novel for users, their perceived difficulty is much higher than Hipmunk. From task 3 through task 5, perceived difficulty in OneTravel all are higher than Hipmunk. From learning effect perspective, comparing task 3, 4 and 5 to task 1, users felt not difficult as beginning. That is to say, they feel familiar with the interface as they performed more tasks.

Average!Score!

Perceived!Difficulty! 7! 6! 5! 4! 3! 2! 1! 0!

OneTravel! Find!Flight! Find!the! Compare! Find!Flight! Find!Hotel! and!Hotel! how!to!pay! Prices! w/!specific!w/!specific! Bme! locaBon!

Hipmunk!

Task Figure 5. Overall perceived difficulty across tasks among OneTravel and Hipmunk

By comparing results of three questions together, we find that the relationship between satisfaction and perceived difficulty. From task ‘Find Flight and Hotel’ and task ‘Find the how to pay’, they are satisfied when they feel the task is easy. But they are not satisfied when they feel the task is more 8! !


!

difficult. Comparing to first task, task 3, 4 and 5 are not as difficult as task 1. Therefore, their satisfactions are higher than first task. Table 2 shows overall scores of post-task in OneTravel.

Average!Score!for!Onetravel Average

8! 6! 4!

Q1!

2! 0!

Q2!

Find!Flight! Find!the! Compare! Find!Flight! Find!Hotel! and!Hotel! how!to!pay! Prices! w/!specific! w/!specific! Bme! locaBon!

Q3!

Task Figure 6. Overall post-task results for OneTravel. Table 2.Overall score of post-task questionnaires in OneTravel.

Find Flight and Hotel Q1 Q2 Q3 4.7 3.1 3.2 5 3 3 6 5 1

Average Median Mode

Find the how to pay Q1 Q2 Q3 2 6 5.9 1 7 7 1 7 7

Compare Price Q1 Q2 Q3 3.9 4.4 4.5 4 5 4.5 6 5 4

Find Flight w/ specific time Q1 Q2 Q3 3.2 4.3 4.4 3 5 5 3 5 1

Find Hotel w/ specific location Q1 Q2 Q3 3.5 4.7 4.2 2 5.5 4 2 6 4

3.4. System Usability Scores (SUS) At the end of each testing session, we gave users a post-test questionnaire. We used system usability scale (SUS) to compared the usability of each website. OneTravel has an average score of all users is 55.3±21.8 in while Hipmunk has 51.8 ± 21.2. Figure 7 shows SUS score of each user. These average scores tell us that users like OneTravel than Hipmunk in general since OneTravel get higher average SUS score. Despite the longer time take for user to complete the tasks; OneTravel received a higher Average System Usability Score from the users. Table 3 lists all SUS scores from users and their first websites in this test. Results show that 7 out of 10 users do not like the website that they began with in this test. This can be inferred that they have to accustom to all tasks and try to learn the process. Once they finished first set, they have a big picture in their mind. Therefore,

9! !


!

when begins with second website, they can handle, feel easier and more satisfied with tasks.

SUS!Score!of!ParFcipants!

SUS!!Score

100! 80! 60! 40! 20! 0!

P1!

P2!

P3!

P4!

OneTravel! 32.5! 52.5! 62.5! 42.5! Hipmunk!

62.5!

25!

82.5!

55!

P5!

P6!

P7!

P8!

P9!

P10!

35!

45!

37.5!

65!

100!

80!

70!

80!

30!

37.5!

32.5! 42.5!

Figure 7. SUS score of each user for OneTravel and Hipmunk

Table 3.SUS scores of each participant. OneTravel

Hipmunk

First Website

P1

32.5

62.5

OneTravel

P2

52.5

25

Hipmunk

P3

62.5

82.5

OneTravel

P4

42.5

55

Hipmunk

P5

35

32.5

OneTravel

P6

45

42.5

OneTravel

P7

37.5

70

OneTravel

P8

65

80

OneTravel

P9

100

30

Hipmunk

P10

80

37.5

Hipmunk

Average

55.3

51.8

STD

±21.8

±21.2

10! !


!

4. Usability Testing Results We have classified the usability testing results based on the web pages they were found. The three different webpages of OneTravel are as following: • • •

Homepage Loading Page Search Result Page

4.1. Homepage Finding 1: The majority of the participants do not notice the "Vacation” booking Problem Description

The label “Vacation” is not clear enough for the user to recognized. This is one of the reasons that the user never actually notice the availability of this funciton.

Severity

4

Recommendation for Improvement

Make the label more obvious e.g. use Flight + Hotel as the label or give a pop up description when user put the mouse over the tab

Relation to the heuristic result

Confirmed with the heuristic finding.

Encountered by participants

1,2,3,4,5,6,8,9,10

Screenshot

11! !


!

Finding 2: Search tab reset to “Flight” every time the user refreshes or comeback to the homepage from the previous page. Problem Description

Every time the user have to come back to the homepage to search again, the tab position will reset to “Flight”. This becomes troublesome especially when user is doing a vacation booking. Due to a similar search parameters many users did not even recognize that they are searching in a wrong type of booking.

Severity

2

Recommendation for Improvement

Remember the user’s last used preference.

Relation to the heuristic result

This is a new finding as compared with the heuristics evaluation.

Encountered by participants

1,2,4,5

Screenshot

Finding 3: OneTravel only allows Single Session. Problem Description

When an expert user would like to do a multiple search, it is likely that they would open search in different tabs. However, OneTravel do not allow this utilization of the website.

Severity

2

Recommendation for Improvement

Provide multiple sessions to a single user.

Relation to the heuristic result

This is a new finding as compared with the heuristics evaluation.

Encountered by participants

12! !


!

Screenshot

4.2. Loading page Finding 4: Load Page Progress is not visible Problem Description

The progress indicator, dots at the bottom of the page, is not visible to the user. User may think that the system stopped or get confused that they are in the wrong place and go back to home page just to find out that there is nothing wrong.

Severity

3

Recommendation for Improvement

Add animation to show progress.

Relation to the heuristic result

Confirmed with the heuristic finding.

Encountered by participants

1,3,4,5,9,10

Screenshot

13! !


!

4.3. Search result page Finding 5: Confusing 4 steps task separation Problem Description

Booking steps are confusing (the user do not know that their flight is preselected, and have trouble finding a way to change the flight)

Severity

2

Recommendation for Improvement

Make it more clearly for the user to change flight.

Relation to the heuristic result

Lowered the severity rating from 4 to 2.

Encountered by participants

1,2,3,10

Screenshot

Finding 6: Page Information Overload Problem Description

There is a lot of information in the page. This causes the user to not be able to find many different important elements within the page. For example, the user can not find the “TripAdvisor” score, can’t find show map button, can't find the Change Flight link)

Severity

3

Recommendation for Improvement

Change the layout to make room for a clutter-free interface with focus on new and specific information.

Relation to the heuristic result

Increased the severity from 2 to 3. The users were totally lost in search of tripAdvisor rating, it’s meaning, change flight button after selecting the hotel.

Encountered by participants

1,6

14! !


!

Screenshot

Finding 7 Checkout button is not clearly visible Problem Description

Checkout button is not clearly visible and lost among the less important information. The user needs to search for it after making his/her choices.

Severity

1

Recommendation for Improvement

Make the button stand out from the other information present on the page.

Relation to the heuristic result

Lowered the severity rating from 4, in heuristic evaluation, to 1. The users managed to search the checkout button.

Encountered by participants

10

Screenshot

15! !


!

Finding 8 Unclickable Breadcrumb Problem Description

Unclickable Breadcrumbs on the search result page, making it hard for the users to move backward/forward between each stem

Severity

2

Recommendation for Improvement

Provide clickable breadcrumbs to the user to ease out their forward/backward movements.

Relation to the heuristic result

Lowered the severity rating from 3, in heuristic evaluation, to 2. The users used the browser back button all the time to go back to the previous page.

Encountered by participants

1,4

Screenshot

Finding 9 Browser Back Navigation does not work properly Problem Description

When pressing the back button, the user will be taken back to the loading page not the page before. For example if they have selected hotel and then select flight. From the flight page if they press back they will go back to loading page instead of hotel.

Severity

2

Recommendation for Improvement

Allow user to go back each step using this browser back button.

Relation to the heuristic result

This is a new finding as compared with the heuristics evaluation. The experts didn’t account for the backward/forward movement of the website in the heuristic evaluation.

Encountered by participants

1,3,4

Screenshot

16! !


!

Finding 10 Confusion between the hotel’s Star Rating and TripAdvisor Problem Description

There is a serious confusion between the hotel’s Star Rating and TripAdvisor user review ratings. Even with the users who already understand the different between them. This is caused by the inconsistency of the availability of filter on the website (only for Star-rating)

Severity

4

Recommendation for Improvement

Clearly differentiate between the Star Rating and TripAdvisor rating and put a legend describing them.

Relation to the heuristic result

This is a new finding as compared with the heuristics evaluation.

Encountered by participants

1,3,4,8, 9,10

Screenshot

Finding 10 No Multiple Hotels map view Problem Description

User cannot view all hotels in the single map. This makes it very difficult for the user to compare location of each hotel in the result. If they would like to compare each hotel then they have to print the map out and then mark the location and price of interested hotels.

Severity

4

Recommendation for Improvement

Add a single map to illustrate the hotel’s location.

Relation to the heuristic result

Confirmed with the heuristic finding.

Encountered by participants

1,2,3,4,5,6,9,10

17! !


!

Screenshot

Finding 11 Difficulties in comparing hotels with 2 or more criterion Problem Description

OneTravel support the sort and filtering result with a single criterion. However, not every users desire to find the cheapest hotel or the most popular hotel. User can also want to select a hotel with 2 criterion, price and users’ rating. In this case, OneTravel cannot support this need.

Severity

4

Recommendation for Improvement

Add a filter or secondary sort function.

Relation to the heuristic result

This is a new finding as compared with the heuristics evaluation. The experts compared the hotels based on the given filters only.

Encountered by participants

1,2,4,6,7,9,10

Screenshot

18! !


!

Finding 12 Unclear label for Selected Search Date, Destination in the search result page

Problem Description

Unclear label for Selected Search Date, Destination in the search result page.

Severity

2

Recommendation for Improvement

Make the Search Date more prominent

Relation to the heuristic result

This is a new finding as compared with the heuristics evaluation.

Encountered by participants

1,3,4

Screenshot

Finding 13 Sort by requested time does not sort by requested time ! Problem Description

Sort by requested time does not sort by requested time in the search result page. There are instances when early departure flights are shown way down the list.

Severity

4

Recommendation for Improvement

Make the sort by requested time option more powerful.

Relation to the heuristic result

Confirmed with the heuristic finding.

Encountered by participants

1,4,5

19! !


!

Screenshot

Finding 14 Unrequested Search Result ! Problem Description

The result page gives the Near-by airport result at the top and not the actual search queried by the user.

Severity

4

Recommendation for Improvement

Only show result from selected airport, or give option for user to hide near-by airport.

Relation to the heuristic result

Increased the severity rating from 2 to 4. The users made the mistakes of selecting the first flight shown in the search result page.

Encountered by participants

1,2,3,4,5,6,7,8,9,10

Screenshot

20! !


!

5. Synopsis 5.1. Usability Problems The findings of the user testing can be summarized in to 3 aspects that OneTravel underperformed when compare to Hipmunk: Finding information in the website, Comparing hotels and Comparing flights. •

Finding information in the website OneTravel has serious problems with how important information is placed in the website and what label is used to hint the user of the function. 9 out of 10 users never used the Vacation booking option due to these two reasons. There are too many competing pieces of information, which make the user unable to make sense of and/or choose to ignore. In addition, “Vacation” is not an adequate label for the functions. Users only notice the meaning of vacation by process of elimination of the 4 available tabs. However they never realize this without being asked by the facilitator.

Comparing Hotels with multiple constraints or location constraint OneTravel perform exceptionally well in the case when user would like to get the cheapest, most popular or best ranking hotel. However when the user have more than one decision criteria such as, wanting to get the cheapest hotel which is still reasonably good quality within its price range. User will then look at both price and TripAdvisor review, however, with the current functionality it is impossible achieve this without having to compare each hotel one by one. There is another major problem when the users want to compare a hotels using location, e.g. want the hotel to be close to downtown. This can be said to be impossible without using other tools. Many users try to use Google Maps to help locate the hotels and compare them individually. In real situation, users are more likely to entirely change the booking website and this is the sign of failure to support users’ goal.

Comparing Flight: Unrequested information and Time Selection Similarly to hotel comparison, there is a major usability issue within flight comparison process. The problem is mainly caused by OneTravel providing unrequested information such as Near-by airport or other airlines even the user specifically select one airline. Instead of providing a flexibility of choice, this prevents users from find the desired flight, let alone comparing them.

21! !


!

Another relating problem is when the user would like to book flight that has a particular time, for example at 10am for both depart and return flight. There is no functionality to support this comparing process. The user will have look at every result in order to find the desire flight. OneTravel also force user to choose depart and return flight at the same time, which make the process even more difficult.

5.2. Recommendation Despite the lower success rate, longer time to finish the task, higher perceived difficulties and lower System Usability Score (SUS). Many users, in the final interview, prefers OneTravel due to then conventional style of e-commerce website. However, many users also notice the ease of use and the ease of learning in Hipmunk. Without any prompt action to these serious issues might cause the user to use the competitor service instead. Here are the general guidelines about the strength, something to maintain in the new design, and OneTravel Strength •

Conventional e-commerce style: Most users feel at ease when using OneTravel because they can predict what will happen and therefore feel safe when using this website. Most users prefers the conventional style therefore when redesigning OneTravel can reconsider highlighting important functions and removing unnecessary information instead of redesigning the entire website. In addition, novels approach will be more likely to be accepted by tech savvy users only.

Allowing Manipulation: Users feel more confident when they can manipulate the result before finding the desire ticket or hotel rather than being pre-selected by the system.

Single Payment: Users agree that buying hotel and ticket together is easier and more convenient.

OneTravel Needs for Improvement •

Information Architecture and Labeling: OneTravel can be improve immensely with a better arrangement of information, for example highlighting or emphasize on important information. This can eliminate the possible confusion that might occurs. Appropriate label will help guiding the users.

Better Cooperation of Functionality: There are many different functionality provided by the OneTravel. However, many times the users are loss of how they can be used together to achieve their goals. OneTravel have to consider what the users’ goals are and how can they easily understand and use the functionality provided. 22!

!


!

•

Removing unnecessary information: Extra information can be useful to some users; however, this might not always be true. By providing an option for users to show extra information when needed, and only show information that the user deliberately ask for will help users find the information that they want.

23! !


!

6. Appendix 6.1. User Testing Package Usability Testing Script

Usability Script Preparation 1 Open Chrome in Private Browsing Mode (CMD + Shift + N) 2 Open OneTravel or Hipmunk depending on the order 3 Open Silverback Software (Recording should start when you start saying the introduction 4 Minimize all the Webpages Activity Outline 1 Start Introduction Script 2 Scenario 1 Website A 3 Scenario 2 Website A 4 SUS Questionnaire 5 <Repeat Scenario step 2 - 5 for Website B> 6 Final Interview

1. Introduction <Start Silverback Recording> Welcome ● Thank you for agreeing to help us! My name is _______________, we are students in the school of informatics and this work is for our class project. ● Today, we would like you to help us evaluate on an existing website and give us some feedbacks. Encourage Comments ● Before we move on, I would like to stress that we are not testing you or your ability to complete the things that we ask you to do. ● We’re looking for your honest feedbacks. ● We would like to see how people use it. If you do not know what to do or the website does not work as you expected, that is the best kind of feedback you can give. Brief Background ● I am going to give you some task to complete with the website that I am going to show you. ● I would like you to “think out loud” as you try things. Say what you are looking at, what you are trying to do or what you are thinking. ● For example, if something doesn’t work the way you expected you might say “Hmm that’s not what I expected; I thought it was going to do this...” Warm up questions ● Your name is _______, right? ● What is your major? ● What do you think of the weather in Indianapolis?

24! !


!

Now we would like to talk about a online booking website. Have you ever use any online booking website?

2. Scenario 1 <Open the website A> Now, let’s say that you know you are going to have a holiday in March. You would like book a trip to New York. But before we start I would like you take a minute or two to look at the website and get familiar with it. Please tell me when you are ready. <Allow 1-2 minutes for them to look at the website> As you are completing this please try to “think out loud” as much as possible. Please start. Task 1.1: You want to book a trip to New York from 5th March 2013 to 8th March 2013. You need both air ticket (for both departure and return) and a hotel. You will be leaving from Indianapolis International Airport (IND) and be going to Kennedy International Airport, New York City (JFK). Find the cheapest direct air ticket and find the cheapest hotel with a User’s rating of at least 4 points. There are 2 different ratings in the website, Star Rating and User Rating. Star Rating is used to classify different classes of hotels, for example, 4-star is for First Class hotels, 5-star is for Luxurious hotels. However, User rating is the points that user gives to the hotel rating their experience, such as 1 points if they are not satisfied or 5 points if they are satisfied. <If they start booking using flight and hotel separately redirect them to vacation. > You have done great so far. Now I would like to redirect your attention to this page. <Redirect them to the Vacation Booking> This provides an option for people to book air ticket and hotel together. Please continue to book the same trip using this function. <Post-Task Questionnaire> Task 1.2: Now that you have selected all desired air ticket and hotel, find out how you will continue to pay for the trip. <Post-Task Questionnaire> Task 1.3: As you are about to pay for your trip, you remember that you do not have to leave on the 5th March but you can also leave on the 6th and the 7th March. Compare the price of the current trip, the trip from 6th and return on the 9th March and the trip from the 7th and return on the 10th March

25! !


!

<Post-Task Questionnaire> Now you can take a small break, please tell me when you are ready to move on :D

3. Scenario 2 So this time, let’s say you are booking for your parent. They will go to Atlanta for 3 days. Since they are older there have certain preference for the flight and the hotel. Book an air ticket and a hotel with the following requirements. Task 2.1 Find any flight (round-trip) accordingly in these criterions. ● Date: 5th - 8th March ● Depart from Indianapolis International Airport (IND) to Atlanta Hartfield-Jackson International Airport (ATL) ● For both departure and return flight, the plane should leave between 10am - 6pm so that they do not have to wake up early or arrive too late at night <Post-Task Questionnaire> Task 2.2 Find any hotel according to these criterions. ● 4-star hotels ● Hotel should locate in downtown Atlanta or the closest to downtown. <Post-Task Questionnaire>

4. SUS Questionnaire 5. Interview Now you can take a small break, please tell me when you are ready to move on :D

6. Repeat Step 2 – 5 for Website B <Open the website B> Great. Now I would like you to do this scenario for this website. Like the last website, I would like you take a minute or two to look at the website and get familiar with it. Please tell me when you are ready. <Allow 1-2 minutes for them to look at the website>

7. Final Interview

26! !


!

Pre-Test Questionnaire Website:___________________)))))))))))))))))))))))Participant)Number:)______________________________) !

Pre-Test Questionnaire Name: _______________________________ Sex: Age:

□ Male □ Female □18-24 □25-30 □30-41 □Over 41

Occupation: (if a student, please provide your field of study) __________________________________

1.

How long do you usually use Internet, for both web browsing and email in a week?

□ Less than 1 hour 2.

□0 4.

□0 5.

□ 4 – 5 hours

□ 6 hours or more

How would you rate your expertise using Online Travel Reservation website?

□ No Experience 3.

□ 1 – 3 hours □ Beginner

□ Competent

□ Expert

In the past six months, how many times have you booked a flight or a hotel?

□1

□2

□3

□4

□ 5 or more

In the past six months, how many times have you used Online Travel Reservation website?

□1

□2

□3

□4

□ 5 or more

What are the Online Travel Reservation websites that you have used before ? (Select all that apply)

□ OneTravel.com □ Expedia.com □ Kayak.com □ Hipmunk.com □ Other (Please specify) __________________________________________________________

Page!1!

27! !


!

Task Sheet

28! !


!

Scenario)2) So this time, let’s say you are booking for your parent. They will go to Atlanta for 3 days. Since they are older there have certain preference for the flight and the hotel. Book an air ticket and a hotel with the following requirements.

Task% No.% 221%

Task%Description% Find a flight according to these criterions. • •

222%

5th – 8th March Depart from Indianapolis International Airport (IND) to Atlanta Hartfield-Jackson International Airport (ATL) For both departure and return flight, the plane should leave between 10am - 4pm so that they do not have to wake up early or arrive too late at night

Find a hotel according to these criterions. • •

4-star hotels Hotel should located in downtown Atlanta

)

Post-Task Questionnaire

Website:___________________)))))))))))))))))))))))Participant)Number:)______________________________) ! Post-Task Questionnaire Task Number: _______________________________ 1. How was the overall task? Very Easy

Neutral

Very Difficult

2. I was satisfied with the ease of completing this task? Strongly Disagree

Neutral

Strongly Agree

3. I was satisfied with the amount of time it took to complete the task? Strongly Disagree

Neutral

29! !

Strongly Agree


!

System Usability Score (SUS) Questionnaire

Website:___________________)))))))))))))))))))))))Participant)Number:)______________________________) ! SUS Questionnaire 1. I think that I would like to use this website frequently. Strongly Disagree

2.

Neutral

Strongly Agree

I found this website unnecessarily complex. Strongly Disagree

Neutral

Strongly Agree

Neutral

Strongly Agree

3. I thought the system was easy to use. Strongly Disagree

4. I think that I would need the support of a technical person to be able to use this system. Strongly Disagree

Neutral

Strongly Agree

5. I found the various functions in this website were well integrated. Strongly Disagree

Neutral

Strongly Agree

6. I thought there was too much inconsistency in this website. Strongly Disagree

Neutral

Strongly Agree

7. I would imagine that most people would learn to use this website very quickly. Strongly Disagree

Neutral

Strongly Agree

8. I found this website very complicated to use. Strongly Disagree

Neutral

Strongly Agree

Neutral

Strongly Agree

9. I felt very confident using this website. Strongly Disagree

10. I needed to learn a lot of things before I could get going with this website. Strongly Disagree

Neutral

Page!5!

30! !

Strongly Agree


!

6.2. Quantitative Data Success Rate Data One Travel Task1

Task2

Task3

Task4

Task5

Find Flight and Hotel

Find the how to pay

Find and Compare Flight

Find Flight w/ specific time

Find Hotel w/ specific location

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 Success

0 0 0 0 0 1 1 1 1 1 5

1 1 1 1 1 1 1 1 1 1 10

1 0 0 1 1 1 1 1 1 0 7

0 1 0 1 1 1 1 1 1 1 8

0 1 0 1 1 1 1 1 1 1 8

Success rate (%)

50

100

70

80

80

Participant

Hipmunk Task1

Task2

Task3

Task4

Task5

Find Flight and Hotel

Find the how to pay

Find and Compare Flight

Find Flight w/ specific time

Find Hotel w/ specific location

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 Success

1 1 1 1 1 1 1 1 1 1 10

1 1 1 0 1 1 1 1 1 1 9

0 1 1 1 1 1 1 1 1 1 9

1 1 1 1 1 1 1 1 1 1 10

1 1 1 1 1 1 1 1 1 1 10

Success rate (%)

100

90

90

100

100

Participant

31! !


!

Time-on-Task Data Time-on-Task in One Travel (min) Participant

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 Min Max Average Std

Task1

Task2

Task3

Task4

Task5

Find Flight and Hotel

Find the how to pay

Find and Compare Flight

Find Flight w/ specific time

Find Hotel w/ specific location

8.7 2.5 5.3 6.4 8.4 17.4 5.1 12.6 6.7 11.8 2.5 17.4 8.5 4.4

1.5 0.9 0.8 1.2 0.1 1.0 0.1 0.3 1.3 1.9 0.1 1.9 0.9 0.6

2.0 3.5 2.9 8.0 3.0 9.5 6.2 11.6 5.3 7.9 2.0 11.6 6.0 3.2

7.9 5.4 4.6 1.6 0.9 10.2 11.1 5.5 3.1 6.7 0.9 11.1 5.7 3.4

7.5 1.6 3.3 0.0 1.3 6.1 9.0 5.4 4.0 10.3 0.0 10.3 4.8 3.4

Hipmunk Time-on-Task in Hipmunk (min) Participant

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 Min Max Average Std

Task1

Task2

Task3

Task4

Task5

Find Flight and Hotel

Find the how to pay

Find and Compare Flight

Find Flight w/ specific time

Find Hotel w/ specific location

3.8 3.7 5.0 5.9 4.0 13.5 3.0 9.8 4.9 7.0 3.0 13.5 6.1 3.3

2.2 3.2 2.2 0.5 1.3 1.0 0.2 0.3 0.5 3.3 0.2 3.3 1.5 1.2

3.5 2.3 4.3 5.6 3.0 10.2 5.0 12.0 6.1 4.7 2.3 12.0 5.6 3.1

2.7 2.7 1.2 1.6 1.3 10.1 3.0 2.6 4.6 1.4 1.2 10.1 3.1 2.7

1.3 0.6 2.1 1.9 2.1 6.6 5.0 4.0 3.1 2.5 0.6 6.6 2.9 1.8

32! !


!

Post-Task Questionnaire Data Q1

How was the overall task? (1 very easy - 7 Very Difficult)

Q2

I was satisfied with the ease of completing this task. (1 Strongly Disagree - 7 Strongly Agree)

Q3

I was satisfied with the amount of time it took to complete the task (1 Strongly Disagree - 7 Strongly Agree)

33! !


!

One Travel Post-Task Questionnaire Participant

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 AVG MEDIAN MODE

Task 1

Task 2

Find Flight and Hotel

Find the how to pay

Q1 6 6 5 3 5 6 6 2 4 4 4.7 5 6

Q1 4 1 1 2 1 1 1 1 4 4 2 1 1

Q2 3 2 5 1 3 2 1 5 4 5 3.1 3 5

Q3 3 3 7 1 2 1 1 5 4 5 3.2 3 1

Q2 5 5 7 5 7 7 7 7 7 3 6 7 7

Q3 5 5 7 4 7 7 7 7 7 3 5.9 7 7

Task 3 Find and Compare Flight Q1 Q2 Q3 6 6 6 4 5 4 6 2 4 4 5 4 2 5 2 5 2 5 3 3 2 5 4 5 1 7 7 3 5 6 3.9 4.4 4.5 4 5 4.5 6 5 4

34! !

Task 4 Find Flight w/ specific time Q1 Q2 Q3 7 1 1 7 1 1 3 2 7 3 5 5 1 7 7 1 5 1 2 6 6 3 5 5 4 4 4 1 7 7 3.2 4.3 4.4 3 5 5 3 5 1

Task 5 Find Hotel w/ specific location Q1 Q2 Q3 6 2 1 2 4 4 7 6 4 1 7 7 6 3 2 2 6 4 2 5 6 1 6 6 7 1 1 1 7 7 3.5 4.7 4.2 2 5.5 4 2 6 4


!

Hipmunk Post-Task Questionnaire Participant

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 AVG MEDIAN MODE

Task 1

Task 2

Task 3

Task 4

Task 5

Find Flight and Hotel

Find the how to pay

Find and Compare Flight

Find Flight w/ specific time

Find Hotel w/ specific location

Q1 2 5 7 4 5 4 4 5 7 3 4.6 4.5 5

Q1 4 7 7 2 7 5 7 5 4 2 5 5 7

Q1 3 2 2 5 5 3 1 2 1 2 2.6 2 2

Q1 2 4 1 2 2 2 2 1 1 1 1.8 2 2

Q1 3 1 3 3 2 3 3 1 7 2 2.8 3 3

Q2 6 5 2 4 2 5 3 2 1 5 3.5 3.5 5

Q3 6 4 4 3 6 5 5 4 1 7 4.5 4.5 4

Q2 4 4 1 2 1 4 1 3 4 7 3.1 3.5 4

Q3 3 4 3 2 1 3 1 2 4 7 3 3 3

Q2 5 6 6 4 4 5 3 6 1 7 4.7 5 6

Q3 3 7 6 3 3 7 4 7 1 7 4.8 5 7

35! !

Q2 6 3 7 6 5 7 3 7 1 7 5.2 6 7

Q3 5 3 7 6 5 3 3 7 1 7 4.7 5 3

Q2 5 6 6 6 5 5 1 7 7 6 5.4 6 6

Q3 6 7 6 6 5 6 2 7 7 6 5.8 6 6

Onetravel final report onetravel  

A comprehensive report presenting all the findings of usability testing of www.onetravel.com

Onetravel final report onetravel  

A comprehensive report presenting all the findings of usability testing of www.onetravel.com

Advertisement