Issuu on Google+

x

City University London

Evaluating Interactive Systems Coursework Human Centred Systems Yen-Hsiang Wang word count: 4111 2014


Evaluating Interactive Systems Coursework Royal Mail website Yen-Hsiang Wang

Executive Summary The researcher conducted a usability test in London from Feb 28th to Mar 6th, 2014. The purpose of the test was to assess the usability of the web interface design, information flow, and information architecture. The goals of this research are outlined as follows: • Observe natural interaction with the services on The Royal Mail website • Understand users’ mental model of the structure of The Royal Mail website and what are important factors when searching for specific information • Implement usability metrics into the website to measure and optimise user experience • Identify any barriers to use or understanding with the new service ‘Smilers’ as well as the potential usability problem when using it. • Gather insight of user impression and expression of the website • Understand any reasons for not interacting with specific features of the online services provided by The Royal Mail website These goals will help identify quick wins that can be used to improve the user experience offered by The Royal Mail website, and longer term goals that can be pursued to better align the interaction process with users’ expectations of the service searching process. Five participants were participated in the usability test. Each individual session lasted about 30 minutes, consisting of background questions, task scenarios and follow-up interviews. Task scenarios included online service Smilers according to The Royal Mail’s request to evaluate user experience of its user journey. The rest of tasks were sending parcel overseas and booking a redelivery. The researcher had chosen the tasks as sending and receiving mails / parcels are still the main staples of The Royal Mail according to users’ feedback. Prominent user experience on those services of the website would be not only crucial but also essential to the client. On the whole, all participants thought The Royal Mail website straightforward and user-friendly. 80% thought the website overall user experience was good and all of the participants gave positive feedback on Smilers being easy to use. Four out of five participants understood the structure and services of the website. The test result identified few usability problems including: • • • • • • •

Useful links provided on the homepage didn’t attract users’ attention. Difficulty to actually measure weight of sending parcels when filling in online form. Frequent failure when entering address to book redelivery. Difficulty to locate Smilers service. Confusing button orders when uploading images for Smilers service. Lack of flexibility to edit text for Smilers. Lack of advertising of Smilers and other online services.

This document contains completion rate, time on tasks, errors, satisfaction rating and recommendations for improvements. A copy of the scenarios and scripts are included in the appendices section.


Evaluating Interactive Systems Coursework Royal Mail website Yen-Hsiang Wang

Evaluation Techniques The researcher combines both quantitative and qualitative techniques in this project. By using the quantitative technique, the researcher aims at collecting data from user tests to evaluate against different metrics. Three tasks are designed by the researcher for recruited participants to carry out. The metrics for each task including: • • • •

Completion rate Task time Errors Test level satisfaction

The reason to include ‘Completion rate’ is to evaluate whether users can locate the right service through navigation. The Royal Mail website contains huge amount of online services, it is vital for users to know where to access to the right ones. ‘Task time’ measures the whole duration time for users to complete a task. It provides efficiency of the website so that The Royal Mail could compare the productivity of the services they provide to other competitor’s ones. The metric ‘Errors’ gives the evaluator clues of possible user journey flaws for further improvement. Finally, ‘Test level satisfaction’ is supposed to obtain users’ general impression of the user experience that The Royal Mail website brings and potentially for optimising it in the future based on users’ feedback. The researcher asked each participant to rate on a 5-point Likert Scale with measures ranging from Strongly Disagree to Strongly Agree after completing all task scenarios. Silverback Guerrilla has been installed in researcher’s laptop when visiting each participant. Research purpose and each session were fully explained to participants and consent forms were signed before researcher commenced interviews and recordings. For qualitative data gathering, think-aloud technique has been adopted to capture users’ mental models when implementing representative tasks. A follow-up interview will be conducted after each participant completes the task to get a holistic overview of user experience, drawbacks of the website and potential improvement in participant’s perspective.

Participants Five participants were scheduled within a week from Feb 28th to Mar 6th, 2014. Three participants were English, the rest were Turkish and Canadian. One participants had no experience of using the website, whereas one frequently used it. Of the five participants, one was female and four were male. Participants’ age ranged between 24 and 42 years old.

Participant 1

Participant 2

Participant 3

Participant 4

Participant 5

25 International student

34 Web designer

42 Medical service industry

30 International student

24 UK student


Evaluating Interactive Systems Coursework Royal Mail website Yen-Hsiang Wang

Website Evaluation - Scenario Task A Your friend Jessica’s birthday is coming and you have prepared some presents for her. Jessica is currently studying music in Vienna, Austria. You pack up the present (£50) into a small packet (15cmx15cm, 800g) and you want Jessica to get her birthday present within a week. Please use The Royal Mail website to find out the cheapest price to send a small parcel to Austria.

Task B You were away on business trip for few days. When you come back home, you found a card in your mailbox saying a mail couldn’t be delivered and requiring you to book a redelivery. You want it to be delivered as soon as possible, please use The Royal Mail website to book a redelivery by using information below:

Date on card: Feb 15th Name on card: David Jones Address: 45 Newington Green, London Postcode: N16 9QH

Task C You heard about the customized online service Smilers and you want to use the service to make your own stamps. Try to use the image provided on the desktop and make it as your unique stamp. The style and postage are up to your preferences.

Evaluation process The moderator (researcher) will introduce himself and describe the purpose of the research, the format of the 30-minute session and set expectations about the participant’s role in this process. A consent form will be handed over to participants to read through and sign. This is done in a way that reassures the participant about how their data will be used and that they are not being tested in anyway. The participant will then be asked background questions to establish their typical browsing behaviour and criteria when using online services, which will help the moderator to frame the think aloud exercise. Tasks will be explained to participants so as to the essential material to complete those tasks (e.g. photos needed to be uploaded, information needed to be filled in the form).


Evaluating Interactive Systems Coursework Royal Mail website Yen-Hsiang Wang

Task Results -Completion rate All participants successfully completed Task A and Task B (to send a parcel to overseas, to use Smilers to personalise stamps). Four out of the Five (80%) completed Task B (to arrange a redelivery). Participant 4 abandoned Task B because the website didn’t not accept the address and then turned into a temporary unavailable error page.

-Task time All participants were content to the time spending on each task. For Task A, participant 3 expected to complete task within a minute, whereas he was confused about parcel postages which slowed him down a little. Participant 2 predicted to compete Task B within a minute but turned out spending three minutes. In the follow-up interview, he said it was the address error that took him much more time on the task. All participants completed Task C much faster than expected time. In general, completion time of Task B ranged from 214 seconds (3.5 minutes) to 441 seconds (7.35 minutes), having the biggest completion time difference among all tasks (3.78 minutes).


Evaluating Interactive Systems Coursework Royal Mail website Yen-Hsiang Wang

-Errors For each task, the researcher recorded all unintended actions, mistakes and slips while participants tried to complete the tasks. Problem severity ratings were added on each error and then categorised into a table. Jakob Nielsen’s four-step scale was implemented in this report from minor to major: 0 = I don't agree that this is a usability problem at all 1 = Cosmetic problem only: need not be fixed unless extra time is available on project 2 = Minor usability problem: fixing this should be given low priority 3 = Major usability problem: important to fix, so should be given high priority 4 = Usability catastrophe: imperative to fix this before product can be released Task A had the least errors among all tasks, most of the errors occurred by user mistakes and slightly confusion of labelling. Task C had some minor usability problems such as editing images and texts, misunderstanding of uploading orders and confusion of clicking upload button. However, Task B had the most problematic usability issues when it came to find address. All participants ignored required information as ‘Flat, property number and name’ but put full address instead, which resulted in not being able to find address to arrange a redelivery. One participant had to abandon task when the page showed ‘website temporary not available’. Other minor errors including confusion of putting space and capitals when filling in postcode, accidentally cleared address as well as lack of information on available time period to arrange a redelivery.


Evaluating Interactive Systems Coursework Royal Mail website Yen-Hsiang Wang

-Test level satisfaction Each participant answered a general satisfaction question after completing all tasks. The test level satisfaction was based on the comparison of the expected time on tasks and the actual time they spent on tasks. The researcher asked participants to rate from strongly agree (5 points) to strongly disagree (1 point) of the website in terms of ease to use. The result showed the average rating of satisfaction was 3.6 points, and the percent agree was 60%

Interview Results The follow-up interviews acquired participants’ opinions of overall user experience of The Royal Mail website. Also, the researcher asked questions based on observation of participants’ natural behaviours during the tasks. The aims of the questions were to understand potential barriers, summarize usability problems and collect users’ feedback in order to give design recommendation. - General feedback summary Good about the website: • The first impression of The Royal Mail website was professional and consistent. The background red drew users’ attention. • The structure of the website was comprehensive and understood by users. • Layout was clear, breadcrumb helped users to understand where they were. • Homepage was organized and contained most of services normally people will look for. Bad about the website: • Nothing stood out from the menu, users had to read through everything on the menu. • Useful links on the homepage were completely ignored by users, they were more used to traditional navigation menu. • Too much happening on the homepage. One user was confused whether The Royal Mail was driving away from their core business. • Users did not instantly associate many other online services provided by The Royal Mail. They had very little knowledge regarding to online services


Evaluating Interactive Systems Coursework Royal Mail website Yen-Hsiang Wang

- Task scenario feedback summary Task A (Sending a parcel overseas): • ‘Sending mail’ label was confusing; one user mentioned that he couldn’t relate a parcel to that category. He suggested get rid of ‘mail’, kept ‘Sending’ and ‘Receiving’ only. • The point of splitting up ‘UK’ and ‘International’ delivery. One user stated that he thought they could be integrated into the same interface because users got to choose ‘Country’ as the first step anyways. • Might be easier to ask users a quick question such as ‘Are you sending a mail or a parcel?’ One user questioned that why bothered to ask people to choose First class or Second class? Why do users need to have this knowledge? Why couldn’t the website just offer the best deal for users? • Users wouldn’t know how to measure weights if it wasn’t given in the task. One user suggested having some instructions showing for analogy, for example the parcel weighs two bottles of water, etc. Task B (Arranging a redelivery): • Was good to show the redelivery card on the page. • All users complained about the error occurred when filling in address. Only one user realised he had to put down flat number instead of the whole address. Other users complained it’s not normal for them to put flat number or house name. They were quite surprised that The Royal Mail didn’t make find address easier. • Company name was mentioned by one user to be ideally put on the first page when filling in address instead of on the ‘Enter manually’ page. • ‘Clear address’ was recognised as ‘Next’ by one user. He also complained right clicked ‘Undo’ option didn’t do anything helpful. • When users tried to go back to previous step, one user clicked on breadcrumb and hoped it could take him back to where he was. He believed his user experience would have been enhanced if breadcrumb were clickable. • One user got confused about the text in box to fill in address. He tried to block out the text in box but he couldn’t. • Users wished to have some instruction to fill in address and postcode correctly. • Two ‘Redelivery’ tabs appeared on the same page, causing confusion. • One user complained that even if he put the flat name and postcode as requested, the address found and shown was not correct. He still had to put house number manually. Task C (Smilers): • All users had positive opinion of Smilers service. They imagine using this service to make personalised stamps for their kids and friends. One user particularly liked to use it for business purposes because it would make her company more professional and unique. • All users expected a longer time to accomplish this task. It turned out the service was much easier than they thought. • All users couldn’t find the exact location of Smilers in the first place. Two of them used search bar to find it. All users agreed that Smilers should be advertised more on the homepage. • Two users tried to edit images and texts from preview. They all expected Smilers to provide more freedom for users to edit them. • None of the users noticed there were more than one page of templates to choose. One user was trying to make personalised stamp with ‘Hello’ text like sample stamps showing on Smilers page, but he couldn’t find it and got frustrated. • One user noticed when entered Smilers service, the menu changed into ‘Business service’. He got confused because the menu was different from the original one.


Evaluating Interactive Systems Coursework Royal Mail website Yen-Hsiang Wang

• Two users clicked ‘Please select image’ area instead of ‘Upload image’ icon. One user clicked on ‘Select’ before ‘Upload image’ because he thought it made more sense if he selected images first. • Buttons’ order in the process was recommended to be re-arranged. • One user suggested the ‘Proof’ disclaimer being more prominent; he didn’t tick the box because he didn’t see it. • For first time user, he didn’t have the knowledge of the difference between First / Second class. He wished the website could explain it because not everyone knew it.

Recommendations Based on observation and interview data, the researcher provided recommended changes and justifications for homepage and content page for each task. Frequency, impact, persistence, criticality and market impact were considered when ranking severity.


Homepage 1 2

1 3

2

Observation 1

2

Red stands out and draws users’ attention, which matches The Royal Mail logo colour. Organized layout, which contains most of services that people will need.

1

2

3

Nothing stands out from the menu, need to read through everything. No participant used ‘Useful Links’ to undertake tasks. Too much happening on this page, users got distracted easily.

Change

Justification

- Move ‘Useful links’ to a simpler background. Re-design ‘Useful links’ menu to make it look more clickable.

Users ignored ‘Userful links’ because the background was too busy and distractive. Moreover, users couldn’t associate links as clickable so they went straight to navigation bar.

- Reduce background image size, show more carousel images.

Reduce background photo size can spare more room for other links and also make them more visually stand out. Current layout images are blocked by ‘Userful links’, big arrows and jigsaw boxes. Show more images to promote more services.

- Remove jigsaw text boxes.

Jigsaw shape makes everything clutter, looks like there is too much happening, too much information.

Severity Moderate

High

Moderate


Content Page - Sending mail

1

2

1

3

2

Observation 1

2

3

Breif explaination of the service before users click in.

1

Feedback tag is clearly seen on every page.

2

Some users can’t associate ‘Sendin mail’ to ‘Sending parcel’. No indication of parcel sizes and weights, hard for users to make decision on what to choose for.

The website provide different accesses for users to obtain the same information (Compare our services.)

Change

Justification

Severity

- Label as ‘Sending’ instead of ‘Sending mail’, which includes a larger aspect of sending services.

Current ‘Sending mail’ service includes mail, parcel, online shop / postage and other special services. It would be clear to put ‘Sending’ as the main category so users will not get confused.

Moderate

It is easier for users to match what the choices are and compare to the items they are sending. It is hard to ask users to measure the weights of their items. One solution is to provide the price difference so uers will have the idea of how they will be charged.

High

- Offer shape sizes of parcel for users to select. Also, give instruction of prices on different weights.

W

H L Post card Maximum H: 10.8cm L: 15cm T:0.04cm £0.1 per gram

Small packet Maximum W: 30cm L: 15cm H:25cm £0.3 per gram


Content Page - Redelivery

1

2

2

1 3

Observation 1

2

3

Instruction to explain every step for first time users. Breadcrumb gives user a clear idea where the progress is.

1

2

Same button appears twice on the same page causes confusion. Cannot use breadcrumb to go back to previous steps.

Redelivery card shows on the page, which lets users to recognize the form that they need to book a redelivery.

Change

Justification

Severity

- Remove one ‘Book a Redelivery’ button.

Two same buttons are just confusing. Users only need one redelivery button to undertake the service.

Moderate

- Make breadcrumb clickable to go back where users were.

Users expect breadcrumb to take them back to the previous step when errors happen. There should be more than one excess for users to recover from errors. Especially when current breadcrumb looks like a clickable button.

High


Content Page - Redelivery

1

2

1

3

Observation 1

Good for users to choose various way to deliver.

1

No option for users to put down address.

2

No example of correct postcode style.

3

Clear address is right under address boxes, users clear address accidently.

Change

Justification

- Add address box or highlight ‘Flat, property ’ instead to prevent users from putting down address.

Users tend to put address without reading the requested flat name or number. There should be an alternative for users to put down address in the first place. Otherwise it should be highlighted that it’s NOT address that users should key in.

High

Users should know the correct way to put postcode, whether it should be in capitals or space in between.

High

When users are forced to put down address manually, it is annoying enough. A double confirmation will avoid users’ frustration when click on it by accident.

High

Flat, property number/name e.g. 33, Flat 1 or Rose Cottage

- Add example in postcode box. Enter a postcode

e.g. KA27 8AA

- Add ‘Are you sure you want to clear address’ dialogue after users click on the button.

Severity


Content Page - Smilers

3

1 1

2

2

Observation 1

2

Multiple accesses for users to find Smilers.

1

Good to have instruction on how to use Smilers.

2

3

Mother’s Day or Smilers? Though advertise Smilers on the homepage, it is not obvious enough to let users notice this service. Too many repetitive images, which distract users’ attentions. Not easy for users to associate Smilers to be under ‘Shop’.

Change

Justification

- Emphasize ‘Smilers’ instead of ‘Mother’s Day’. Carousel image should be reduced and carousel speed is too slow for users to notice the service.

Users commented that it was hard to find Smilers, one of the reasons was because it was not properly advertised. Reduce image size , fully use the speed of carousel and increase text contrast to make Smilers stand out should catch more users attentions.

- Take down repetitive images.

It is uncalled for to put up so many repetitive images. During the tests, some users got confused and didn’t know where to start because of all the distractions.

- Put ‘Stamps’ , ‘Gifts’ and ‘Others’ under ‘Shop’.

Three users checked ‘Shop’ menu but didn’t find Smilers. There are more than 3 services under ‘Shop’. and most of them are stamp businesses. It would be more reasonable to sort them into different categories.

Shop Stamps

Gifts

Others

Current stamp range Smilers

Fashion Frame

Postage Supplies Stationery

Doctor Who

Homeware

Order Catalogue

Severity High

Moderate

High


Content Page - Smilers 1

1 3

2

Observation 1

Clearly explains Smilers service to users.

1

2

3

Jumps into ‘Business Services’ and navigation menu is completely different as well. Users didn’t notice there were more options for stamp templates in the tests. Some users didn’t understand the differences between ‘First Class’ and ‘Second Class’.

Change

Justification

Severity

- Make navigation menu consistent.

Unless there is a good reason to put Smilers under ‘Business Services’, otherwise it confuses users when they cannot find a consistent navigation menu as previous pages.

Moderate

- Make arrows visually bigger or animation if needed.

None of the users noticed there were more templates to choose. It’s vital to hint users that there are more choices so users would have more interests on the service.

High

- Add explaination tag ‘?’ aside ‘First Class’ and ‘Second Class’.

For foreign users they might not have knowledge of the differences of classes. It’s important to explain those to them.

High

First Class

Second Class

Worldwide


Content Page - Smilers 2 1

1

3

Observation 1

1

Instruction for users to user the service.

2

3

1

Users clicked on ‘Please Select Image’ instead of the icon to upload image. No explaination indicating it is a preview. The prvious step asks users to ‘Select’ an image, but actually the right sequence is to ‘Upload’ first.

Change

Justification

- A linear pattern should be implemented on this page. Users only have to follow and system takes users to the next step.

Users need to follow one step after another in a straight line; a linear pattern prevents users from getting away of the right track. Also, Integrate each step with the linear so users don’t get confuesed.

High

Users were surprised when the preview started loading, they didn’t expect to see that as a preview

Moderate

Choose images to upload

2

Edit images

Image 1

3

Severity

Preview images

Image 2

Click here to upload

* You can upload more than one image

Cancel

- Preview tag should be put aside the image.

Done


Content Page - Smilers 2 1

1

3

4

Observation 1

Good to let user edit images.

1

Didn’t tell users the differences of options.

2

Users tried to edit text from preview.

3

‘Compare’ doesn’t work properly.

4

Users overlooked ‘proof’.

Change

Justification

Severity

- Should add captions of effects in front of every option.

Users would want to use ‘Edit’ more if they knew what effects they could get.

Moderate

- If possible, let users have the freedom to edit text. Otherwise text alternative should be offered on the preview.

One user tried to edit text during the test, and commented she didn’t like it. When users try to choose different text they will have to start all over again, which is not very user-friendly.

High

Change text

- ‘Compare’ function should be more efficient. - Disable ‘Add to basket’ until users tick ‘Proof’ box.

The function didn’t work very well and lag caused when loading images.

Moderate

Give users limits reduces error rates.

Moderate


Appendices A Interview Script Introduction Thank you for taking part in this study. My name is Ian and I am a postgraduate student in Human-Centred System at City University London. The aim of the research today is to help our client understand how people go about using online services provided by The Royal Mail. The session today will last about 30 minutes. After this introduction I’ll start with some questions about yourself so we can get to know you and how you go about using online services on The Royal Mail website. The majority of the session will be taken up by searching three representative online services and observing how you carry out them. The test will be recorded both in video and audio format during the process in order to analyze user data . Following that I’ll ask some questions to get your opinion and thoughts on the website. The most important thing to remember today is that I am not testing you in anyway. You’re helping me to test the website and learn how it can be improved for you. There are no right or wrong answers to any questions I ask you today and there are no trick questions. I am interested in you and what you currently do so please try and use the website as you would at home. Also, I’m not involved with the design of anything you’re going to look at today, so please feel free to be open and honest in your opinions. Background questions


Appendices A Interview Script Think aloud observation


Appendices A Interview Script Follow-up interview questions


Appendices B Consent Forms Participant 1


Appendices B Consent Forms Participant 2


Appendices B Consent Forms Participant 3


Appendices B Consent Forms Participant 4


Appendices B Consent Forms Participant 5


Appendices C Interview transcript

Participant 1


Appendices C Interview transcript

Participant 2


Appendices C Interview transcript

Participant 3


Appendices C Interview transcript

Participant 4


Appendices C Interview transcript

Participant 5


Evaluation of The Royal Mail website