USABILITY TESTING: INSTRUCTIONAL MANUAL IN ASSEMBLING PORTABLE PAPER TOILET

USABILITY TESTING: INSTRUCTIONAL MANUAL IN ASSEMBLING PORTABLE PAPER TOILET

Trapond Hiransalee - Interaction Design Tipsarin Chavalitmontien - Design Management Master in Design and Planning King Mongkut's University of Technology Thonburi

CONTENTS Introduction

1

Research question

1

Instruction Methods

2

Usability Plan

4

Users Profile

5

Methodology

6

Data: Timeline Guiding Sticker (Method 1)

7

Data: Timeline Exploding view Manual (Method 2)

9

Data: Timeline Step-by-step (Method 3)

11

Data: Timeline Improved Step-by-step (Method 4)

13

Data: Time Duration within each Task

15

Analysing: Time for each Methods

16

Analysing: Time within each tasks for each Methods

17

Data: Errors

18

Data: Timeline Successful Assembling

19

Data: Timeline Unsuccessful Assembling

20

Analysing: Pattern Recognition

21

Analysing: Users Profile and Errors

23

Conclusion

24

INTRODUCTION In 2011, Thailand oppressive flood caused such massive damage to the country. Local people in flooded areas suffered from this unfortunate disaster, and have been forced to abandon their homes and stayed in temporary shelters which were provided at stadiums, universities, schools and government buildings. However, for people who were unable to leave their houses or evacuate from flooded areas, had to inhabited in a poor conditioned environment; no electricity, no clean water or any proper resource. Defecation were one of the main serious problem in this situation.

As a case study, King Mongkut's University of Technology Thonburi has developed a low cost self-assembled portable toilet model in order to help people in this similar situation. This model of the toilet have been designed to have three pieces of corrugation paper, due to the production and transportation. To occupy this toilet, users must first assembly the paper toilet.

As a result, we would also need to think of a way to communicate with the users on how to assembly the toilet. Therefore, developing the most practical and easy to understand manual becomes one of the key factors to make this portable toilet project succeed.

Since this model of portable toilet will be given to users in three separate pieces, and users must assembly the structure by themselves, researchers would need to develop the most user friendly and easy to understand manual which can be used effectively in various groups of Thai people.

In this experiment, we have observed on four different kinds of instruction methods. However, all of the four methods have the same quality which is have no written description, only numbers and diagrams.

RESEARCH QUESTION Researchers have come up with four different designs which in this research our goal is to find out which type of instructional method will be the most effective type of manual. Additional explorations are finding the attributes to make a instruction manual fails and succeed.

From this research we could develope the research furthermore on finding out which method suits for which type of people, such as some methods might suit for engineers or mechanics, or maybe suit for assembling different types of products. Results from research could help us improve the instruction manuals for any types of products that need assembling and communicate with users efficiently.

1

INSTRUCTION METHODS METHOD 1: Guiding Sticker This approach is executed by indicating the connection between the joints by matching the same number together. This method also indicates the sequences by numbers.

METHOD 2: Exploding View Manual This approach shows action diagram and structure diagram. The action diagram is similiar to exploded view drawing which shows the relationship or order of assembly of various parts.

2

METHOD 3: Step by Step Manual This approach is executed by indicating the connection between the joints, and also indicate the sequence order by the order of numbers. This type of manual can be seen commonly.

METHOD 4: Step by Step Manual (improved version of Manual 3) This approach is the improved version of the previous manual by indicating the piece within the manual.

3

USABILITY PLAN FOR TESTING INTRUCTIONAL METHODS NEED ANALYSIS 1. Model set up test (study, and test the final model of portable toilet which will be used to create manuals for this research project) 2. Manual system analysis (researching common manual system which could use effectively in this case) 3. Create manual design (create number stickers and illustration manual) 4. Pilot test (test the final version of manuals before use them with real subjects in this research project)

TASK ANALYSIS No.

Main task

Level 2

1

Reading the Instructions

Observing

Level 3

Understanding the System

2

Setting up the Pieces

Fold Piece A

Inserting the Interjoints

Fold Piece B

3

Assembling Structure

4

Placing Seating Pad

Fold Seating Pad

Place Seating Pad on top of Structure

Insert the Seating Pad’s joints to Structure

PROBLEM STATEMENT No.

Product component

Expected problem

1

Manual

User are unable to understand the manual system

User are do not follow the manual steps

2

User are unable to fold the piece completely

Pieces

User are unable to assemble structure completely

User are unable to fold and insert the seating pad correctly

4

USERS PROFILE No.

Method

Name

Gender

Age

Educational Background

Assembling Experience

Assembling Results

Notes

1

1

Akkachai

Male

20

Undergraduate

Yes

Accomplished

-

2

1

Bright

Male

16

Highschool

Yes

Accomplished

assemblied different form

3

1

Lek

Female

36

Undergraduate

Yes

Assisted

-

4

1

Ooy

Female

54

Elementary

No

Assisted

-

5

1

Teu

Male

53

Elementary

No

Assisted

-

6

1

Jeab

Female

33

Elementary

No

Assisted

-

7

2

Mark

Male

20

Undergraduate

Yes

Accomplished

-

8

2

Wassana

Female

34

Elementary

No

Failed

-

9

2

Pom

Female

16

Highschool

Yes

Failed

did not finished inserting

10

2

Ake

Male

37

Elementary

No

Failed

didn’t read manual, reformed

11

2

Rer

Female

37

Undergraduate

Yes

Assisted

-

12

2

Game

Male

25

Undergraduate

No

Accomplished

-

13

3

MuayLek

Female

43

Elementary

No

Accomplished

-

14

3

Anut

Female

46

Undergraduate

Yes

Accomplished

-

15

3

Chai

Male

48

Undergraduate

Yes

Accomplished

-

16

3

Wan

Female

53

Undergraduate

Yes

Accomplished

-

17

3

Somyot

Male

47

Elementary

Yes

Failed

didn’t read manual

18

3

Poo

Male

40

Undergraduate

No

Failed

-

19

4

Noot

Female

42

Elementary

No

Accomplished

-

20

4

Kaek

Male

40

Elementary

No

Accomplished

-

21

4

Palm

Male

17

Elementary

Yes

Accomplished

-

22

4

40

Highschool

Yes

Accomplished

-

23

4

Teung

Male

35

Highschool

Yes

Accomplished

-

24

4

NodNoi

Female

23

Undergraduate

Yes

Accomplished

-

NamPueng Female

5

METHODOLOGY: PERFORMANCE MEASUREMENT Users will be asked to read the manual and assemble the structure.

Users' actions will be video recorded. After users had assemblied the toilet they will be asked to answer questionnaire. All the information from the tests will be analysed and collected data.

Action

Modifiers

Error

Emotion

Task

Read

Peice

Too much

Uncertain

Behaviour1

Observe

Manual

Skip

Confuse

Behaviour2

Fold

Interlock Joint

Too long

Frustrate

Behaviour3

Inseart

Joint

Incomplete

Embarass

Behaviour4

Unfold

Number

Wrong direction

Discourage

Arrange

Grab

Rotate

Romove

Push

Other

6

DATA: TIMELINE

GUIDING STICKER (METHOD 1)

7

DATA: TIMELINE

GUIDING STICKER (METHOD 1)

8

DATA: TIMELINE

EXPLODING VIEW MANUAL (METHOD 2)

9

DATA: TIMELINE

EXPLODING VIEW MANUAL (METHOD 2)

10

DATA: TIMELINE

STEP-BY-STEP (METHOD 3)

11

DATA: TIMELINE

STEP-BY-STEP (METHOD 3)

12

DATA: TIMELINE

IMPROVED STEP-BY-STEP (METHOD 4)

13

DATA: TIMELINE

IMPROVED STEP-BY-STEP (METHOD 4)

14

DATA: TIME DURATION WITHIN EACH TASK TOTAL TIME DURATION FOR TASKS (SECS) UNDERSTANDNG THE MANUAL

TASKS

1.1

FOLDING PAPER

ASSEMBLING STRUCTURE

PLACING SEATING PAD

GUIDING STICKER

93.5

1163.7

836.1

527.1

EXPLODING VIEW MANUAL

89.4

288.6

715.3

506.2

STEP-BY-STEP INSTRUCTION

51.7

482.1

618.3

475.7

STEP-BY-STEP IMPROVED

38.4

563.8

163.9

559.4

1500.0 1163.7

Time (sec)

1125.0 836.1 715.3

750.0 527.1

482.1

563.8

475.7

559.4

288.6

375.0 93.5 0

618.3 506.2

163.9

89.4

Guiding Sticker

51.7

Exploding View Manual

38.4

Step-by-step instruction

Step-by-step (improved)

Types of Instruction UNDERSTANDING THE MANUAL

FOLDING PAPER

ASSEMBLING STRUCTURE

PLACING SEATING PAD

AMOUNT OF MEAN TIME FOR TASKS (SECS) UNDERSTANDNG THE MANUAL

TASKS

1.2

ASSEMBLING STRUCTURE

PLACING SEATING PAD

GUIDING STICKER

15.6

194.0

139.3

87.9

EXPLODING VIEW MANUAL

14.9

48.1

119.2

84.4

STEP-BY-STEP INSTRUCTION

8.6

80.4

103.1

79.3

6.4

94.0

27.3

93.2

STEP-BY-STEP IMPROVED

194.0

200.0

150.0

Time (sec)

FOLDING PAPER

139.3 119.2

100.0

87.9

80.4

94.0

93.2

79.3

48.1

50.0 15.6 0

103.1 84.4

Guiding Sticker

27.3 14.9

8.6

Exploding View Manual

Step-by-step instruction

6.4

Step-by-step (improved)

Types of Instruction UNDERSTANDING THE MANUAL

FOLDING PAPER

ASSEMBLING STRUCTURE

15

PLACING SEATING PAD

ANALYSING: TIME FOR EACH METHODS Diagram 1.1 and 1.2 shows that users spend time most on the guiding sticker (Method 1), second in order is the Exploding view manual (Method 2), third is the step-by-step manual (Method 3), and least on the improved step-by-step manual (Method 4). From this solution, it could designate that the most effective method would be the improved step-by-step manual (Method 4), for users use least time on assembling the portable toilet.

ANALYSING: TIME WITHIN EACH TASKS FOR EACH METHODS In analysising the data we have seperated the tasks in to four main ones; understanding the manual, folding paper, assembling structure, placing seating pad. Diagram 1.1 and 1.2 shows that different methods results differently for time duration for each tasks.

UNDERSTANDING THE MANUAL From the results, users take most time on understanding the manual on the Guiding sticker (Method 1), and secondly on the Exploding view manual (Method 2). The fact that all four instructions have no written description, user must take time to understand the method. In all the four methods, the Guiding sticker (Method 1) is the only method that does not have a mental model, giving difficulties for user to complete the assembly. Moreover, on the Exploding view manual (Method 2), some users did not realize there was a complete model placed in the corner of the manual, understanding that it was a logo.

The Guiding sticker (Method 1), users must realize that the parts could be assemblied by matching the joints together with the same number, also vaguely they must start assembling number 1 then the rest of the numbers in order. Same for the Exploding view manual (Method 2), users must comprehend how the diagram conceal the parts. It could be assumed that both of the Step-by-step manuals (Method 3, 4) are the most familiar to users, resulting least time consuming on understanding the manual.

FOLDING PAPER Results shown that folding on the Guiding sticker (Method 1), users spend most amount of time on, also the most in all the tasks. For the rest of methods there are no significant difference between the time duration. The results are not so much surpising owing to the fact that Guiding stickers indicates matching joints and sequences in assembling except for how to fold. From this fault, there are great chances that users would skip this task.

ASSEMBLING THE STRUCTURE From diagram 1.1 and 1.2 shows that the least effective method is the Guiding sticker (Method 1), and the most effective method would be the improved step-by-step manual (Method 4).

Assumed that there is a chain reaction from the previous task, folding the parts, when users cannot complete the folding task this will affect the assembling the structure. Users tend to go back and fourth between the two tasks. Other explanation for the failure in Method 1 is that Guiding stickers does not express parallel actions, for example inserting the two pieces together; users must insert number 3, 4, 5, 6 together the same time. Observation also shows that in Guiding stickers (Method 1) and Exploding view manual (Method 2) give complication on users not inserting the two parts all the way through. Due to both of the methods does not give a clear perception of the finished toilet.

16

COMPLICATIONS ON ASSEMBLING THE STRUCTURE: METHOD 1 All of the users below have the same problem on inserting the two parts together. Users does not know that sticker number 3, 4, 5, 6 must be inserted at the same time,for there are no clear visual image of the finished toilet (Mental Model).

COMPLICATIONS ON ASSEMBLING THE STRUCTURE: METHOD 2 Also there are no clear visual image of the finished toilet (Mental Model).

For the Step-by-step manuals (Method 3, 4), the most effective methods, users used least time.

During testing Method 3, the step-by-step manual, we have found out that users had difficulties in indicating which pieces was which. So we made some improvements on the step-by-step manual by labelling the pieces and the manual. Significant improvements for assembling the structure can be seen in diagram 1.3.

750.0

Time (sec)

1.3

618.3

375.0 163.9 0

Step-by-step instruction

Step-by-step (improved)

PLACING THE SEATING PAD In this task there are no significant differences shown between the four methods, since all the methods haven’t focused enough on this task. Due to the fact that this task, still takes quite amount of time and have many errors before achieving this task, such as many users cannot tell which way to fold and they ended up folding inside out. Observers suggest that improvement should be made for this task.

17

DATA: ERRORS NUMBER OF ERROR WRONG DIRECTION

ERRORS

2.1

TOO LONG

TOO MUCH

SKIP

INCOMPLETE

TOTAL

GUIDING STICKER

15.0

15.0

11.0

13.0

0.0

41.0

EXPLODING VIEW MANUAL

11.0

3.0

0.0

8.0

5.0

14.0

STEP-BY-STEP INSTRUCTION

28.0

2.0

0.0

3.0

4.0

30.0

STEP-BY-STEP IMPROVED

5.0

0.0

3.0

1.0

7.0

8.0

50.0

Number of Error

41.0 37.5 30.0

28.0 25.0 15.0 15.0 12.5

0

11.0

13.0

14.0

11.0

0 Guiding sticker

8.0 3.0

5.0

2.0

0

Exploding view manual

0

3.0 4.0

Step-by-step instruction

7.0 8.0

5.0 0

3.0

1.0

Step-by-step (Improved)

Types of Instruction

WRONG DIRECTION

TOO LONG

TOO MUCH

SKIP

INCOMPLETE

TOTAL

ANALYSING: NUMBER OF ERRORS IN EACH METHODS Observers have categorized the errors in to six categories, below are the details: Wrong Direction parts rotate in wrong direction, folded incorrectly (inside-out), insert joint incorrectly.

Too Long takes too much time on the task Too Much do more than the objective of task; such as fold the parts where shouldn’t be folded Skip skip the task to do other task Incomplete did not complete the task; such as not inserting the joint on the seating pad From the results, users using Guiding sticker (Method 1) have most errors, least errors for the Improved step-by-step manual (Method 4). Therefore, looking only at the total number of errors we couldn not tell whether the instructions are ineffective. In some case users might gave up assembling before completing the toilet and the number of errors stopped. Considering Guiding sticker (Method 1), the number of incomplete errors are zero, shows that all users assembly the paper toilet successfully however with serveral errors.

Actually, in the real situation, the users using Guiding sticker (Method 1) was assisted when users reached the point of performing too long and skipping the task, while other methods did not assist the users. Some users did not complete and stopped before accomplishing in the other three methods, given the result of less total errors. Using the “skip” error would be best in pointing out the weakest method, due to when users skipped the task, demonstrates failed communication between users and manual.

18

DATA: TIMELINE

SUCCESSFUL ASSEMBLING

3.1

Guiding Sticker (Method 1) - Akkachai

3.2

Exploding View Manual (Method 2) - Mark

3.3

Improved step-by-step (Method 4) - NidNoi

19

DATA: TIMELINE

UNSUCCESSFUL ASSEMBLING

4.1

Guiding Sticker (Method 1) - Bright

4.2

Guiding Sticker (Method 1) - Lek

4.3

Explodong View Manual (Method 2) - Rer 20

ANALYSING: PATTERN RECOGNITION For users to accomoplished the assembling, user should perform the task in order; from Task 1 to 4 (Behavior 1 > Behavior 2 > Behavior 3 > Behavior 4). If users skipped the task to do another task without finishing the previous task, users will not accomplished assembling.

Examples are shown below in diagram 5.1 and 5.2.

5.1 PERFORMED TASK CORRECTLY Initial Task Understanding Folding Paper Assembling Structure Placing Seating Pad

Guiding Sticker (Method 1) - Akkachai [180 s.ff

5.2 PERFORMED TASK INCORRECTLY Initial Task Understanding

Understanding

Folding Paper

Folding Paper

Assembling Structure

Assembling Structure Placing Seating Pad

Guiding Sticker (Method 1) - Lek [700 s.ff]

You can see that for the users who performed correctly (diagram 5.1) the tasks were done in order from Behavior 1 to Behavoir 4. For users who did not accomplished (diagram 5.2) or took too much time on assembling will go back and forth between the tasks. This observation involve all four methods.

21

ANALYSING: PATTERN RECOGNITION From the pattern analysis, we have also found some attribute for different types of methods. In the Guiding sticker (Method 1) shows some varying from other manuals which is in the “Fold” action. You can see in the Guiding sticker (Method), action “Fold” is done in every task except for the understanding task. On the other hand ,“Fold” in all three methods (Exploding view manual, Step-by-step manual, Improved step-by-step manual) are only seen in the Folding Paper Task, meaning that users must fold the pieces first then move on to other actions in the next task. Accordingly, we have mention before that in the Guiding sticker (Method 1) there are no indicators for folding, so it correlates with the pattern analysis.

6.1 SEQUENCES OF ACTIONS DONE IN EACH TASK: GUIDING STICKER (METHOD 1) Initial Task Read

Read

Read

Fold

Read

Fold Insert

Fold

Insert

Insert

Fold

Insert

Insert

Unfold Grab

Grab

Unfold

Grab

Rotate

Rotate Remove Push

Rotate

Rotate

Rotate

Remove Push

Push

Initial Task Understanding Folding Paper Assembling Structure Placing Seating Pad

Guiding Sticker (Method 1) - Akkachai [180 s.ff]

6.2 SEQUENCES OF ACTIONS DONE IN EACH TASK: IMPROVED STEP-BY-STEP (METHOD 4) Initial Task Read

Read

Fold Insert Grab

Read

Fold Insert

Insert

Grab

Insert Grab

Rotate

Rotate

Rotate

Rotate

Push

Initial Task Understanding Folding Paper

Assembling Structure Placing Seating Pad

Improved step-by-step manual (Method 4) - NidNoi [160 s.ff]

22

ANALYSING: USERS PROFILE AND ERRORS Another observation is to find the correlation between how well user assembly and the user profile. Later when we looked on all users’ profile, we have found out there was no clear cut sign from this experiment. Eventhough we found for users’ who alway use tools normally in daily life do better in assembling, and there is also the age and education to be looked at.

We found out that low education and older users have more amount of errors than others.

Therefore there are some conflicts, for users with the elemantary education background have high age.

For now, we can conclude that low educational background give more errors in assembling,and high school students seem to have less error and less duration of the task. It may relate to the high processing of the cognitive task.

7.1 NUMBER OF ERRORS IN DIFFERENT EDUCATION BACKGROUND

AMOUNT OF TIME FOR TASKS (NUMBER)-1 SKIP

ERRORS ELEMENTARY

TOO LONG

INCOMPLETE

10.0

9.0

8.0

HIGH SCHOOL

4.0

10.0

4.0

BACHELOR DEGREE

11.0

1.0

4.0

WRONG DIRECTION 36.0

TOO MUCH

TOTAL

2.0

65.0

6.0

2.0

26.0

17.0

10.0

43.0

70.0

52.5

35.0

17.5

0

WRONG DIRECTION

Elementary

TOO LONG

High school

TOO MUCH

23

Bachelor degree

SKIP

INCOMPLETE

TOTAL

CONCLUSION From this research, we have found many attributes and remarks proving which type of method have disadvantage and advantage, which type of method is effective or not any why. The best method is the Improved step-by-step manual. Below is the list for important attributes that make instructional manual easy for users to understand.

• Users must see how the complete toilet looks like, sometimes we call it Mental Model.

When users read the manual, what they need to focus on first is the goal of assembling, therefore when they don’t what the finished toilet looks like, it becomes difficult for them to succeed.

• In assembling, it is important that users understand all the actions that must be done. For example in the Guiding sticker method (Method 1), there is a major disadvantage on not having indicators for folding. Guiding stickers (Method 1), could be effective if it requires only on type of action such as insert.

• The most ineffecient method is the Guiding sticker (Method 1), then second is the Exploding view Manual (Method 2) • Exploding view manual (Method 2) should suit best for users who have high-assembling skills, such as engineers or architects. For normal users, it would be difficult for them to understand the method.

• The step-by-step manual (Method 3, 4) show best results for assembling, for people easily understand when they are visually clear with what they have to do step-by-step. It aslo give users confidence while assembling. Therefore the step-by-step manual still need many improvement, such as indicating the pieces and should show more steps and details. Good example for this type of manual would be instructions for Origami.

• In observing, there were many complications and errors, such as have not recorded users from the start when reading the manual, having two different observers which make the task analysing not consistant, not enough testers in this usability testing, parts of toilets difficult to assembly, parts of toilet start to perish, etc.

• Skipping the correct sequence of task, could lead to unsuccessful assembling.

• In this case, instructions should be in 3D for better understanding.

24

SPECIAL THANKS Professor Sakol Teeravarunyou Anuthida Jungsura Nanthikarn Kohkiattaweechai Pongsak Kiti P. Band