Lost in Hyperspace

Utrecht University Beta Faculty Department of Information and Computing Sciences

Lost in hyperspace: lostness and users’ mental models of hypertext systems Concept final report – version 1.0 Dirk Menkveld & Ed’son de Pary {djmenkve, epary}@students.cs.uu.nl 18 June 2007

Abstract Becoming lost within a website is one of the main problems users experience while navigating on the web. Users visit websites they’re familiar with, but sometimes they want to find information in a certain domain which is sometimes unfamiliar to them. This is a report of an empirical study how users perform on a task in searching for information on a familiar and an unfamiliar domain, these domains are represented by two websites. To see the effects; the lostness and the mental models were calculated. The study showed that users with an certain knowledge domain (IT in this study) performed better on the familiar domain than on the unfamiliar domain, besides their navigational expertise.

Keywords: usability, hypertext systems, lostness, mental model, content domain expertise, familiarity

‘Lost in hyperspace’

Page 2 of 22

Index 1. Introduction ......................................................................................................................... 3 Motivation ....................................................................................................................... 3 Report remainder ............................................................................................................ 3 2. Theoretical background ...................................................................................................... 4 Mental model .................................................................................................................. 4 Unfamiliarity ................................................................................................................... 4 How to measure mental models ...................................................................................... 4 Lostness ........................................................................................................................... 4 How to measure the lostness ........................................................................................... 5 Perceived disorientation .................................................................................................. 5 Expertise ......................................................................................................................... 5 3. Hypotheses .......................................................................................................................... 7 Expectations .................................................................................................................... 7 Familiar domain (FD) ..................................................................................................... 7 Unfamiliar domain (UD)................................................................................................. 7 4. Empirical Study .................................................................................................................. 8 Participants ...................................................................................................................... 8 Design ............................................................................................................................. 8 IR-tasks ........................................................................................................................... 8 Mental model task ........................................................................................................... 9 Questionnaires ................................................................................................................. 9 5. Results ............................................................................................................................... 10 Experience of participants............................................................................................. 10 Lostness ......................................................................................................................... 10 Task Performance ......................................................................................................... 11 Mental model accuracy ................................................................................................. 12 Perceived disorientation ................................................................................................ 12 Correlations ................................................................................................................... 13 Familiar vs. unfamiliar domain ..................................................................................... 13 6. Discussion and Conclusions ............................................................................................. 15 References ............................................................................................................................. 17 Appendix A. Questionnaires ................................................................................................. 18 Prequestionnaire ............................................................................................................ 18 Postquestionnaire .......................................................................................................... 20 Appendix B. Information Retrieval Tasks ............................................................................ 21 Familiar domain ............................................................................................................ 21 Unfamiliar domain ........................................................................................................ 22

‘Lost in hyperspace’

Page 3 of 22

1. Introduction Being disoriented or lost is one of the fundamental difficulties which users experience when trying to navigate a website. If it has been designed so that it does not correspond with users’ mental models, then the users are more likely to become lost (Otter & Johnson, 2000). Consequently, evaluating and detecting users’ misunderstandings early in de design process saves a lot of problems in terms of satisfying user needs, defects, cost, etc.

A characteristic that plays an important role by developing a mental model is experience.

Experts in a knowledge domain became experts by repeating a task (Anderson, 2005). It is known that experts have an optimal mental model of a hypertext system (Otter & Johnson, 2000). But how great is it when there are navigating in an unfamiliar domain? That’s what we want to investigate. Thus, the main goal is to see whether an expert in a certain knowledge domain is more lost in an unfamiliar domain than in a familiar domain regarding its content.

Motivation A lot of research in web navigation has been done. One of the reasons we want to examine the level of lostness of the users interacting in various hyperspaces is because in our social network people always ask us, viewed as experts in content retrieval to help them find what they are looking for.

We consider ourselves as people who are more advanced in navigational tasks and content recognition, we earned our expertise in the portfolio of our study (Information Science), in fact, we are almost all the time interacting with websites whose content is mostly ICT lining, in those, we feel at ease in our knowledge domain, the main question is to know if we feel the same when the content of the website changes to an unfamiliar domain.

While reading the various papers in the references list of the assignment, we found that unfamiliarity with various hypertext systems was declared as a common cause of lostness by subjects (Otter & Johnson, 2000). That’s why we want to look at the relationship based on familiarity with the content within a hyperspace, concerning lostness and mental models.

Report remainder The remainder of this report proceeds as follows. In chapter 2 we begin with describing some theoretical background about unfamiliarity, expertise, the mental model and lostness.

Then we present our hypotheses in chapter 3 and in chapter 4 we describe our empirical study according to these hypotheses. Chapter 5 covers the results and we discuss those in chapter 6 and also conclude this report.

‘Lost in hyperspace’

Page 4 of 22

2. Theoretical background Mental model Mental models are psychological representations of real, hypothetical, or imaginary situations. Mental model research is based on the assumption that knowledge of how users represent systems and how users should represent systems will lead to a better understanding of usable systems (Ackermann & Tauber, 1990). In fact, these are conceptual and operational mental representation that the users develop while interacting with a system. It traces the path followed by the user while trying to retrieve the demanded content; mostly, the trace is not reproduced in a complete representation of the actual taken path, but it gives an idea of the way the system works. Generally, the state of the mental model of the users affects their experience within the hyperspace; they will surely be disoriented, if the mental model is poor. Merrill (1991) remarked that a complex mental model enables the learner to engage in some complex human enterprise or integrated activity. We are therefore going to evaluate and measure the deepness of the mental model of various participants, based on their ability to retrace the paths they took to retrieve content within the premises of the hyperspace.

Unfamiliarity In the introduction we said that unfamiliarity with various hyperspaces was declared as a common cause of lostness by subjects. Otter and Johnson (2000) found that in their research about ‘Lost in hyperspace: metrics and mental models’. It was the focus of the second experiment they performed to find the common causes of becoming lost or disorientated when using hypertext systems. When they specifically asked about unfamiliarity as a question, the participants considered it as an important common cause of becoming lost.

How to measure mental models The measurements of the state of mental model of users can be done with different types of methods. We decided to use the method of Otter and Johnson (2000) in which they measure the accuracy of mental models for hypertext: AMMH = ⅓ (C/AD + CCP/RD + LBE/RD) Where: AD = number of nodes actually drawn RD = number of nodes required to be drawn C = number of nodes that are correct CCP = number of nodes that are both correct and correctly placed LBE = number of different levels of the hierarchy correctly drawn before the subject made an error

Lostness As we have mentioned before being disorientated or lost is one of the fundamental difficulties which users experience when trying to navigate within hypertext systems.

Hypertext systems have a flexible structure and give the users the freedom to browse and interact with the information contained within it. Generally, the users need to know where they are in a hyperspace; they need to know where they want to be and what the best path is. So the main goal of an interface or hypertext system is to make the way easier for the

‘Lost in hyperspace’

Page 5 of 22

users through the system in other to retrieve the information he or she seeks, it’s just imaginable how frustrating it can be when one can’t just do that. What is this related to, is it the interface, the users IQ, his or her knowledge about hyperspaces or information retrieval techniques? That’s what we will try to find out. We will use two ways to measure the lostness of the users: one way is from Smith and the other way is from Ahuja and Webster.

So, we will proceed by using the defined method below to come to our conclusions.

How to measure the lostness According to Smith (1996), lostness is indicated by degradation of user’s performance, which can be observed by an increased numbers of revisited pages. She uses the following formula the measure lostness: L = √ [(N/S - 1)2 + (R/N - 1)2] Where: L = represents the lostness R = number of nodes required to finish the task successfully S = total number of nodes visited while searching N = number of different nodes visited while searching.

0 indicates no lostness at all, and a user is considered to be lost when the measure of lostness is 0.42 or higher.

Perceived disorientation Another way of measuring the lostness is directly asking the users how they feel after doing the IR-tasks. This can be done by presenting the questionnaire of Ahuja and Webster (2001). They found that their questionnaire predicted performance on web information retrieval tasks better than user actions. It’s about self-perceived disorientation. And it’s also an easier way of collecting and interpreting the perceived disorientation.

Expertise A user’s level of expertise, in a number of respects, is a critical consideration. Danielson (2002) said in his book that one has to take into consideration the fact that there are four types of expertise that are particularly important to determine who are expert is in various retrieving tasks: General web navigation expertise allows a user to make predictions based upon knowledge of how websites are typically organized and designed. For example, one might predict that by clicking on a hyperlink in a top-level navigational mechanism, the local context hyperlinks along the left column of the page will change, since many hierarchically organized sites are designed this way.

Content domain expertise allows a user to make predictions based upon knowledge of how information in the domain is structured and interrelated. For example, one might predict that clicking on a hyperlink would likely lead to information about disorientation and possibly links to information about navigation design, based on her knowledge of the subject.

Site domain expertise allows a user to make predictions based upon knowledge of how a particular class of sites are typically organized and designed. For example, one might predict that clicking on the name of an author at a bookstore site would lead to a page with a list of books for sale by that author, since many bookstore websites are designed this way.

‘Lost in hyperspace’

Page 6 of 22

Within-site expertise allows a user to make predictions based upon previous interaction with a specific Web site. For example, one might predict that within a particular site, clicking on the name of an Italian restaurant will lead to a page with the restaurant’s menu, since other similar hyperlinks within the site have done the same (that is, one notices both linguistic homogeneity).

With regards to all these various types of expertise, we will define our candidates with their ability in ‘content domain expertise’. Contain domain expertise allows a user to make predictions based upon knowledge of how information in the domain is structured and interrelated. This choice is made based on the fact that we aim to retrieve content in both hyperspaces, one being a familiar knowledge domain to participants, this means that they are familiar with the content within the website, and the other being an unfamiliar knowledge domain, meaning that, the content within the website is new to all participants.

‘Lost in hyperspace’

Page 7 of 22

3. Hypotheses We assume that the level of lostness of a user in a familiar knowledge domain is lower than the level of lostness of the same user in an unfamiliar knowledge domain; hereby the mental model will equally develop.

Knowledge domain Lostness users

Familiar Low 1)

Unfamiliar High 2)

Table 1. Level of lostness

This results in the following hypotheses: H0: The level of lostness of a user will be the high in the unfamiliar domain, and the depth level of their mental models will be the same.

H1: The level of lostness of a user will be low in the familiar domain, and the depth level of their mental model will be the same.

Expectations 1) Low: 2) High:

The level of lostness will be low, because the user is operating in his or her knowledge domain.

The level of lostness will be high, because the user has no prior knowledge about the content and is unfamiliar with the domain.

Familiar domain (FD) We can speak of a familiar domain when a user recognises content within which he or she is working in. For example we will use the website of information about vista in this research (http://www.vistainfo.nl/) for our users who are familiar with ICT ideas. We expect them to easily retrieve information being asked to them, because the test will be purely based on retrieving the ICT-related content. We also expect their mental models to be very high, because of their familiarity with web design, content in it, and structural search methods from one depth level to the other.

Unfamiliar domain (UD) The website chosen here will be totally unusual to the majority of the test persons of this research, because they will all be computer science students, who are usually familiar with topics related to ICT, they will be confronted with the a history website (http://www.geschiedenis.nl/). Being confronted with pure questions related on the content of the subject, we expect them somehow to be more lost than in the previous situation. Also their mental models are expected to be more or less low, because of the unfamiliarity to the content found on the website.

‘Lost in hyperspace’

Page 8 of 22

4. Empirical Study The main goal of this study is to see whether an expert in a certain knowledge domain is more lost in an unfamiliar domain than in a familiar domain. In our experiment participants performed several information-retrieval tasks on the websites of both the domains. We conducted the experiment in Dutch to avoid inconsistencies, because all our participants were Dutch students, and translations of some items could be a problem for some users.

Participants The number of participants in the experiment was 20. One participant was female (5%).

The average age of the participants was 23.5, ranging between 19 and 30 years. The subjects were all students of the Faculty of Information and Computing Sciences at the Utrecht University.

Design At the beginning the participants had to fill out a prequestionnaire, after that the participants performed 12 information retrieval tasks and 2 drawing tasks per website to give a representation of the mental model at the current time. In total, they had to perform 24 IRtasks and 4 drawing tasks. And finally, the participants filled out a postquestionnaire concerning perceived disorientation. The order of the websites equally changed among the participants. Some had to begin with the IR-tasks with the familiar domain and others had to begin with the unfamiliar domain. The ordering of the tasks was held constant across the participants. We selected one website containing ICT-related content, and familiar to the participants, and the other containing non ICT-related content, and unfamiliar to the participants. The first one, http://www.vistainfo.nl/, has the kind of content that is widely known and familiar to our participants. The second website, http://www.geschiedenis.nl/, is a site containing information about the history of the World and the Netherlands; it’s content is purely history related, and in most cases unfamiliar to our participants, because they are their level of expertise lies within another domain. The experiment took approximately 50 minutes. Camtasia Studio software was used to record the moves and clicks of the participants.

Website: http://www.vistainfo.nl/

Website: http://www.geschiedenis.nl/

IR-tasks There are tasks to be done in the familiar and the unfamiliar domain. To see how lost a user can be, we presented tasks with different depth levels, 4 tasks of the 1st level, this level gives the participants the ability to retrieve content while clicking just once on a link; 4

‘Lost in hyperspace’

Page 9 of 22

tasks of the 2nd level, meaning that, the participants would click twice on different hyperlinks to be able to retrieve the content they are looking for, and 4 tasks of the 3rd level, where the participants would have to click on three different hyperlinks to retrieve the asked content. The main idea of the tasks was to answer a question by searching for the information on the website. An example in the familiar domain is: What is ‘phishing’? (depth level=2). And an example in the unfamiliar domain is: Who were the opponents in the Cold War? (depth level=2). When the depth level increases, we expect the participants to become more lost. All the questions are presented in Appendix B.

Mental model task After the first set of tasks (of depth level 1) we presented a mental model task he or she had to perform. We also presented the same at the end of the experiment. A mental model task consists in answering a question from memory in form of a pathway. The pathway represents links to be following to get to the place on the website where the asked information is stored.

Questionnaires The prequestionnaire is needed for seeking general knowledge and information about the participants. The questions will be about the gender of the users, their occupations and their relation with the subject we are researching.

With the postquestionnaire we can measure the user’s perceived disorientation as proposed

by Ahuja and Webster (2001).

Some questions were open and other were posed in a 7-point Likert scale. The questionnaires are presented in Appendix A.

‘Lost in hyperspace’

Page 10 of 22

5. Results The results are partitioned in multiple sections. The experience of the participants, the lostness (L), the task performance (TP), the mental model accuracy (AMMH) and the perceived disorientation are addressed. And finally, we will look at relations between these items.

Experience of participants The results from the prequestionnaire are shown in table 2.

Scale 1 to 7 (1=never…7=always) a. If I search something, I always find it. b. I always know where I have to go within a website. c. The websites I visit are well structured. d. On the web I feel at home. e. If I found something that is close, I am satisfied. f. To find my way in a hyperspace is the same as in the real world. g. I am familiar with www.vistainfo.nl h. I am familiar with www.geschiedenis.nl

Mean 5.60 4.70 4.50 5.70 4.00 2.45 1.10 1.40

SD 1.00 1.13 1.19 1.08 1.30 1.05 0.31 0.94

Table 2. User's experience

The scores on item [(a) to (e)] show that our participants are well known users with the web; they feel at home and are experienced in searching for information. However, a reliability analysis on these items showed a low-interim reliability (Cronbach’s α=.10).

Item f shows us how they compare hyperspaces with the real world. They tend to the left and that means there is a comparison made. Item (g) and (h) were questioned to see whether the users did already had experience with one of the websites and the scores on these items were advantageous, because they underlie that we can statistically prove some relations.

Two items are not presented in table 2, because they are on a different scale. One item showed us how many hours of a day users spends on the internet (scale 1 to 3; 1= < 1 hour, 2= 1-3 hour, 3= > 3 hour), it was favourable because it increased the experience of the user (Mean=2.60; SD=0.60). And the other item showed us how many websites they visited in the amount of time just mentioned (scale 1 to 3; 1= < 5 websites, 2= 5-10 hour, 3= > 10 websites), it was favourable because it increased the experience of the user (Mean=2.40; SD=0.68).

Lostness The lostness score was calculated with Smith’s formula. The scores are presented in table 3.

Lostness per task Vista (FD) Geschiedenis (UD) Table 3. Lostness

Depth 1 Mean SD 0.61 0.20 0.64 0.13

Depth 2 Mean SD 0.32 0.14 0.44 0.16

Depth 3 Mean SD 0.11 0.10 0.31 0.16

‘Lost in hyperspace’

Page 11 of 22

Los tne s s 0.8

0.6

Score

The repeated-measures ANOVA for the familiar domain did show significant differences (F(2,38)=72.54, p<.05) and the repeated-measures ANOVA for the unfamiliar domain did also show significant differences (F(2,38)=27.21, p<.05). This indicates that the lostness fluctuated over the different levels (see graph 1).

Normally the score on Smith’s formula of lostness should increase while the tasks are getting more difficult. But in this case they did not, because the numerator and denominator heavily fluctuated in Smith’s formula.

FD

0.4

UD

0.2

0 1

2

3

Depth level

Graph 1. Lostness

Task Performance Tasks scores is defined as follow: when the answer was correct, participants scored 1 and when it was wrong, participants scored 0. The scores are presented in table 4.

Score per task Vista (FD) Geschiedenis (UD)

Depth 1 Mean SD 0.60 0.19 0.57 0.18

Depth 2 Mean SD 0.76 0.21 0.75 0.24

Depth 3 Mean SD 0.86 0.27 0.55 0.22

Table 4. Task performance

Score

The repeated-measures ANOVA for Tas k Pe rform ance the familiar domain did show significant differences (F(2,38)=6.18, 1 p<.05) and the repeated-measures ANOVA for the unfamiliar domain did 0.8 also show significant differences (F(2, 38)=6.83, p<.05). Our expectations 0.6 were as follow; the higher the depth FD levels, the poorer the performances of UD 0.4 the users will be. Graph 1 shows that the hypothesis above doesn’t match our expectations. There are a couple of 0.2 reasons to justify this effect: when we set the tasks to be retrieved from the 0 websites, we did not expect the links 1 2 3 related to the tasks to be removed from Depth level the website after a couple of days, as that was the case, this created a Graph 2. Task Performance negative effect on the results, because the participants couldn’t retrieve the information they were searching for. The second

‘Lost in hyperspace’

Page 12 of 22

reason that justifies this effect is that when it came to the familiar domain, the four tasks of depth level 3 were too close to each other. So, the participants could easily retrieve the answers. These are two major reasons why the expected effects on the results were not visible.

Mental model accuracy The mental model score was calculated with the formula developed by Otter & Johnson (2000). The scores are presented in table 5.

Score 0…1 Vista (FD) Geschiedenis (UD)

MM1 Mean 0.29 0.48

MM2 SD 0.27 0.38

Mean 0.41 0.50

SD 0.21 0.33

Table 5. Mental model accuracy

M e ntal m ode l 0.6

0.4 FD

Score

The repeated-measures ANOVA for the familiar domain didn’t show any significant differences (F(1,19)=3.34, p=.08>.05). The first AMMH was 0.29 (SD=.27) and the second AMMH was 0.41 (SD=.20). The repeated-measures ANOVA for the unfamiliar domain also didn’t reveal any significant differences (F(1,19)=0.10, p>.05). The first AMMH was 0.48 (SD=.38) and the second AMMH was 0.50 (SD=.33).

However, in both cases the AMMH scores were better the second time, so we say that the mental models evolved over time, even though it was not significant.

UD 0.2

0 1

2 Depth level

Graph 3. Mental model

Perceived disorientation The results from the questionnaires after completion of the tasks of each website are shown in table 6. The scores of the vista website is shown as FD and the scores of the geschiedenis website is shown as UD. Scale 1 to 7 (1=never…7=always) 1. I felt lost.

2. I felt like I was going around in circles 3. It was difficult to find a page that I had previously viewed.

4. Navigating between pages was a problem 5. I didn’t know how to get to my desired location.

6. I felt disoriented.

7. After browsing for a while, I had no idea where to go next.

Table 6. Perceived disorientation

FD UD Mean SD Mean SD 3.45 1.67 3.35 1.93 3.85 1.63 2.95 1.79 2.95 1.57 2.80 1.36 2.50 1.28 2.90 1.65 4.15 1.76 4.05 1.76 3.20 1.44 3.10 1.65 2.50 1.19 3.70 1.69

‘Lost in hyperspace’

Page 13 of 22

The self-perceived disorientation scores didn’t always show low scores indicating that the users were sometimes lost. There were two remarkable differences: item 2 and item 7. Item 2 showed a high difference between both domains, because the vista has a different structure than the geschiedenis website that made the users go around in circles. Item 7 indicates that the user in the unfamiliar domain had no idea where to go next, because the website is far broader than the vista website. The history website contains a lot more information. A reliability analysis on items concerning disorientation on the familiar domain showed a high-interim reliability (Cronbach’s α=.83) and also on the unfamiliar domain (Cronbach’s α=.88).

Correlations There were some significant correlations between task performance & lostness. The analysis was performed with the averages of the total performance and lostness and of the three different depth levels. Depth level 1 (r=-.57, p<.01) and 3 (r=-.48, p<.05) of the familiar domain were significant. There was no overall correlation for this domain. The unfamiliar domain also showed significant correlations. The average total score (r=.65, p<.01) and depth level 1 (r=.66, p<.01) were significant. The mental model accuracy didn’t show any significant correlations with the lostness or the task performance.

Familiar vs. unfamiliar domain The task perfomance and the calculated lostness of both domains are shown in table 7. The scores of the vista website is shown as FD and the scores of the geschiedenis website is shown as UD. Measures Overall average Lostness Average Lostness depth 1 Average Lostness depth 2 Average Lostness depth 3 Overall average Task Performance Average Task Performance depth 1 Average Task Performance depth 2 Average Task Performance depth 3

FD Mean 0.35 0.61 0.32 0.11 0.74 0.60 0.76 0.86

UD SD 0.11 0.20 0.14 0.10 0.18 0.19 0.21 0.28

Mean 0.46 0.64 0.44 0.31 0.67 0.58 0.76 0.55

SD 0.10 0.13 0.16 0.16 0.15 0.18 0.24 0.22

Table 7. Comparison of the two domains

The table above also gives us as expected the logical results, the tasks on the familiar domain were better performed than those of the unfamiliar domain, this can be justified by number of factors; the first reason we can immediately look at is the familiarity with the content of the site, we did our research by testing only student of the Faculty of Information and Computing Sciences, these students are supposed to be familiar with everything related with computers and their innovations. So as you can see on the table above, these students scored generally high on the familiar domain, when it came to the unfamiliar domain where they are less familiar with content, their performances decreases in some cases dramatically. We performed paired samples t-tests for all the 8 conditions. The overall averages of lostness showed a significant difference (t(19)=-5.26, p<.05). The averages of lostness of depth 1 didn’t show a significant difference, on the other hand depth 2 (t(19) =-

‘Lost in hyperspace’

Page 14 of 22

2.77, p<.05) and 3 (t(19) =-5.03, p<.05) did so. The task performance averages didn’t show significant differences, except for the task performance of depth 3 (t(19)=4.47, p<.05). All the results confirm our hypotheses. In other words, we can say that students with an ICT background performed better on the familiar domain than on the unfamiliar domain, besides their navigational expertise.

‘Lost in hyperspace’

Page 15 of 22

6. Discussion and Conclusions When we started this research, we had a number of assumptions which resulted in the described hypotheses. These assumptions were based on conducted research regarding navigating in hyperspaces, lostness and mental model accuracy.

The main goal of this empirical study was to see whether an expert in a certain knowledge domain is more lost in an unfamiliar domain than in a familiar domain regarding its content. Therefore we had to measure the lostness and the mental model of participants while navigating on the domains.

Our first finding concerns the lostness of our participants, generally, our hypotheses sounded: ‘The level of lostness of a user will be the high in the unfamiliar domain, and the depth level of their mental models will be the same’. The results presented in table 3 shows that we were right, in fact on all three depth levels, participants scored a lower lostness rate on the website they were supposed to be familiar with (www.vistainfo.nl), than what they did on the website they were supposed to be unfamiliar with (www.geschiedenis.nl).

Smith (1996) said in her study on lostness, that lostness is a common phenomenon on the web. She argued that lostness shouldn’t be viewed in terms of subjective feelings of users, but in terms of the degradation of performance. The perceived lostness could then have various reasons: generally, the first observation we can make is that, the lostness level of the users depended on whether they were familiar or not with the content of the website they were surfing on. The results we obtained shows different levels of lostness of users on both domains. As we said above, our participants were all students of the Faculty of Computing and Information Sciences, so they were all supposed to be familiar with a website like that of http://www.vistainfo.nl/, because of the fact that it deals with computers generally and particularly with all types of innovations in this domain. The lostness graph shows us that almost everywhere, and at all tasks levels, the users were more lost as they were not familiar with the websites they were working on. Overall, participants were not lost on the familiar domain (.35 <.42), however, we found that were lost on the unfamiliar domain (.46 >.42). This verifies our first hypothesis.

We also found that the more the participants got used to the websites, the less they got lost in both websites, and we saw the lostness rate reduce gradually from depth level 1 to depth level 3. According to Chen et al. (2004) the more users are exposed to the same type of knowledge, the better they get at retrieving information concerning these kinds of knowledge. This justifies why the participants got less lost why performing their tasks at various depth levels, in fact they got better, the higher the depth level went.

One other very important finding we made concerns the mental model of participants. Our assumption was: ‘If users have a poor mental model of the website, then it is likely that they will experience disorientation’. The first observation is that the mental model of the participants on the familiar domain is lower than that of the unfamiliar domain, and the bigger the depth level goes, the bigger the mental model becomes, this is certainly due to the learning effect. One of the factors that could influence the mental model of participants on both sites can be the structure of the each websites. The structure of the websites is something that must be taken into account, thus, both websites we used had completely different structures, so if this difference caused the lostness of a participant, it wouldn’t be a great surprise. Nielsen (1993) said one of the reasons of lostness could also be the unfamiliarity with the conceptual structure and organization of the website.

â&#x20AC;&#x2DC;Lost in hyperspaceâ&#x20AC;&#x2122;

Page 16 of 22

The last finding we did was about the task performance of the participants on various depth levels. Generally speaking is the task performance different as you go from one website to the other. We generally realized that the performances were higher with regards to the familiar domain than it is on the unfamiliar domain. There are many reasons that can cause the high or low performances of users on a hyperspace.

The first reason we can immediately look at is the familiarity with the content of the site, we did our research by testing only students of the Faculty of Computing and Information Sciences, these students are supposed to be familiar with everything related with computers and their innovations, these students scored generally high on the familiar domain. When it came to the unfamiliar domain where they are less familiar with content, their performances decreases in some cases dramatically.

One other reason could also be the subjective satisfaction (Nielsen, 1993) indeed says that users who perceive that they have a high degree of control over computers have been found also to have positive towards computers, we realized during our experiments that students were relatively irritated due to the fact that they couldnâ&#x20AC;&#x2122;t use short cuts like Ctrl +F (find) and others to get faster to items they were searching for, these limitations could have a negative effect on the user level of control and satisfaction, thus affecting their performances.

The degree of lostness increases when tasks have a high depth level, although not significant, it can be partially stated that increasingly difficult information retrieval tasks have a tendency to lead to an increasing score on the lostness rate.

However, calculated lostness values do not seem to correlate with mental model accuracy.

The scores on mental model accuracy were rather high; while on average participants were rather lost. Thus, lostness does not lead to a lower comprehension of hypertext systems.

This could indicate that clicking on a hyperlink is not involved in cognitive overhead and therefore not influencing the mental model accuracy in a negative way.

One of the reasons for these findings might be due to the small sample size. Due to a small sample size and not the data in itself, it could be argued that the vulnerability of the formulas might differ and that this could lead to unwanted biases. Hence, this influences the external validity.

Another issue that is also brought forward by Otter and Johnson (2000) is whether the tasks are representative enough. In previous studies, some tasks proved to be unsuited for performing lostness analyses.

To conclude looking at the general picture, we can say that our hypotheses has been fully verified, referring to the results above. As stated before, we can say that students with an ICT-background performed better on the familiar domain than on the unfamiliar domain, besides their navigational expertise More research is needed on the type of tasks that are appropriate for conducting research on lostness. In addition, the relationship between lostness and mental model accuracy (or the lack of it) needs further exploration. Last, to confirm the findings of this research, a followup research could go into more depth using a larger sample size.

â&#x20AC;&#x2DC;Lost in hyperspaceâ&#x20AC;&#x2122;

Page 17 of 22

References Ahuja, J.S. and Webster, J. (2001). Perceived disorientation: an examination of a new measure to assess web design effectiveness. Interacting with Computers, 14, 15-29.

Anderson, J.R. (2005). Cognitive Psychology and its Implications, 6th edition. New York, NY: Worth publishers. ISBN 0-7167-0110-3.

Chen, S.Y., Fan, J.P. and Macredie, R.D. (2006) Navigation in hypermedia learning systems: experts vs. novices. Computers in Human Behaviour.

Danielson, D.R. (2002). Transitional Volatility in Web Navigation: Usability Metrics and User Behavior. M.S. Thesis, Symbolic Systems Program, Stanford University.

Johnson-Laird, P.N., Girotto, V. and Lenrenzi P. (1998). Mental models: a Gentle Guide for Outsiders. Retrieved from http://www.si.umich.edu/ICOS/gentleintro.html.

Kurtz, A. (1986). Mental Models - a theory critique. Retrieved from http://mcs.open.ac.uk/yr258/ment_mod/ Nielsen J. (1993). Usability Engineering. San Francisco: Morgan Kaufman Otter, M. & Johnson, H. (2000). Lost in hyperspace: metrics and mental models. Interacting with Computers, 13, 1-40.

Preece, J., Rogers, Y. amd Sharp, H. (2002). Interaction Design: Beyond Human-Computer Interaction. N.Y.: Wiley, ISBN 0-471-49278-76 Smith P.A. (1996). Towards a practical measure of hypertext usability. Interacting with Computer, December 1996, vol. 8, no. 4, pp. 365-381(17).

‘Lost in hyperspace’

Page 18 of 22

Appendix A. Questionnaires Prequestionnaire Algemene gegevens Leeftijd:……………………………………………………….

Geslacht:

man vrouw

Beroep:………………………………………………………..

Hoeveel uur per dag zit u achter internet? i. < 1 uur ii. 1-3 uur iii. > 3 uur Hoeveel verschillende websites raadpleegt u dan in die tijd? iv. < 5 websites v. 5-10 websites vi. > 10 Ervaring met het web We willen je vragen om hieronder aan te geven in hoeverre je het met de stelling eens bent of niet (op een 7 puntsschaal). Kruis het juiste vakje aan: a. Als ik iets zoek, dan vind ik het altijd niet mee eens

mee eens

b. Binnen een website weet ik meteen waar ik moet zijn niet mee eens

mee eens

c. De websites die ik bezoek zijn goed gestructureerd niet mee eens

mee eens

d. Op het web voel ik me thuis niet mee eens

mee eens

e. Als ik iets heb gevonden dat in buurt ligt, dan ben ik al tevreden niet mee eens

mee eens

f. De weg vinden in een in hyperspace is vergelijkbaar met de weg vinden in de echte wereld

‘Lost in hyperspace’

niet mee eens

Page 19 of 22

mee eens

g. Met de website http://www.vistainfo.nl ben ik bekend niet mee eens

mee eens

h. Met de website http://www.geschiedenis.nl ben ik bekend niet mee eens

mee eens

‘Lost in hyperspace’

Page 20 of 22

Postquestionnaire We willen je vragen om hieronder aan te geven in hoeverre je het met de stelling eens bent of niet (op een 7 puntsschaal). Kruis het juiste vakje aan: 1.

Ik voelde me verdwaald niet mee eens

2.

Ik voelde me alsof ik rondjes maakte niet mee eens

3.

mee eens

Ik voelde me gedesoriënteerd niet mee eens

7.

mee eens

Ik wist niet hoe ik op de gevraagde pagina moest komen niet mee eens

6.

mee eens

Navigeren tussen pagina’s was een probleem niet mee eens

5.

mee eens

Het was moeilijk om een pagina te vinden die ik al had gezien niet mee eens

4.

mee eens

mee eens

Na een tijdje verkennen had ik geen idee waar ik heen moest gaan niet mee eens

mee eens

‘Lost in hyperspace’

Page 21 of 22

Appendix B. Information Retrieval Tasks Familiar domain Website: www.vistainfo.nl IR tasks with depth level 1 I) Waar zorgt het Apple programma Bootcamp voor? (home link ‘Vista nu ook serieuze optie voor Applebezitters’; antwoord dat je windows in OS X Tiger kan opstarten )

II) III)

Met welk programma kun je bepalen of je klaar bent voor Vista? (home bent u vista ready?; antwoord Windows Upgrade Advisor)

Waardoor kan Windows crashen naast de bewegende cursor?

(home link ‘vista kan crashen door een bewegende cursor’; antwoord door iets simpels als een opzettelijk verkeerd geprogrammeerde bewegende cursor)

IV)

Wat moet je zijn (soort gebruiker) om backups te maken? (home link ‘backups maken in windows vista’; antwoord administrator)

<MENTAL MODEL TASK depth level 3> V) Wat is het doel van Windows ReadyBoost? (home wat biedt vista? algemeen Windows ReadyBoost; antwoord prestaties verbeteren)

IR tasks with depth level 2 VI) Wat houdt een ‘schone’ installatie in? (home vista installeren schone installatie; antwoord dit houdt in dat u Windows Vista op een lege harde schijf zet of dat u tijdens de installatie de vorige versie van Windows eerst wist )

VII)

Noem 2 voordelen waarom je zou kiezen voor een 64 bit systeem boven een 32 bit systeem? (home bent u vista-ready? 32 versus 64 bits; antwoord naast de betere snelheid en performance, de betere veiligheid en stabiliteit)

VIII) Wat is ‘phishing’? (home wat biedt vista? beveiliging; antwoord een techniek waarbij door middels van slinkse wegen geprobeerd werd om persoonlijke gegevens te ontfutselen)

IX)

Wat bereik je met de wizard ‘Programmacompatibiliteit’? (home vista gebruiken vista workshops; antwoord kunt u veel oudere programma's toch goed binnen Vista laten werken)

IR tasks with depth level 3 X) Wat houdt een sluimerstand in? (home wat biedt vista? algemeen sluimer- en slaapstand; antwoord uw sessie wordt op de harde schijf opgeslagen en de computer wordt uitgeschakeld. Wanneer u de pc de volgende keer ingeschakeld wordt uw sessie weer hersteld.)

XI)

Wat kan je met een schaduwkopie? (home wat biedt vista? beveiliging link; antwoord Dan kunt u gebruik maken van de optie 'Schaduwkopie' waarmee voorgaande versie van documenten snel zijn terug te halen.)

XII)

Wat is de toevoeging van Aero aan de taakbalk? (home wat biedt vista? uiterlijk aero glass; antwoord dat er miniatuurweergaven zijn)

XIII) Wat doet Windows SideShow?

(home wat biedt vista? handige extras windows sideshow; antwoord Hiermee wordt het mogelijk om apparaten van een extra - SideShow compatible - schermpje dat extra informatie verschaft.)

<MENTAL MODEL TASK depth level 3> XIV) Wat is het doel van Windows ReadyBoost? (home wat biedt vista? algemeen Windows ReadyBoost; antwoord prestaties verbeteren)

‘Lost in hyperspace’

Page 22 of 22

Unfamiliar domain Website: www.geschiedenis.nl IR tasks with depth level 1 I) Wat voor conferentie vond er plaats in Den Haag in 1899? (home thema; antwoord vredesconferentie) II) Waar koos Hendrikus Colijn voor?

III) IV)

(home personen; antwoord onderwijs)

Wie is de tijdelijke voorzitter van de Europese Unie? (home actueel; antwoord Duitsland) Wanneer bouwde China de Chinese muren? (home recensies; antwoord tussen 221 en 206 v.C.)

<MENTAL MODEL TASK depth level 3> V) Wie beheerde eerder de kolonie Nederlands-Indië? (home thema nederlands-indië verenignigde oostindische compagnie)

IR tasks with depth level 2 VI) Wie stonden er tijdens de koude oorlog tegenover elkaar?

VII)

(home thema koude oorlog; antwoord Sovjet Unie en de Verenigde Staten)

Waardoor is Leopold van Ranke bekend geworden?

(home personen (historici) leopold von rake; antwoord als grondlegger van de kritische (objectieve) geschiedschrijving en als meester van de bronnenkritiek)

VIII) ‘The Rise of Sharpe’ is een tv-serie. Over welke periode gaat dit?

IX)

(home recensies (historische film) The Rise of Sharpe; antwoord napoleontische periode)

Hoe noem je de periode waarin Leonardo Da Vinci leefde? (home periodes renaissancetijd en de 16e eeuw; antwoord renaissancetijd)

IR tasks with depth level 3 X) Waarvoor is het spel kaatsen opgericht? (home thema sportgeschiedenis geschiedenis van het kaatsen; antwoord )

XI)

Wat was het beroep van Dries van Agt?

XII)

In welke landen speelde ‘Een burg te ver’ zich af?

(home personen van colijn tot kok dries van agt; antwoord premier) (home recensies historische films een brug te ver; antwoord engeland en Duitsland)

XIII) Tot wanneer regeerde koning David?

(home periodes oudheid 1000 tot voor christus; tot 966)

<MENTAL MODEL TASK depth level 3> XIV) Wie beheerde eerder de kolonie Nederlands-Indië? (home thema nederlands-indië verenignigde oostindische compagnie)