10
E. Karapanos et al. / Int. J. Human-Computer Studies ] (]]]]) ]]]–]]]
4. Study 2: benefits and drawbacks of the constructive and the value-account version of iScale While iScale appeared to be a viable alternative to free hand graphing, the comparative benefits and drawbacks of both iScale variants merited a second study. We compared the constructive and the value-account version of iScale to a control condition that entailed reporting one’s experiences with a product without any support through graphing. We focused on the number, the richness, and the test– retest consistency of the elicited experience reports. 4.1. Method 4.1.1. Participants Forty-eight individuals (17 female, median age¼ 23, min ¼ 18, max ¼ 28) participated in the experiment. They were all students at a technical university and were rewarded for participating in the experiment; 19 of them majored in management related disciplines, 16 in design, and 13 in natural sciences and engineering. They all owned a mobile phone for no less than four and no more than 18 months; 16 participants owned a smart phone. No significant differences were found between participants in the constructive and the value-account condition in length of ownership (Mcon ¼ 13 months, Mva ¼ 10 months, t(46) ¼ 1.51, p ¼ .13) and type of mobile phone (five participants owned a smart phone in the constructive condition, 11 participants in the value-account, w2 ¼ 2:3, p ¼ .13). 4.1.2. Materials Three different versions of iScale were used in the experiment: constructive, value-account, and no-graphing (control). The constructive and value-account versions employed the two distinct graphing approaches described earlier. No-graphing was a stripped-down version of iScale, with the graphing interface completely removed. Thus, users were only provided with the interface to report experiences (see Fig. 3a) and this was used as a control condition to test the effect, if any, of graphing. 4.1.3. Study design A 3 # 2 study design was employed with mode of recall (constructive, value-account, no graphing/control), and product quality being reported (i.e. ease-of-use versus innovativeness) as independent factors (see Fig. 5).
Product quality
Mode of recall Constructive
Value-Account
Control
Ease of use
A
B
C
Innovativeness
D
E
F
Fig. 5. Study design.
4.1.4. Procedure Participants joined two sessions, each one lasting approximately 40 min and separated by approximately 1 week (minimum: 7 days, maximum: 10 days). During the first session, participants used two different tools (either of the two graphing versions of iScale and the no-graphing version) to report on two qualities of their mobile phones (see Section 3.1.2 for a motivation of the chosen product qualities). For instance, participants in condition 1 (see Fig. 6) used the constructive iScale to report on ease-ofuse, followed by the no-graphing tool to report on innovativeness. During the second session, participants used the same combinations of tool—product quality, but in reverse order (see Fig. 6). As such, participants’ consistency across those two sessions could be used a measure of test–retest consistency of the recall process. 4.1.5. Dependent variables and expectations Despite the fact that the study was explorative in nature, a number of predictions about the differences in performance of the three versions of the tool can be made. Number of elicited experience reports: Based on existing evidence that the reconstruction of events in a serial chronological order cues the recall of temporally surrounding experiences and related contextual cues (Anderson and Conway, 1993), it was expected that the constructive iScale will result in an increase in the number of experiences being reported. For the value-account iScale, which makes it more difficult for participants to reconstruct their experiences in a chronological order, the difference to the control condition was expected to be smaller. Richness of elicited experience reports: Similar to number of elicited experience reports, we expected that reconstructing in a chronological order would lead to more contextual cues, thus providing richer insight into users experiences. Such contextual information may relate to temporal (i.e. when did the event happen), factual (i.e. what happened), social (i.e. who was present) and others. To identify these different factors of richness, we submitted the experience reports to a qualitative content analysis (Hsieh and Shannon, 2005) (see Section 4.2.2 for a more elaborate description of this process). Test–retest consistency in time estimation: As participants are expected to recall more contextual cues in the constructive iScale, this should increase the test–retest consistency in recalling factual details of the past experiences, such as temporal information (e.g. when did the experience take place) (Kahneman et al., 2004). We further predict that graphing in general (even in the value-account condition) will result in a more consistent recall of such temporal information, as graphing provides a temporal overview of the recalled experiences. To assess this, we coupled experience reports from the two sessions of the study that referred to the same experience and computed the difference in estimated time across the coupled experience reports (see Section 4.2.3 for a more elaborate description of this process).
Please cite this article as: Karapanos, E., et al., Reconstructing experiences with iScale. International Journal of Human-Computer Studies (2012), htt p://dx.doi.org/10.1016/j.ijhcs.2012.06.004