AR[t] 3

03

may 2013

augMeNted reality, art aNd tecHNology

re+Public: re+imagining Public space bc “Heavy� biermann

Pre-digital augmented reality Maarten H. lamers vestibular stimulation

as an augmented reality layer? antal ruhl

AR[t] Magazine about Augmented Reality, art and technology

May 2013 2

How & Nosm Mural Augment, The Heavy Projects, article on page 8

3

Colophon ISSN 2213-2481

Contact The Augmented Reality Lab (AR Lab) Royal Academy of Art, The Hague (Koninklijke Academie van Beeldende Kunsten) Prinsessegracht 4 2514 AN The Hague The Netherlands +31 (0)70 3154795 www.arlab.nl info@arlab.nl

Editorial team Hanna Schraffenberger, Mariana Kniveton, Yolande Kolstee, Jouke Verlinden

Contributors AR Lab & Partners: Wim van Eck, Edwin van der Heide, Pieter Jonker, Maarten Lamers, Maaike Roozenburg Guest Contributors: Alejandro Veliz Reyes, Antal Ruhl, Lotte de Reus, Matt Ramirez, Oliver Percivall, Robin de Lange, BC “Heavy” Biermann

Graphic design Esmé Vahrmeijer

printing Klomp Reproka, Amersfoort

CoVer Our AR[t]y cover is a work by Royal Academy of Art student Donna van West who participated in the Smart Replica’s project, see: www.donnavanwest.nl

www.arlab.nl

4

Table of contents

30 welcome to AR[t]

Re+Public: Re+Imagining Public Space

06

Who owns the space 2

08

The Misadventures in AR

BC “Heavy” Biermann — PhD

MaRty, the new affordable headset — Part 2

From: Hanna Schraffenberger To: Lev Manovich

Pre-digital AR Maarten H. Lamers

Vestibular stimulation as an AR layer?

Yolande Kolstee

Oliver Percivall

augmented pedagogies

16

Alejandro Veliz Reyes

A Study in SCARLET

Pieter Jonker and Yolande Kolstee

Subject: Interview

50

Matt Ramirez

18

Introducing AR to First Year Graphic Design Students

Edwin van der Heide

Nonlinear stories told by cups and saucers

44 50 56 62

Yolande Kolstee

24

Augmented Education

26

How did we do it

Robin de Lange

Wim van Eck

Antal Ruhl

Believability

40

30

AR[t] Pick: Immaterials Editors' Choice

64 72 76

34

Lotte de Reus 5

fiNe art studeNts sHow tHeir work oN tHe PoP-uP gallery

two graPHic desigN studeNts (dave PoPPiNg aNd gabor kerekes) Made a MoirĂŠ-PatterN as a decodiNg key

6

WELCOME... to the third edition of ar[t], the magazine about Augmented Reality, art and technology!

In this issue we present articles by contribu-

In the second part of this issue, special attention

tors from all over the world who are involved in

is given to education: Experiences of bringing AR

stretching the borders of augmented reality, on

into the classroom are presented in separate ar-

the edge of art and technology. We feature both

ticles by Alejandro Veliz Reyes, Matt Ramirez and

articles with a philosophical perspective and ar-

Yolande Kolstee. The results are a wide variety

ticles with a more technical point of view.

of implementations and reflections in different creative contexts. Robin de Lange considers the

In Re+Public’s article we can read about blurring

long-term ramifications of extending both the

private properties’ boundaries, in the attempt to

mind and cognition itself with AR.

leverage AR to allow artists to make incursions into public spaces, and in ways they were previ-

Wim van Eck continues his AR tutorials in the se-

ously physically unable to do.

ries ‘How did we do it’, describing some AR software programmes which are widely available.

We are very pleased to introduce to you the improved version of Marty: the video see-through

In our new section AR[t] Pick, we share artworks

AR head-up display. These drawings & 3D model

that caught our eye. In this issue, we have chosen

will be downloadable from our website, to print

'Immaterials' — the result of a collaboration be-

out at your local 3D print facility!

tween the onformative design studio and Christopher Warnow.

Hanna Schraffenberger sets about interviewing Lev Manovich, well-known among many people

We invite you to visit our new website, via the

since the publication of his book ‘The Language

same uRL www.arlab.nl, which will provide you

of New Media’ in 2001. Maarten Lamers takes us

with all the information on our artistic and tech-

back to pre-digital AR with his story on Pepper’s

nical research — but also that of fellow research-

Ghost and at the other end of the spectrum An-

ers, and information about our experiments in

tal Ruhl explores the potential of using vestibular

the cultural domain. On this website, in the part

stimulation in order to create new AR experienc-

News Picks we post short news items on AR art-

es. Crucial to AR experiences is the concept of

ists, scientists, events and developments. Please

believability, explored by Edwin van der Heide.

feel free to contact us to tell us what caught your

Lotte de Reus discusses a spatial audio interven-

eye, which might lead to an item there. We are

tion to enhance museum exhibits.

confidant you will enjoy this issue. Should you like to contribute to issue #4, look out for our call for

A recurring topic is set out by Yolande Kolstee:

contributions, which will be posted on our web-

the legal ramifications of AR initiated in AR[t] #2.

site soon!

Furthermore, we feature a short science fiction sequel by Oliver Percivall.

Yolande Kolstee, Head of AR Lab 7

outdoor advertisiNg: ar | ad takeover (Nyc, 2011), roN eNglisH, photo by will sHerMaN

re+Public: re+iMagiNiNg Public sPace

BC “HEAVY” BIERMANN — PHD

8

RE+PuBLIC A creative collaboration between The Heavy Projects (Los Angeles) and the Public Ad Campaign (New York City), Re+Public is dedicated to using emerging media technology, and augmented reality (AR) in particular, to alter current expectations of our public media environment generally dictated by property ownership, the ability to pay for its usage, or a willingness to break the law. Blurring private property boundaries, Re+Public seeks to leverage AR in an effort to allow artists to make incursions into public spaces in ways they were previously physically unable to do. With this goal in mind, Re+Public has developed an experimental mobile device application that digitally resurfaces three specific areas of public space: outdoor advertising, murals, and buildings. As such, this article focuses on these domains and demonstrates how Re+Public has used AR to transcend current private property boundaries, which lies at the heart of our endeavor to re+imagine public space as a more open visual commons.

OuTDOOR ADVERTISING: AR | AD TAKEOVER (NYC, 2011) AR presents unique opportunities to creatively problematize the political and economic systems that shape the awareness of both individuals and the public writ large. Seeking to understand a profit-driven governmentality and its affect on

9

public space, Re+Public investigated how civic

sumptive message. Specifically, users could trigger

authorities allow certain private parties to profit

web-based information related to the showcased

while preventing or discouraging other forms of

artists whose work has historically addressed com-

public media production. To this end, we envi-

mercial advertising in public space such as: Ron

sion AR as the first step in the evolution of better

English, Dr. D, John Fekner, PosterBoy, and OX.

tools of expression that can democratize public media production.

We foresee AR mobile device technology as a first step in the transformation of public space into an

The AR | AD Takeover used street level ads and bill-

arena shaped by user created content. In other

boards to trigger a curated digital art installation

words, AR is an incremental step towards showing

that displayed on mobile devices. In New York City,

the public an alternate view of their landscape,

we augmented ads in Times Square with artistic

which commercial ads do not necessarily have to

content. Our digital infiltration into public space

dominate. Once individuals experience this AR

and takeover of commercial ads created a place of

version of reality, they might start demanding a

dialogic interaction rather than a monologic con-

better version of public messaging than the billboard default.

Outdoor Murals: Bowery Wall (NYC, 2012) and Wynwood Walls (Miami, 2012) In contradistinction to the use of AR to problema-

they move in perspectival relation to the viewer.

tize the consumptive monologue of outdoor ad-

It is precisely this kind of spatial “aura” that dis-

vertising, at both the Bowery and Wynwood Walls

tinguishes AR from other types of emerging me-

sites, we used AR to rupture public space with a

dia technology. AR connects the digital with the

new kind of artistic interactivity.

physical in an intimately “present” way, whereas other technologies tend to disconnect the view-

The Bowery Mural Wall is an outdoor mural ex-

er from their immediate physical surroundings.

hibition space in Manhattan. Owned by Goldman

Viewing previous murals online, for example, re-

Properties since 1984, real estate developer and

moves the viewer from the space by placing them

arts supporter Tony Goldman started the Bowery

squarely in the absent, digital world.

Mural with Jeffery Deitch and Deitch Projects.

In 2008, the mural series commenced with a

In 2009, Goldman Properties and Tony Goldman

recreation of Keith Haring’s famous 1982 mural

who was looking to transform the industrial ware-

followed by work by such recognized artists as Os

house district of Miami also conceived the Wyn-

Gemeos, Faile, Barry McGee, Aiko, Kenny Scharf,

wood Walls. Beginning with the 25th and 26th

Retna, and Shepard Fairey.

Street complex of six separate buildings, Goldman endeavored to create a center that devel-

In 2012, Re+Public used AR to resurrect murals

oped the area’s pedestrian potential.

that once existed on the wall. In other words, by pointing a mobile device at the present mural, us-

Drawing from an international pool of talent, art-

ers are able to see the former murals, in situ, that

ists who have contributed to the Wynwood Walls

the current artist has painted over. While users

include Os Gemeos, Invader, Nunca, Saner & Sego,

could certainly view the previous murals online,

and Swoon among many others. During Art Basel

AR permits users to see these murals as if they

2012, Re+Public was commissioned by the Wyn-

were actually back on the wall, in the space, as

wood Walls to create an AR experience. In addi-

10

outdoor Murals: bowery wall (Nyc, 2012), resurrected keitH HariNg

tion to resurrecting a Shepard Fairey mural that he

sion of creative content. In addition to an immer-

recently painted over with a new mural in tribute

sive garden with a bridge, stream, and flowing

to the recently deceased Tony Goldman, Re+Public

waterfall, we created a 3D Kabuki theater that al-

created 3D, interactive environments for four

lowed users to walk into a digitized version of the

other murals by How and Nosm, Ryan McGinness, Aiko, and Retna. Additionally, we worked directly with MOMO and collaborated on an AR version of his indoor mural at the Nicelook Gallery on site.

In considering our deployment at the Wynwood

AR is an incremental step towards showing the public an alternate view of their landscape, which commercial ads do not necessarily have to dominate.

Walls in particular, we wrestled with this new

mural where all of the elements were separated in

type of work and wondered if it constituted a new

Z-space. With the How and Nosm, we created an-

mode of art. Without pretending to discern any

other immersive, rather abstract 3D environment

immediate resolution, it is arguable that the AR

and permitted users to both pull apart and recon-

assets represent original works in that they con-

struct the mural elements. With the Retna, we

tain a sufficiently new visual expression of ideas,

built the mural shapes in 3D and animated them

over and above those embodied in the 2D mural. In

to extend out of the wall and placed them both

the case of each traditional mural, the AR overlay

on the ground and above the wall. Finally, with

used the 2D paintings as feature tracking markers

the McGinness, we made it appear as if the paint

and source material to produce an original expres-

colors were draining out of the mural.

11

Retna Mural Augment, full view

Retna Mural Augment, screenshot

12

AuGMENTED ARCHITECTuRE: PEARL PAINT, WILLIAMSBuRG ART & DESIGN BuILDINGS (NYC, 2012) BRADBuRY BuILDING (LA, 2012)

cal aura that may have drawn the viewer to the building in the first place. In other words, the AR assets should maintain some logical connection to the building or space upon which we have attached them.

FINAL REMARKS

In addition to using both outdoor ads and murals as the markers that trigger our AR deployments, we

Whether it is outdoor ads, murals, or entire build-

have also experimented with using buildings, and

ings, Re+Public seeks to continue to deploy AR

their unique architecture much in the same way.

in an effort to democratize access to our shared

To this end, we digitally resurfaced, or “skinned”

visual environment and alter the current expec-

physical buildings in urban centers by overlaying

tations of urban media in accomplishing our core

3D content onto the physical environment.

mission of re+imagining public space. It is vital to the health of any city that its inhabitants are

ultimately, in our attempt to use AR to re+imagine

able to participate, in some meaningful way, in

public space, we really see the city as a canvas

the visual urban messaging systems that surround

that allows for a multiplicity of voices to enter

them. With the coming advancements in wear-

into our visual landscape, rather than the cur-

able computing, the digital overlay will become a

rent commercial hegemony. Working with mu-

much more seamless and natural part of our daily

ralist MOMO, we chose three buildings that had

existence. It is our hope that these early entrants

a particular cultural significance and, using AR,

will help create experiences that consider art and

made it possible for MOMO to put his art on build-

design as an important part of the way the public

ings that he could not have accessed in his tradi-

adopts this technology.

tional 2D format. Converting MOMO’s 2D designs into digital 3D models optimized for mobile, we placed his art on both the Pearl Paint and the Williamsburg Art & Historical Center buildings in New York City. Both structures have a long “art history” in the city and AR allowed us to blur the lines between private and public space.

In Los Angeles, we converted the Bradbury Building, site of many interior shots in the film Blade Runner (1982) into a futuristic version of itself.

In this way, instead of placing converted art on the structure, we created the first example of what we refer to as “city visions.” Specifically, we used AR to provide an artistic rendering of the re+imagined building by projecting it into a Blade Runner style future. This city vision type of deployment potentially provides more practical architectural and urban planning uses and maintains our notion that the AR experience should be spatially relevant in order to maintain the physi-

williaMsburg art & Historical ceNter buildiNg augMeNted witH MoMo urbaN art

13

bc “Heavy” biermann Deriving his pseudonym from his penchant for philosophical discussion, BC “Heavy” Biermann possesses an interdisciplinary background that comprises technology, academia, and the arts. With a PhD in Humanities [Intermedia Analysis] from the universiteit van Amsterdam, BC has worked as both a university professor and a tech developer in Anaheim, Prague, and Saint Louis. Since 2007, BC has internationally presented his academic work, which explores augmented reality, art and semiotics in public space.

As a kind of synthesis between scholarly inquiry and emerging media, BC founded The Heavy Projects [and its collaborative spin-off Re+Public] to investigate how the fusion of creativity and technology can uncover new modes of relaying ideas. Building upon existing technological and theoretical frameworks, Heavy creates innovative interfaces between digital design and physical worlds in ways that that provoke the imagination and challenge existing styles of art, design, and interaction.

After finally giving up his painfully amateur skateboarding career due to a bum right knee… BC plans to use his extra time continuing to examine meaningful ways to fuse tech + creativity.

outdoor Murals: wyNwood walls (MiaMi, 2012), resurrected sHePard fairey, photo by jordaN seiler 14

15

Marty, the new affordable headset — Part 2 Pieter Jonker AND Yolande Kolstee

In AR[t] 2 we announced the new MARTY video-

hence there is extra knowledge on scale. In mark-

see-through headset as a design based on the

erless systems, natural features like dominant cor-

Sony HMZ T1. It was designed by Niels Mulder

ners or lines in the scene are tracked, no assump-

from studio RAAR on assignment of the TU-Delft,

tions on scale can be made and two cameras that

partner of the AR Lab, with the aim to do re-

see that same feature are necessary.

search on co-operative AR. One of the aims of the AR Lab is to stimulate AR in the world by putting

And finally: Yes we were already decades ago

our developments in public domain. When the

inspired by fighter pilots with heads-up display

new AR Lab website will be in the air at about the

and of course Steve Mann, the geek that walked

same time as this AR[t] 3 magazine is published,

around with an AR display. But also lower back-

we will post the design files as well as a photo

pain, cramped shoulders from long computer days

series how to assemble the Marty, so anyone with

and the sighing, “why can laying on bed or in an

access to rapid prototyping facilities can repro-

armchair not the ceiling or the white wall, the air,

duce Marty for his/her own scientific or artistic

or anything at any time be my display?”

research. Keep an eye on our website as we will also link to the software to do 3D pose tracking

Disclaimer

based on natural feature tracking.

The industrial design of Marty is meant to do exWe are well aware that the Sony HMTZ T2 is on

periments by researchers, artists and designers

the market, which has another face mask. We are

in the area of AR and stimulate industry to come

aiming at adapting Marty to version 2.0 that is

up with affordable AR equipment. It is, however,

based on the HMZ T2. Due to a lack of manpower

subject to copyright, meaning that no large scale

and funds this might, however, take a while. In

industrial production can be based on this design

the mean time be our guest and contribute to AR

without the consent of the copyright owner. For

to come up with your own solutions. If they are

more information: info@arlab.nl.

valuable we will publish about it in our next AR[t] magazine.

References

We are also aware of Google glasses and other

¢¢

Bridging the multiple reality gap: Applica-

equipment and we welcome those developments!

tion of Augmented Reality in new product

However, still many companies do not understand

development. Anna P. Chatzimichali, Wim

that you need two camera’s to see virtual objects

Gijselaers, Mien S. R. Segers, Piet Van den

in 3D and two camera’s to track (salient) key-

Bossche, Hetty van Emmerik, Frido E. Smul-

points in 3D in order to track your head pose while

ders, Pieter P. Jonker, Jouke C. Verlinden In

walking around with your headset in any unknown

proceedings of the IEEE International Con-

environment. AR-toolkit like markers can do with

ference on Systems, Man and Cybernetics,

one camera as the size of the marker is known and

Anchorage, Alaska, USA, October 9-12, 2011

16

MARTY, THE NEW AFFORDABLE HEADSET. Image by FERENC MOLNAR

ON OuR NEW WEBSITE (WWW.ARLAB.NL) WE WILL SOON POST A PHOTO SERIES ON HOW TO ASSEMBLE THE MARTY HEADSET.

17

---------- Forwarded message ---------From: Hanna Schraffenberger <Hanna@arlab.nl> Date: 2013/4/3 15:53 Subject: interview AR[t] magazine

Dear Lev, Maybe you remember me from Facebook.

I work at the Augmented Reality Lab in The Hague and I am one of the editors of the AR[t] magazine. When I read your article The Poetics of Augmented Space, I realized that I would like to interview you about Augmented Reality for the AR[t] magazine. A short time ago, I finally also read The Language of New Media. As a consequence, I’d like to interview you even more. So I hope you’ll agree to an interview for the magazine?

Best regards, Hanna P.S. After my last few interviews, my supervisor (Edwin van der Heide) told me that I could/should be more critical towards my interview partners. So I’ll challenge myself to challenge you.

P.P.S. Maybe we can print my questions in issue 3 and your answers in issue 4?

18

Manovich, Lev. The language of new media. The MIT press, 2001. image source: www.jewishphilosophyplace.wordpress.com

To: Lev Manovich

Augmented Reality

where to go and what to look at. In contrast to ‘typical’ visual AR, the user is presented with au-

what is augmented reality?

ditory information that relates to the immediate surrounding space. Personally, I would call this

To begin with, I would like to ask you what you

Augmented Reality. Wouldn’t you?

consider Augmented Reality (AR) to be. In The Poetics of Augmented Space you describe AR as ‘the laying of dynamic and context-specific infor-

Augmented Space

mation over the visual field of a user’. It would be great if you’d address the topic once more.

Firstly, because our readers might not have read

what is special about ar compared to other forms of augmented space?

your article. And secondly, because I think that this point of view unnecessarily limits AR to the

In your article The Poetics of Augmented Space

visual sense.

you discuss the concept of Augmented Space. Augmented Space refers to all those physical spaces

In The Poetics of Augmented Space, you mention

that are overlaid with dynamic information such as

Janet Cardiff’s audio walks as great examples of

shopping malls and entertainment centers that are

laying information over physical space. These

filled with electronic screens and all those places

walks are designed for specific walking routes.

where one can access information wirelessly on

While navigating the environment, one gets to

phones, tablets or laptops. Besides AR, you men-

listen to a mix of edited sounds that blend in with

tion several other technological developments in

the sounds of the surroundings, as well as spo-

the context of Augmented Space, among which,

ken narrative elements and instructions such as

for example, monitoring, ubiquitous computing,

19

tangible interfaces and smart objects. Is AR just

that in AR, something virtual augments something

one of many related recent phenomena that

real. More specifically, the virtual augments that

play a role in overlaying the physical space with

to which it relates. In our view, space is one of

information? What’s special about AR compared

the possibilities, but likewise, we have considered

to other forms of Augmented Space?

things like augmented objects, augmented humans, augmented perception, augmented content

What else can be augmented?

and augmented activities. What is augmented depends on what the additional content relates to.

Something I really like about your article is that

I am curious whether you’d agree. Do you think

you see augmentation as an idea and a practice

that all forms of augmentation bring along an

rather than a collection of technologies. However,

augmentation of space or influence our experi-

so far, you have only discussed the augmentation

ence of the immediate surrounding space?

of space. I was wondering whether you have considered other manifestations of augmentation as well. I don’t think augmentation is limited to a

Information and space — one coherent gestalt?

space or an environment. I’d even say that often it’s not the space that is augmented, but some-

In The Poetics of Augmented Space you raise a

thing else.

question that intrigues me a lot. Do the real space and the dynamically presented information add up

For example, you mention software that performs

to one single coherent phenomenological gestalt

tasks according to the mood, pattern of work, fo-

or are they processed as separate layers?

cus of attention or interests of their user. However, I am doubtful whether our experience of a

I am a bit of a sound-person and it has always fas-

space is affected by this kind of information. Let’s

cinated me that sometimes the sounds of a radio

imagine that my phone registered that I have been

seem to mix in with environmental sounds. For

sitting still for a long time and reminds me to take

example, the ticking of a red streetlight might

a short break to stretch my legs. This information

perfectly mix in with the rhythm of the song that is currently playing. Listening to a radio play, an

In AR, something virtual augments something real.

event could sound so real and so nearby, that I’d turn around, just to find, that nothing is happening there. But of course, most often, the sound of the radio just exists as a separate, independent layer of content. The voice of the newsreader doesn’t mix

relates to one individual in the space (me), to the

with the voice of my colleague, nor does it relate

activity the person is performing (sitting still), but

to my environment. Most of the time, a song is just

I don’t think it has anything to do with the sur-

a song, and has nothing to do with the surround-

rounding space. Hence, I might consider it an aug-

ing space. So judging from my experience of listen-

mentation of the activity (not moving, sitting still)

ing to the radio, information and the surrounding

or an augmentation of the user (me), but I don’t

space can be perceived as one single mixed thing

consider it an augmentation of space.

as well as independently. But besides these two options, there are more possibilities. For example, the

Edwin (my supervisor) and I have recently given

newsreader might tell me about a traffic jam and

this topic a lot of thought and we were fascinated

thereby inform me about my immediate physical

by the questions: “What is actually augmented in

space. Here, the information and my spatial sur-

Augmented Reality? What else can (we imagine

roundings aren’t perceived as a single gestalt, but

to) be augmented?” We came up with the answer,

nevertheless, there is a relationship between both.

20

I think the same is true for Augmented Space.

equally to renaissance paintings and to mod-

Often, information and space might be related,

ern computer displays. When we imagine a

even when they don’t add up to one phenomeno-

typical AR scenario in which virtual objects

logical gestalt. So some questions I’d like you to

are integrated into a real scene (e.g. a virtual

answer with respect to Augmented Space are:

bird is sitting on a real tree) there is no sec-

when are information and space perceived

ond space. It’s the same physical space, which

independently from each other — would you

contains both virtual and real elements. is

still call these occurrences augmented space?

this a fundamental change in visual culture?

when are information and space perceived as separate but related layers? and when and why

ar & the quest for realism

do they add up to one single gestalt?

The quest for realism in computer graphics

New Media

is something that has always bored me. You note that new technological developments illustrate how unrealistic the previous ex-

One of the main questions I want to ask you is:

isting images were. At the same time they

what makes augmented reality special? I have

remind us that current images will also be

posed that question with respect to other forms

superseded. I was wondering: How does ar

of augmented space. I’d like to ask it again with

fit in the widespread aspiration towards

respect to the history of new media.

realism? On the one hand, visual AR could be considered a huge step back. The 3D models

Personally, I don’t think of AR as a recent phe-

that are usually integrated in real space don't

nomenon. Of course, there are more and more

come close to the level of photorealism we

so-called AR applications, AR technologies and

know from cinema. On the other hand, the

new media works that work with AR. However,

virtual leaves the realm of virtual space and

when we consider the concept of AR, we find

enters our real physical environments — with

examples that date back centuries. An example

respect to that the images might be experi-

of ancient AR is the Pepper’s Ghost trick (which

enced as more realistic than ever…

is discussed by Maarten Lamers on page 24). It uses a second room, glass and special lighting in

will ar take the quest for realism to a new

order to let objects seem to appear, disappear

level? I can imagine, when striving for real-

or morph into each other in an otherwise real,

ism, the virtual things that appear to exist in

physical environment.

our physical space should not only look like real things — ideally they also feel like them,

But even if the concept isn’t new, current manifes-

smell like them, taste like them and behave

tations of AR might still bring something new and

like them. will photorealism be traded in

special to the table. if we look at contemporary

for a form of realism that encompasses all

ar and compare that with other forms of new

senses? do you think new media will de-

media, what’s special about it and what isn’t?

velop towards a more multimodal form?

ar & the second space

ar & cinema

From The Language of New Media, I understood

In The Language of New Media, you relate dif-

that throughout media history, the screen was

ferent forms of new media — e.g. Virtual Real-

used to separate two absolutely different spaces.

ity, websites and CD-ROMs — to cinema. How

For example, this function of the screen applies

about the relation between ar and cinema?

21

I’m certainly not a cinema expert, but I guess

lationship to something real. Could we say that

most of what we see in visual AR has been pres-

when working with AR, artists and designers

ent in cinema for a long time. For example, AR

create a database for an existing interface?

research is very concerned with registering virtual objects in real space. As far as I understand it,

I have one more question about databases. In The

this can be seen as an analogy to compositing in

Language of New Media you write about the ele-

films: an attempt to blend the virtual and the real

ments of a database:

into a seamless whole ‘augmented’ reality. Do you agree?

“If the elements exist in one dimension (time of a film, list on a page), they will be inevitably or-

You oppose compositing to montage: while com-

dered. So the only way to create a pure database

positing aims to blend different elements into a sin-

is to spatialise it, distributing the elements in

gle gestalt, montage aims to create visual, stylis-

space.”

tic, semantic, and emotional dissonance between them. Do we have montage in AR as well? (You

In AR, virtual elements are distributed in real

give the example of montage within a shot, where

space. Can we understand this as a pure database?

an image of a dream appears over a man’s sleeping

What are the consequences of working with

head. The same could easily be done in AR.)

spatialized elements? What are the inherent limitations and possibilities when working with

Does visual AR use similar concepts as cinema?

this form? (I can imagine it has consequences, e.g.

Does cinema use other techniques to create

for storytelling? As you point out, we cannot as-

fictional realities that are not (yet) used in AR?

sume that elements will form a narrative when

Does AR use techniques that might be adapted

they are accessed in an arbitrary order...)

by cinema in the future?

AR as spatialized databases

AR & future research With The Language of New Media, you did not only One of the main claims in The Language of New

provide a theory of new media; you also pointed

Media is that at their basis, all new media works

your readers towards aspects of new media that

are databases. You argue that what artists or de-

were still relatively unexplored at that time and

signers do, when creating a new media work, is

you suggested directions for practical experimen-

constructing an interface to such a database.

tation. Are there certain aspects of Augmented Reality you consider especially interesting for

Let’s apply this database theory to a typical AR

future experiments and explorations?

scenario in which virtual objects (seem to) appear in a real environment. We can see this as

References

a database filled with virtual objects. The database might hold a virtual chair, a virtual pen and

¢¢

a virtual painting. These virtual objects are displayed as part of a real room when a user views

Manovich, L. (2001). The language of new media. The MIT press.

¢¢

Manovich, L. (2006). The poetics of aug-

the augmented environment with a smartphone.

mented space. Visual Communication, 5(2),

(Technically speaking, we could say the real world

219-240.

serves as a database index for those virtual elements.) What is the interface to access the database? Is it my phone? What does the artist create?

I think it is usually the virtual content and its re-

22

â&#x20AC;&#x153;[...] the only way to create a pure database is to spatialise it, distributing the elements in space.â&#x20AC;?

Image courtesy of Lev Manovich

Lev Manovich, The Language of New Media

23

nd, 1978

Disneyla

n Valentij

rten a a M

Pre-digital augMeNted reality MAARTEN H. LAMERS In the first issue of this magazine, I mentioned how the “Mozzies” mobile game of 2004 was my earliest encounter with augmented reality. i lied... by 26 years! in 1978 my parents took me and my brother to disneyland. besides meeting Mickey Mouse, i was most impressed by the augmented reality of Pepper’s ghost. with no computer required...

24

.net lbourke rke, pau Paul Bou Image

of courtesy

Image

cou rtesy of D avid http Llewellyn ://ravenmanor , .com

Sitting beside my brother, a mechanical fun-

ment parks and museums today, but also as

house car drove us through Disney’s “Haunted

part of modern optical see-through AR tech-

Mansion” ride. Scary stuff, of which I can re-

nology. In Microsoft Research’s HoloDesk (see

member only one thing: the car stopped, ro-

research.microsoft.com/projects/holodesk/)

tated 90 degrees to the right, facing a large mir-

project, its use is apparent. However, in cur-

ror, and on the seat between us appeared the

rent head-up see-through displays Pepper’s

scariest moving ghost ever! Instantly our heads

Ghost technology is less apparent but nonethe-

turned, facing each other. Thank God, the ghost

less used in the same fashion.

wasn’t really between us. The mirror showed us

To me it is interesting that we still rely on John

some weird illusion, most realistically.

Pepper’s idea to add digital content to our op-

Naturally, I asked my dad how the illusion

tical reality. Actually, digital technology lets us

worked. He explained something about glass,

define, render and interact with virtual con-

darkness and reflecting light. In effect, he ex-

tent. But good old-fashioned Pepper’s Ghost

plained what is known as Pepper’s Ghost: a

projection is what augments our reality with

simple but clever technique that creates holo-

that content.

graphic scenes in 2 or 3 dimensions. Combining

Who would have guessed that such basic

this with a large mirror, Disney augmented the

illusionary tricks are crucial to what we now

reality that we hold our mirrored image to be:

consider cutting-edge technology? In fact,

pre-digital augmented reality.

if you know other pre-digital augmented

Pepper’s Ghost technique was first described

reality techniques, send a short description

in the 16th century and later refined by John

to lamers@liacs.nl, and help me put AR in

Pepper around 1860. It is still used in amuse-

perspective.

25

Image by Kristin Smith

vestibular stimulation as an augmented reality layer?

Antal Ruhl Over the past few years the field of Augmented

intensity and the direction of the current. This

Reality has started to include non-visual forms

technique has been around for a long time and

of augmentation to their scope. When I became

many studies have been done in order to learn

interested in galvanic vestibular stimulation I re-

more about the human balance system, but never

alized that this could become a whole new form

has it been used as an extra sensory input. To find

of AR.

out whether GVS can be used as a new form of

Galvanic Vestibular Stimulation (GVS) stimulates

AR, I constructed a device and developed a series

the human vestibular system (balance organ) by

of experiments. In order to use GVS as a form of

means of electric stimulation. Two electrodes,

augmented reality, I needed a connection to the

usually an anode and a cathode, are placed on the

physical world. Since I also wanted to investigate

mastoids (the bones behind the ear). Applying a

what the effect would be to our balance system,

specific current to those mastoids will result in an

I used my body orientation to alter my balance.

altered balance perception related to both to the

26

GALVANIC VESTIBuLAR STIMuLATION CAN BE uSED TO ALTER BALANCE PERCEPTION

Setup To do so, I mounted an accelerometer on top of

own orientation. There were two possible modi

my head-mounted GVS device in order to measure

for the control unit. Modus A could counteract my

its orientation. This provided me with the setup

balance, while modus B could amplify it. In other

as shown in Figure 1.

words, if I was tilted to the left in modus A, it

The electrodes are incorporated in a pair of head-

would push me back to the center, in modus B, it

phones to make sure they are pressed against the

would push me further to the left.

mastoids properly. The headphones are in fact

Testing this simple setup immediately revealed

merely rings around the ears. The middle part is

some potential AR applications of vestibular stim-

left uncovered to retain the full use of usersâ&#x20AC;&#x2122; au-

ulation. While testing, I realized that, when put

ditive orientation, and to make sure this wouldnâ&#x20AC;&#x2122;t

in modus A, it felt like I was moving through a liq-

influence the results. The data from the acceler-

uid or a thick syrup-like medium. The GVS device

ometer is sent to the control unit. The control

counteracted all my movements, so it took more

unit calculates the appropriate intensity and di-

effort to move around. In modus B, it had the op-

rection of the current in real-time and provides

posite effect; it felt like my resistance was really

feedback to the electrodes.

low since the device backed up every movement

Conditions

I made. Without even using external data to augment an extra layer to reality I created an altered

The first step of my research project was measur-

world based on data from the physical world. Is

ing the effect of an altered balance, based on my

this a form of augmented reality or is it an alter-

27

The tests that I have done focused on enhanced performances in everyday life situations. For example, I was wondering whether I can diminish or amplify motion sickness, improve my balancing skills etc. During the testing phase, I have used this device for over two weeks to see if there was any progress. Tests involved bus rides, walking to targets on the street, walking over a balance beam, playing Wii Balance Board games and many other examples. Although we have to be very careful about drawing conclusions from these results (given the self-experimentation and singlesubject constraints), I found that in certain cirnate reality? Be that as it may, it does fall within

cumstances my balance actually improved while

the definition of mediated reality, which is the

using this device. This might be of use for people

broader field of manipulating users’ sensory per-

with an impaired balance system or in those situ-

ception through a wearable device, and letting

ations when accurate balance is crucial.

them experience their surroundings in a modified fashion. Using only these two simple orientation-

Can this be used as an interface?

based modi, we can already create an enriched

I demonstrated a system in which the real world

environment, but for what use? The examples

environment, (in this case body orientation,) is

point towards simulations of the physical world,

measured, modified and used as an input to alter

which could be used to train divers or astronauts

the actuators (in this case the electrodes which in

who work in other environments. But you can also

turn alter my own balance). This creates a broad

think about fighter pilots who fly in simulators

range of possibilities. But can we also use it in a

that don’t alter pilots’ balance when flying upside

more traditional AR environment? In other words,

down. This system could enhance a simulated en-

could we use this system to add computer gener-

vironment by distorting the balance organ based

ated data to an extra sensory layer of our envi-

on the simulators’ virtual orientation.

ronment? The easy answer is: Yes! It’s not hard to imagine that you can enrich game play with wave motions for example. Or simulate the G-forces in a racing game’s sharp corners. However, before we can use this, there are some practical considerations to keep in mind. Using my self-constructed device, I experienced some burn marks of the skin around the mastoids. The skin’s resistance is an important factor in getting the electric pulses down to the desired area. Using electrode gel and covering the electrodes with sponges soaked in a salt-water solution did the trick, for now. But a more practical solution must be found. Other issues were the bright light flashes I experienced while testing, and nausea; both are not very userfriendly. But these issues might all be overcome with the right execution. Another, perhaps more

Figure 1: Galvanic Vestibular Stimulation device

28

troublesome, experience I had was the adapta-

tion effect. You might remember that study from the 1960’s where a test subject was wearing reversal goggles. After a few days the brain adapted to the newly displayed environment and reversed the image back to normal. After removing the goggles, the subject saw the world upside down because the body had adapted to this new situation. The same effect occurred with my balance organ, but at a much quicker pace. While my body needed some time to get used to my ‘new and improved’ balance, it did so quite convincingly.

And it did so in two ways. When I used a constant pulse on the electrodes (a constant current in the same direction) the effect diminished quite soon.

But when I used an alternating pulse, and only changed the pulse-width to affect my balance, the effect was constant. Apparently I was much more sensitive to a change in stimulation than to the actual stimulation itself. Secondly, when I was wearing the device for about fifteen minutes, and removed it after that time, my body had to adapt to this ‘new’ non-stimulated situation. I saw that, when I was wearing the device while it was set in modus A, it gave me the experience as if I was moving through a liquid. After removing the GVS device I experienced the exact opposite.

So when I turned it off, it felt like I had switched it to modus B; thus, giving off the impression of moving through a low friction environment. Your body is extremely good at adapting to new situations and environments, which are very important design-considerations if we want to use a GVS device as an interface.

While using GVS for augmented reality purposes is still in a research stage, the possibilities are endless if you use your imagination.

This article is based on research presented at Chi Sparks 2011 (Chi Nederland 15th conference), June 23, 2011, Arnhem, The Netherlands. The

Antal Ruhl Antal Ruhl is a media artist with a background in design, science and art. Antal studied Industrial Design at the The Hague University of Applied Sciences, Media Technology at Leiden University and Design and Media Arts at UCLA, Los Angeles. Antal creates objects that let us rethink our environment. These objects vary from kinetic sculptures to interactive installations. Sometimes they are purely conceptual or formal and sometimes they serve a more commercial purpose, but they always share the goal to intrigue people. “Using the technological possibilities at hand we can enrich our work and create an engaging object. Interactive and playful objects are much better in holding someone´s attention while they convey their message”. His work can be described as visually and technically attractive with a focus on natural and physical phenomena. Antal worked at distinguished design companies in Amsterdam and Barcelona, has had several (national and international) performances and exhibition, published a scientific paper, travelled the world and currently works a freelance artist/ designer.

Antal has also started a company to develop interactive installation for brands, festivals and event. Creative in Motion is a creative brand activation studio: www.creativeinmotion.nl

presented paper, Experiments with Galvanic

For an overview of his work or to contact

Vestibular Stimulation in Daily Activities:

Antal please visit:

An Informal Look into its Suitability for HCI,

www.antalruhl.com

can be found here: www.antalruhl.com/media_tech/paper.pdf

29

Believability Edwin van der Heide

Believability is something we deal with on a con-

nevertheless draws us in without questioning its

tinuous basis. When we receive an e-mail that

realness or truthfulness. The story is believable,

claims to be from our bank with the request to

while not real, and we enjoy it. An important fac-

enter our account information on ‘their’ website

tor for a story to be believable is that we can

in order to upgrade its security, we enter a situ-

relate ourselves to it (or the story relates to us).

ation in which the content appears to be believable. We’ve learned to get suspicious and need to

The examples from the bank’s e-mail as well as

verify the credibility of the story. The e-mail is

the novel show us that believability is actually

believable, appears to be authentic, it pretends

independent from something being real or fake;

to be real, but is in fact fake. When we read a

or in other words, independent from something

novel, the story can be entirely fictional and it

being real or ‘virtual’. What we also learn from

30

Spatial Sounds (100dB at 100km/h) at Wood Street Galeries, Pittsburgh, USA, 2009 - Image courtesy Studio Edwin van der Heide

the example of the novel is that things that hap-

(2007) that fascinated me: The believability of a

pen in the story don’t have to be possible in real-

Robot. The reason that it fascinated me is that it

ity. In a novel we can meet creatures from Mars,

made me not only think about believability in the

time-travel, or never go to bed. We can ‘simply’

context of robots, but it also triggered me to think

imagine these things and believe them.

about the believability of an artwork and, more specifically, the believability of the behavior of

When I was writing my paper about the interactive

an interactive artwork (as opposed to the believ-

art installation Spatial Sounds (100dB at 100km/h)

ability of a robot’s behavior). I realized that the

for the Third International conference on Human-

believability (of the behavior) of an artwork was

Robot Personal Relationships (HRPR) I was intro-

not (yet) seen as a fundamental topic and might

duced to a topic addressed by Kerstin Dautenhahn

deserve its own study and experimentation.

31

Kerstin Dautenhahn has an interest in socially in-

reason that in Japan a lot of machines talk or in-

telligent robots: “A robot companion in a home

clude animations to, for example, welcome and/

environment needs to ‘do the right things’, i.e.

or thank the user for using it. It is interesting to

it has to be useful and perform tasks around the

question whether these machines indeed make us

house, but it also has to ‘do the things right’,

believe they show affection, and if so, if it lasts

i.e. in a manner that is believable and accept-

or wears off.

able to humans.” I tried to imagine an example and came up with the idea of a robot that makes

There is another possible difference between

and brings you a cup of coffee. Soon after I had

the coffee machine and the robot making coffee.

that idea I had to think about a full automatic

Robots are often made to look and behave like

coffee machine that grinds the beans and steams

humans (the humanoid) and the coffee machine

fresh milk for each individual cup of coffee that

isn’t. Does this then mean that the representa-

it makes. How do the two differ from each other?

tion is a requirement for believability? No, a ro-

With the robot we can imagine that he read our

bot (or interactive installation) doesn’t have to

mind and therefore made a coffee for us. We

represent something (else). It can be an abstract

imagine that the robot has a certain amount of

work that is believable on its own. In the HRPR

intelligence and, perhaps, even ‘feels’ affec-

paper about Spatial Sounds (100dB at 100km/h)

tion for us. In the case of the coffee machine,

I put it like this: “The installation can be seen as

we don’t imagine any intelligence and we simply

a non-verbal abstract robot and does not imitate

think of it as a machine without affection for us.

an animal or human-like look or behavior. It is a

If we, however, imagine that the coffee machine

machine-like object but does not resemble exist-

made that coffee especially for us things start

ing machines. Nevertheless, it allows us to some-

to change. It’s by imagining that we can change

how identify ourselves with it. Spatial Sounds

what we believe and thereby turn a machine into

(100dB at 100km/h) is an example of a believable

a believable affective robot. This might be the

robot in the sense that the visitors believe they

32

understand the behavior of the installation and

There are certain things we believe in that, once

find it worthwhile to interact with. The aspect

we discover they are fake, completely lose their

of believability is so strong that people accept

believability, while we keep on believing other

the installation as a real being and want to inter-

things that we know are fake.

act with it over and over” (van der Heide, 2011).

Interesting to read in this context is Alwin de Rooij’s (2010) graduation research project for the

References

Media Technology Master program on ‘Abstract Affective Robotics’.

¢¢

Dautenhahn, K. (2007). Socially intelligent robots: dimensions of human–robot interac-

I’m curious how the abovementioned thoughts

tion. Philosophical Transactions of the Royal

apply to augmented realities. What aspects make

Society B: Biological Sciences, 362(1480),

combinations of, and interactions between, the

679-704.

real and the virtual in augmented reality believable? We have learned that something doesn’t

¢¢

van der Heide, E. (2011). Spatial Sounds

have to be true to be believable. We have also

(100dB at 100km/h) in the Context of Human

learned that something doesn't have to be possi-

Robot Personal Relationships. In: Lamers M.

ble in order to be believable. Besides that, we’ve

H., Verbeek F. J., Human-Robot Personal

learned that something doesn’t have to represent

Relationships (HRPR 2010), LNICST Vol. 59,

something; it can be abstract and nevertheless

27-33.

be believable. The former makes me believe that believability forms an interesting perspective to

¢¢

de Rooij, A. (2010). Abstract Affective Ro-

think about what we can imagine in augmented

botics, http://mediatechnology.leiden.edu/

realities!

research/theses/abstract-affective-robotics

Spatial Sounds (100dB at 100km/h) at DAF Tokyo, Japan, 2006 — Image courtesy Studio Edwin van der Heide

33

Nonlinear stories told by cups and saucers Smart Replic as with responsive 3D audio Lotte de Reus, Jouke Verlinden, Maaike Roozenburg

Introduction

copy of the original, but to analyse, communicate

In museum exhibitions historical objects are usu-

and enhance those qualities of the historical arte-

ally shown by visual display, in a showcase with

fact that are the most meaningful to us, now.

extra textual information added to it. Museum

Here we present the design project of Lotte de

visitors can never touch the objects, let alone

Reus in connection with this paradigm shift. Com-

use them. As a result, visitors ‘scan’ the displayed

pleted as a graduation project, it presents an au-

objects from a distance, something that needs

ditory environment to augment the artefact in an

reconsideration in our present time where the

unobtrusive and non-linear way. The objects that

‘experience’ is essential. To provide a way around

are central in this project are seven teacups and

this situation, the so-called ‘smart replica’ was

saucers that are part of the collection of Museum

proposed in the previous issue of Ar[t] (Roozen-

Boijmans Van Beuningen. These are currently on

burg, 2012): a new kind of reproduction, in the

display in a new exhibition on design and pre-in-

shape of a 3D print, that stretches the boundaries

dustrial design. Depicted in figure 1, each of these

of the replica’s concept as an autonomous object

teacup and saucer sets represents a milestone in

based on a historical artefact. New methods of ac-

the Dutch history of porcelain. Starting with the

cess and new digitization strategies based on the

first import of porcelain from China in the sev-

study of the relationship between the ‘bits’ and

enteenth century by the Dutch East Indies Com-

‘atoms’ are being developed. We hypothesize that

pany; followed by the invention of Delft’s blue as

by using these 3D imaging techniques the value of

an attempt to copy Chinese porcelain, and ending

our cultural heritage can be increased. In other

with the small scale production of porcelain in the

words, the goal is not to make the most realistic

Netherlands.

34

active enrichment but not entertainment?

went on to the next; leaving the exhibition behind, full of untold stories. Our hypothesis is that more

Museums and their collections house a specific

active means will lead to a more comprehensive

group of objects with their corresponding historical

museum experience; thereby increasing the op-

data. They draw audiences in through exhibitions

portunity to reflect and learn even after the visit.

and related facilities, including museum shops,

An important part of the collection are the non-

restaurants and so forth. The artefacts shown and

linear narratives, networks of information as-

experienced in an exhibition construct a powerful

sociated with the object, consisting of stories,

medium which instils visitors with thoughts they

locations, materials, rituals, the collection’s past

perhaps have not yet fostered. Knowledge and

and the like. Augmented matter — the mixture

culture become more accessible because of the

between bits and atoms — allows novel interac-

actual presence of the objects. In the vision of the

tion techniques to embody these networks of in-

curator, paying a visit to an exhibition enables the

formation. For this design project our aim was to

visitor to have an intimate encounter with the ar-

convey the following qualities of interaction: in-

tefacts on display (van Dongen, 2012). These types

trigue, understanding, satisfaction, and integrity.

of encounters could correspond with respectively

These qualities feed the resulting research ques-

a museum or a theme park.

tions: How can museums anticipate and facili-

Figure 1. THE SEVEN CuPS AND SAuCERS picture by HuGO BES

In the spectrum of recreational activities, the ex-

tate the active assemblage of old and new sto-

hibitions of Boijmans Van Beuningen can be char-

ries – and how do these stories refer back to the

acterised as enriching experiences: an encounter

replica’s original? How can digital databases be

in which the visitor is conscious of the artefact

employed in linking smart replicas to their collec-

and the (hi)story it represents. This means that a

tions? On a philosophical level: does the original

museum requires the visitor to reach out for in-

still attract interest? In the case of pre-industrial

formation, while the information passively waits

utensils such as the teacups and saucers this

for the visitor act upon it. The engagement of

question is very relevant.

visitors is limited as they are not experts on the particular subject of the exhibition. Because of responding information, it takes effort on the part

concept of tangible interaction with responsive audio

of the visitors to maintain concentration. In the

For the seven teacups and saucers that have been

project discussed here, a focus group that visited

chosen to represent the history of the Dutch

the museum exactly proved this problem case in

quest for porcelain, the ECR-model is used to

hand. –Visitors’ overall interaction with Boijmans’

classify the valuable information (Wong, 2012).

exhibitions can mainly be described as scanning.

This storytelling model categorises the informa-

The participants walked into a room, started at a

tion in three layers: engagement, context and

random showcase, looked at an object briefly, and

reference, as shown in Table 1.

the passive character of the objects and their cor-

35

ECR_Phase

Information

Interaction

Engagement:

a. Ambient soundscape (audio).

characteristics Spatialised audio,

Draw the visitors’

depending on user

attention; the moment

location / viewing

when a visitor has some

angle. Triggered by

immediate sensory,

proximity to object.

emotional or intellectual response to the artefact.

b. Narration 1, information on the role (audio).

Close proximity to

Context:

c. Identification data (text, visual).

Information plaque.

d. Narration 2 — information on specifics that can directly

Triggered by

object.

Draw the visitors’ attention; the moment when a visitor has some immediate sensory, emotional or intellectual response to the artefact.

Reference:

be related to the object (audio). e. Background information (text and images).

handling an object.

Exhibition app,

Gives the visitor the

available after the

opportunity to draw

exhibition.

conclusions and connect to related resources. It provides more detailed and interpretive information about the work.

Table 1. Proposed dialogue between visitor and exhibition.

In the first phase, visitors are attracted to par-

tor focuses on the features of the teacup or sau-

ticipate in the experience by other visitors who

cer (“audio specifics”). In this stage the visitor

are listening to the 3D audio clips. These audio

is encouraged to turn and explore the object,

clips give the impression that they are experi-

connecting the narrative with what is visualised

encing more: it is as if their auditory attention

about the object. The visitor can put down an

has doubled their visual attention (Erens, 2012).

object and pick up another object, while retain-

Together with an app that is made available, the

ing the soundscape.

visitor is intrigued and triggered to participate in

The app contains more background informa-

the exhibition.

tion on porcelain and the objects. When the

When the visitor is in the proximity of an object,

visitor returns home, she can consult the app

a 3D ambient soundscape that fits in with the his-

at her convenience; to browse the additional

tory of the specific teacup and saucer will appear

information for example.

— once the first soundscape has been heard, it attracts the visitor to go to other objects accompanied by a spatial soundscape. Arriving in close

Enabling spatial audio

proximity of the teacup and saucer, the spatial

As discussed in Table 1, every teacup and saucer

audio of a narrator starts. The narrator tells about

holds a series of three audio clips. The audio clips

the role the specific objects (“audio story”).

are played when the visitor shows interest through

When the visitor picks up a replica, the narra-

their actions; proximity to an object and the han-

36

dling of an object. Auditory feedback enriches the

Preliminary evaluation

visitors’ experience actively, without visual clut-

The core of the concept was tested during the

ter and conserves traditional values on how art

Object design fair (Rotterdam, 7-10 February

should be experienced. Furthermore, audio clips

2013). Information about the teacups and saucers

allow temporal cues; it works more associatively,

was presented via text or audio clips. Ambient

and speaks directly to the visitor (Erens, 2012). The

soundscapes were on or off, yielding four permu-

soundscape is the auditory equivalent of an ambi-

tations. 140 people interacted with a selection of

ent image; non-visual immersive content, typically

the configurations presented in pairs, they were

with so-called earcons which represent specific

asked to choose between the two displays and

objects or events. The technology to spatialise au-

support their reasoning.

dio was developed two decades ago; see Burgess and Verlinden (1993) for example. The character of

The essential observations include:

3D audio relates to the idea that the sounds seem

Participants seem to enjoy the ambient

to come from sound sources placed anywhere in

soundscape: it triggers the imagination and

a space (a surround sound effect). When listening

the recognition that the objects used to be

through headphones, the brain places the sources

utensils and not art object as they are now.

of the audio clip in your head. An example of a high quality 3D audio clip is “In your head” by Big

Presenting the information via responsive au-

Orange. We used AudioStage to produce our audio

dio facilitated the visitors to consciously turn

clips with a visual interface, cf. Figure 2.

the object to find the image that the narrator

To connect this aspect to the exhibition visitor,

referred to.

position and orientation tracking of the human head as well as the objects at hand is required.

Presenting the information via audio, whilst

Here optical or magnetic tracking principles also

the visitor is holding the object, does not

make sense, as they are fit for indoor use. With

clutter the visual sense.

the use of tangible replicas and 3D audio clips, scanning behaviour can be transformed into an

Participants appeared to be pleased with the

immersive encounter.

responsive auditory system, even those visitors who preferred text to audio clips.

Figure 2. SPATIAL AuDIO INTERFACE OF AuDIOSTAGE. 37

Conclusion Once the teacups and saucers were objects of daily-

will lead to a more meaningful experience. Secondly,

life and their form, weight, substance, texture, co-

understanding is nurtured because the encounter is

lour, decoration make sense primarily in the context

intuitive and the information presented by the arte-

of their functions and relations to other objects, as

fact responds to the visitor’s body language. Thirdly,

well as the people who used them. Combining ambi-

a satisfactorily quality is propelled, by the layered

ent soundscapes, tangible interaction with physical

structure of the narratives, which can be browsed in

replicas and the connection between information

a non-linear mode by the visitor. Through the app,

and corresponding visuals, triggers an immersive

the experience is saved, and can be connected to

encounter in which this sensibility is restored: the

various forms of social networking websites and

passive, one-sided encounter with the objects now

location-based services.

becomes an active two-sided encounter.

Lastly, the experience fits in with the integrity val-

The proposed system is by no means the first audi-

ues of the museum Boijmans Van Beuningen – the 3D

tory guide for exhibits; it rather extends the existing

scanned and printed, moulded physical replicas af-

strengths with emerging technologies such as indoor

ford what Dutch historian Johan Huizinga has called

tracking and spatialised soundscapes. Furthermore,

a “historical sensation”, the feeling as though you

it is more or less compatible with existing gear al-

are somehow “in touch” with the past (Ankersmit,

ready employed by many museums.

2005).

In essence this proposal presents a new type of rela-

Future work includes experimenting and research-

tionship between visitor and object that has interac-

ing the effect of the design in the environment of

tion qualities equal to a human conversation. Firstly,

a museum:

this poses an intriguing quality that pulls the visitor in by using 3D audio clips and ambient soundscapes, creating curiosity and making visitors want to engage with the artefacts. Because the visitor has a direct intellectual sensory dialogue with the object, this

1. Experimentation with augmenting untouchable artwork with ambient soundscapes.

2. Implementation of responsive audio tours in the current exhibition context.

Figure 3. Impression of the preliminary evaluation.

38

3.

Prototyping indoor tracking and interaction sensing possibilities, with special attention to

More information ¢¢

smartphone infrastructure.

phones required): http://www.youtube.com/

4. Creating guidelines on how the concept could be implemented to suit different kinds of ob-

watch?v=enR1Ggbuf_8 ¢¢

jects in the museum (or maybe even utensils in

fr/web/en/prods/audiostage ¢¢

com/2008/05/new-work-detroit-institute-of-1 ¢¢

by Cilia Erens, Professor Joris Dik, Dr. Wolf Song and

In your head; a high quality 3D audio example: http://www.big-orange.nl

constructive collaboration. Furthermore, we would like to acknowledge the valuable advice and support

Interactive installations regarding preindustrial utensils: http://new.pentagram.

We would like to thank Alexandra van Dongen, curator at the Museum Boijmans Van Beuningen for her

Software to render binaural output by visually placing audioclips in 3D: http://www.longcat.

our everyday context).

Acknowledgements

Video presentation of this project (head-

¢¢

Official Smart Replica blog: http://smartreplicas.blogspot.nl

Ir. Wim Verwaal. Many thanks to DOEN foundation for funding the Smart Replicas project (including the graduation project of Lotte de Reus) and many thanks to Mareco prototyping for their contribution in the form of 3D micro prints.

References ¢¢

Ankersmit, F.R. (2005). Sublime historical experience, Stanford University Press.

¢¢

Burgess, D. A., Verlinden, J.C. (1993). An architecture for spatial audio servers. Proceedings of Virtual Reality Systems Conference (Fall ‘93).

¢¢

Erens, C. (2012). Interview on soundscapes, Interviewed by Lotte de Reus. Amsterdam, 14 December.

¢¢

Halbertsma, M. (1995). Themaparken, dierentuinen en musea. Rotterdam: Erasmus Centrum voor Kunst- en Cultuurwetenschappen.

¢¢

Ministerie van Onderwijs, Cultuur en Wetenschap (2005). Bewaren om teweeg te brengen. Museale strategie, Woerden: Drukkerij Zuidam.

¢¢

The Futures Channel, (1999). Conversation with Curtis Wong. [online] Available at: http:// www.thefutureschannel.com/conversations_ archive/wong_conversation.php

¢¢

Van Dongen, A., (2012). Interview on the mu-

Lotte de Reus Lotte de Reus recently received her Master’s degree in Design for Interaction at the Faculty of Industrial Design Engineering at Delft University of Technology. In July 2012 she started her graduation project for the Smart Replicas project; she was driven by a fascination for porcelain and the wish to create effective storytelling experiences. During the project she was pleasantly surprised by the world of audio, something she had not yet encountered in her studies. New technologies such as augmented audio, immersive soundscapes and 3D audio now hold a new, special interest for her. In the future, Lotte would again like to work in the domains that combine art and technology. Her portfolio can be found at www.lottedereus.nl.

seum context, Interviewed by Lotte de Reus.

Rotterdam, 21 November 2012.

39

40 40

Image: Revealing Identity, Guido Lagerweij

Who owns the space 2 discussing legal issues of AR Yolande Kolstee

WHAT IS PRESENT IN AR SPACE? As mentioned in ‘Who owns the space’ in AR[t] 2, we keep a close look on the developments concerning the invasion of our physical space with virtual information, and discuss some legal aspects thereof, using information from, amongst other sources, the blog Wassom.com.

In AR[t] 2, I contended that we as passers-by might not be aware of any virtual content at all and, when we don’t have the right app (software) and/or mobile (hardware), we don’t have the means with which to recognize the virtual content floating in the air around us. Although, in theory, everybody has the same access to virtual content, de facto, this is not the case.

For some this might be a bit frustrating, giving off the impression as if people are gossiping about them in a different language or at a very low tone; unsure whether or not they are they are talking about you…

Sander Veenhof, Dutch AR artist, made a lot of people aware that AR could be everywhere, and actually is everywhere and that this is a special characteristic of AR for as a new medium for artists- and artistic purposes.

For many people, seeing his art work Biggar and playing with it for the first time, helps them to realize that this AR art piece encapsulates the entire earth with its 7.463.185.678 virtual cubes whose colours can be changed by simply tapping on a mobile phone. The only thing you need to do

41

to actually see Biggar, is to download this app via

What is at stake is, what happens when this in-

Layar. This virtual sculpture has true omnipres-

formation about our eye-movements is stored in

ence; it floats above holy places in Rome or in

databases, analysed and even sold to marketing

the Himalayas, as well as above industrial places,

companies.

above warzones and above peaceful places…

Brian Wassom, blogger and specialist in AR Law, predicted five issues related to AR Law for 2012,

What is recorded ?

ranging from public resistance to ‘adult’ AR (AR

With the announced introduction of Google glass,

porn that is) via licensing, and negligence, to AR

we see a concern regarding the possibility to re-

patents.

cord without the public knowing this.

“Stop the Cyborgs!”

On March 11,2013, Seattle’s 5 Point Cafe became the first

known

establishment to

publicly

ban Google Glass, the highly anticipated augment-

Mr Wassom recognized he was not right in all his

ed reality device set to be released later this year.

predictions, scoring 1,5 out of 5, but for him it was

The “No Glass” logo that the café published on its

more a matter of time than a matter of substan-

website http://the5pointcafe.com was developed

tial incorrect prediction. However, in February

and released (under a Creative Commons license)

2013 an adult magazine has distributed around

by a new London-based group called “Stop the Cy-

1,000 billboards with Augmented Reality markers

borgs.” The group is composed of three young Lon-

throughout many cities in the Netherlands. When

doners who decided to make a public case against

downloaded, the app distributed via Layar works

Google Glass and other similar devices.

by pointing your smartphone at one of these post-

On the Stop the Cyborgs site, the group raises a

ers, and the girls will come to life allowing you to

significant concern: namely, that there’s no obvi-

watch them in a video right there on the street.

ous way to know when the device is on or what

This adult content will certainly appeal to a lot

it’s actually doing (recording or not).

of people and this will definitely result in another large group of people coming into contact with AR. Since most of the public are not familiar with

TWO WAY TRAFFIC

AR marketing activities like this, it will have a

For 2013, again, some predictions were made

significant informative impact.

and trends were spotted. Concerns raised range from on the one hand, inserting a lot of informa-

EYE-TRACKING

tion from the physical world, maybe voluntarily via players of AR games, into the virtual world,

Another type of warning from Mr Wassom is relat-

and, on the other hand, entering information

ed to the information derived from tracking our

about the physical world (‘burglary information’

eye-movements to detect were we are looking at.

as we called this in the last issue of AR[t]) into

Real immersive AR depends on knowing exactly

the virtual world, without any prior screening or

what our eyes are looking at; this information is

approval from an examination committee.

used to position the virtual content there where our eyes are looking at. Natural feature tracking systems are in rapid development; we can enter

Prevalence of AR

a room, let our camera survey the space, detect

John Moe (the host of Marketplace Tech Report and

the salient points, and make a virtual grid on

handles web content for the program), wrote in

which we position our virtual objects or scenes.

2011 that “Augmented Reality has been the Next

Using the eye-movements of the person wearing

Big Thing for a while now, although it never man-

AR glasses gives extra accuracy.

ages to become the Actual Current Big Thing”.

42 42

Images courtesy of GoldRun, see www.goldrunner2013.com and www.snapsapp.com

This quote gives us a direction for discussing the

We might come to the preliminary insight that

abovementioned AR related legal issues. The

as for now, our actual privacy laws are sufficient

intertwinement of AR in our day-to-day lives is

enough to deal with privacy concerns in aug-

actually quite slow — especially compared to the

mented reality. An incorrect accusation or incor-

revolutionary predictions. We are getting used to

rect information remains incorrect, regardless

it at a nice pace. As an example, I can take our AR

whether it is posted on Twitter, Facebook or in a

Lab, which has been working in the augmented

space around us, via an AR app.

reality field since 2006, and still for some, even

To add correct or incorrect virtual information to

closely related people within our Academy, the

a virtual object or a space, is as simple as assign-

concept of AR is completely new.

ing correct or incorrect information to any sub-

“But if the Google glass will become prevalent”

ject, object or space in our physical world, and

as one the people from ‘Stop the Cyborg’ argues

will be considered as such.

in a conversation with a journalist of arstechnica.

For the time being, we don’t need special AR

com “and suddenly everyone is wearing it and

criminal legislation, our current laws might be

this becomes as prevalent as smartphones—you

sufficient.

can see it becomes very intrusive very quickly.

It’s not about the tech, it’s about the social culture around it”.

Further Reading Critical blog on law and social media:

FALSE STAYS FALSE Recording without the recorded person know-

http://wassom.com Critical site on wearable technology: http://stopthecyborgs.org

ing this, is possible with our smartphones and is widely spread among youtube videos. However, we have privacy laws to handle this- and if those laws are not adequate enough, we have to change those laws.

43 43

44

OLIVER PERCIVALL, The Augmented Star, Book Cover, Illustrator: Paul Hadcock

The Misadventures in Augmented Reality OLIVER PERCIVALL

I FIRST LEARNED ABOuT AuGMENTED REALITY AROuND TWO YEARS AGO. I DISCOVERED ITS MAIN uSES WERE AS A NAVIGATION TOOL uSED IN CITY CENTERS, POINTING PEOPLE FOR EXAMPLE TO PLACES OF INTEREST OR NEAREST TuBE STATIONS. THE THOuGHT OF CONCEALED TEXTuAL MARKERS STRuCK A STORY IDEA IN MY MIND. WHAT IF THERE WAS SOME AuGMENTED REALITY TEXT MOST OF uS WERE NOT SuPPOSED TO SEE AND YOu HAD A DEVICE THAT WAS ABLE TO READ IT… WOuLD YOu FOLLOW ITS INSTRuCTIONS? THIS IDEA FASCINATED ME INTO WRITING A SERIES OF STORIES TITLED THE MISADVENTuRES IN AuGMENTED REALITY. PART ONE FOLLOWS OuR ACCIDENTAL EXPLORER LYNDEN AS HE POSTS HIS ADVENTuRES TO AN ONLINE FORuM. 45

Part One My uncle was one clever soul. He could probably be best described as a recluse, the shy inventor type. But nearly nine years ago now he disappeared from his hometown and has never been seen since. I don’t think he’s lost, not wanting to be found is my guess and I think I know why.

Using his ingenious engineering skills he somehow created a VAARR (Very Advanced Augmented Reality Reader) It works just the same as other regular Augmented Reality apps I am led to believe but with one unique difference… It has a setting that once activated, shows unseen Augmented Reality text none of which is detected by any other device. He created two actually. One was entrusted to me. And because of this I have a story to tell.

A few years ago I wasn’t exactly a technology fanatic although admittedly I had more than a passing interest in new gadgets. But that was nearly three years ago now before I set foot within the Augmented Star. Now there is probably nothing I don’t know about

46

Images courtesy of GoldRun, see www.goldrunner2013.com and www.snapsapp.com

the integrated software algorithms used in Augmented Reality.

Life in this newly discovered domain is a far cry from home.

There’s no one thing that makes it uniquely different. It’s everything. The smell, the temperature, the noise levels, the beings, everything is entirely unfamiliar when you first arrive. But once you start exploring all the hidden signs and directions it’s impossible to stop. You’re immediately plunged into its rich diverse landscapes and to find your way around or locate anything at all you must be in possession of the special device — The VAARR or Very Advanced Augmented Reality Reader. But be warned: Augmented Reality text as we know it was not developed within The Augmented Star so it makes it pretty hard to find your way out.

It’s not easy to write long paragraphs on this VAARR. My uncle made it possible to post directly online from this device. So my blog entries below are concise.

The Ultimate Frequency Forum This forum was created for anyone to post to that feels lost, those that do not wish to be found or find themselves beyond the confines of earthly boundaries.

Blog Entry #1

Lynden

Apr 12 2011

‘It’s not about gazing into the future, it’s about seeing the present through a unique lens.’

That’s the first message you see when you arrive in The Augmented Star. Every time. And you never arrive at the same place twice. I’ve come here with a strict agenda. To find my uncle.

Blog Entry #2

Lynden

Jul 12 2011

OK it took me a couple of hours to get acquainted here this time. There’s just so much to get distracted by. From learning a new food recipe to breathtaking advertising campaigns, the Augmented Reality here really is something else. On the downside crossing the street can be something of a perilous activity. It puts things in perspective when vehicles move from 0 to 600 MPH in less time than it takes to chalk a pool cue.

Blog Entry #3

Lynden

Oct 12 2011

Just noticed a glitch with this online forum that I would like to point out to the moderators. The first post I made earlier was date stamped Apr 12 2011 but the post I made just a couple of hours after that is stamped July 12 2011. Three months later? I think not!

47

Blog Entry #4

Lynden

Jan 12 2012

It’s funny because since I’ve been wandering the streets here the last few hours I’ve regretted never going to Cambridge University. Although I feel like I have been. It appears there is nothing you can’t learn here by waving the VAARR around. The whole place is an encyclopedia of knowledge.

They promote learning a Science degree in Artificial Intelligence can be completed in three months here. Knowledge is fluidly brought to astonishing life everywhere by Augmented Reality.

I just learned that a thimbleful of a neutron star would weigh over 100 million tons. Must be something in the air here as my clothes are becoming dirty and ragged really quickly. I feel I’m ready for that science degree now!

o. l. Percivall i have always had an interest in technology and gadgets. after leaving school i took a basic computer studies diploma, which eventually led to a career in various i.t support roles leading up to Project Management. the mysterious side of technology especially intrigued me including the possibilities of where it could take us in ‘what if’ scenarios. so equipped with a reasonable understanding of augmented reality and enjoying a challenge, i set myself the complex task of plotting my science fiction novel and created an alternative fictional world that became the augmented star. the augmented star is now available on the http://tiny.cc/augmented-star

amazon kindle store, jsut follow the Qr code on the left!

48

Blog Entry #5

Lynden

Apr 12 2012

The risk of being struck by a falling meteorite for a human is one occurrence every 9300 years.

Wow that’s a curious fact. I wonder what the odds are for walking into an Augmented Universe like this one.

I saw an advert on a billboard for an automobile just now. Using my VAARR triggered a full Augmented Reality breakdown of its features from performance figures to finance options and then it invited you to take a virtual test drive in a car simulation game. The AR campaigns here really resonate with customers in a way that most other ad platforms fall miserably short.

Lynden

Jul 12 2012

Turns out my VAARR is a pretty valuable commodity here. It can read over 150 types of Augmented Reality text and other types too that alternative devices can’t. After an exhausting and bloody battle earlier today, I have learned some creatures here would even kill to own one.

Blog Entry #7

Lynden

Oct 12 2012

Just received some news that was pretty hard to comprehend. I’ve met a lot of people here within The Augmented Star and most of them know my uncle but apparently he’s on the run from some bad people.

Someone is coming. I’ve gotta move quickly. It feels like I’m on the run myself.

I’ve come to realize now that I may never be able to leave the Augmented Star.

End of part one

Images COuRTESY OF GOLDRuN, SEE WWW.GOLDRuNNER2013.COM AND WWW.SNAPSAPP.COM

Blog Entry #6

49

Augmented modelling in the design studio:

augmented pedagogies Alejandro Veliz Reyes University of Salford

Digital physicality / Physical digitality From very early stages of the design process

Digital physicality, Physical digitality. One of the

(sketches and diagrams) up to detailing and

main grips in which this topic is grounded is that

construction, models are built by using several

the growing academic concern on the dialogs be-

different — yet interrelated techniques. The

tween physical and digital realms in architecture

construction of a physical model by computer-

is becoming blurry, since the distinction between

controlled machinery requires, unavoidably, a

both cannot be fully depicted in a world where

digital counterpart. Likewise, scanning and re-

digital information is continuously embedded

verse engineering methods allow a fluent interac-

into ‘real world’ situations, such as ubiquitous

tion between models which embeds information

computing systems or environmentally responsive

and knowledge as the design process proceeds. As

technologies. This recent conference had, so far,

a result, a complex orchestration of users, devel-

the biggest amount of published articles in the

opers, tools and techniques merge into tailored

history of the eCAADe Association Conference,

modelling flow works according to each project’s

and further evidence of this trend is the grow-

requirements. This design-technology synchronic

ing participation of architects and designers in

co-evolution has been historically aligned with

professional events such as Ars Electronica, SIG-

the zeitgeist of the architectural practice, in

GRAPH or the ISMAR community.

which new technological developments shape the way we express ourselves, and define new design

This work claims that beyond physical or digital

forms and organizational/social schemes.

modelling techniques, augmented reality now

...augmented reality now gives us the chance to build hybrid, augmented models.

gives us the chance to build hybrid, augmented models. Augmented models are those which blend a physical and a digital counterpart in a resulting synchronic manner, hence taking advantage of the benefits of physical modelling methods (dexterity skills, manipulability) as well as digital (accuracy,

Indeed, this section is titled with the topic of the

modifiability, among others). Also, in addition to

latest conference of the Association of Education

the technical definition of augmented reality, the

and Research on Computer Aided Architectural

concept of augmentation has a major role in this

Design in Europe (eCAADe) held at the Czech

work. Even if the semantics of augmentation sug-

Technical University in Prague (September 2012):

gests an ‘increment on size or amount’, it also

50

traiNiNg sessioN oN augMeNted reality autHoriNg aNd augMeNted ModelliNg, scHool of tHe built eNviroNMeNt, uNiversity of salford.

suggests some enhancement. As a result, this ap-

diagrams and sketches. This scenario might not be

proach to the augmentation of reality fits with

unknown for any architecture student, since the

the major aim of educational research which is to

studio teaching scheme has been largely acknowl-

enhance and improve educational processes and

edged as the core practice-based module in which

methods, thus naming this work as ‘Augmented

both design (composition, planning, representing)

Pedagogies’.

and high-order cognitive skills (critical thinking, analysis, synthesis) are mainly developed in archi-

MODELS IN THE DESIGN STuDIO

tectural education programs around the world. Actually, the architecture studio and its interactions as a subject matter is a quite complex challenge.

“It looks better like this”, says an architectural

As stated by Allen Cunnigham in 2005, after a cen-

design instructor while chopping a piece of card-

tennial adaptation and evolution the studio teach-

board off a building model. In the same design crit

ing scheme and “project-based education around

session, students, instructors and observers engage

architecture employing the studio system is the

in a dialog based on the use of plans and sections,

most advanced method of teaching complex prob-

rendered views of a project, physical mock-ups,

lem solving that exists”.

51

The usefulness of models within the design stu-

from a scientific research perspective, that is fol-

dio is clear. Beyond the fact that its construction

lowing the guiding principles of generalizability,

itself entails the development of technical skills,

communicability and transferability of that re-

models also embed design information and knowl-

sulting knowledge.

edge, affecting organizational dynamics (the design critique or peer to peer collaborative work),

The deep impact of new digital tools in design

the creation of students’ ‘toolkits’ or the final

pedagogy has been explored recently by design

presentation of the design solutions, among other

theorists, such as Dr Rivka Oxman. The particu-

benefits. During this research being conducted

larity of the design studio as a research setting is

at the university of Salford (uK), the extent of

spiced by theoretical underpins that can poten-

the impact of augmented models in this complex

tially lead the path to depict this impact. For ex-

studio-system is yet to be depicted. Augmented

ample, it has been stated that the studio teach-

models will be used, therefore, as a way to un-

ing is usually an unstructured process, in which

derstand how new technologies impact design

perceptions and interpretations of information

education and how can we describe that impact

and models play a major role in the students’

draft versioN of aN augMeNted Model for iNterior desigN. furNiture caN be re-arraNged iN a PHysical Model of a House rooM (studeNts’ work).

52

progression in the courses, mostly based on ‘design dialogs’ between students, and students and instructors. Also, digital tools have the potential to not only re-shape the toolkits being used for design, but also mediate in the way design methods are structured, offers new ‘digital materials’ to work with or changes the very nature of the design problems to be faced in different courses.

It is not clear, however, how this impacts occurs.

The interactions within the studio that make use of representations and models to design are well established ‘rituals’ such as peer-to-peer collaborative activities or the design critique, but the nature of each studio differs from each other.

Variables such as the experience of the instructors, the background of the students, the nature of the design problems to be faced or the institutional standpoint turn the studio into highly context-dependant modules. As a result those variables are usually highly controlled and the study of the impact of different technologies is commonly constrained to the description of technical challenges to be solved, the development of new systems/software or metrics of student satisfaction, rather than on the provision of a theoretical account of their impact into this complex teaching/learning process. The lack of a theory that describes how technology re-shapes the studio results in very limited knowledge re-usability and in turn, into very caged and localized pedagogical frameworks that do not allow cross-institutional or cross-disciplinary collaboration, to evaluate the constant infusion and evolution of new digital tools for educational purposes or to re-use a pedagogical approach and its associated knowledge.

A THEORETICAL APPROACH TOWARDS AuGMENTED PEDAGOGIES

IMAGES OF THE TRAINING SESSIONS ON AuGMENTED REALITY AuTHORING AND AuGMENTED MODELLING. STuDENTS OF THE PROGRAMS MSC DIGITAL ARCHITECTuRAL DESIGN AND MSC BuILDING INFORMATION MODELLING AND INTE-

Indeed, there is not fixed methodology to study

GRATED DESIGN, SCHOOL OF THE BuILT ENVIRONMENT,

the impact of digital technologies in design

uNIVERSITY OF SALFORD.

education. Rather, each study requires an own

53

standpoint in terms of validity and fitness to

scribe the resulting studio dynamics by using aug-

the research problem and the subject matter.

mented models. Several trainings on augmented

As Wanda Orlikowski and Suzanne Iacono (2001)

reality and augmented modelling have been

state on their work on information systems the-

made at the university of Salford (MSc Digital

ory research, this corresponds to the fact that

Architectural Design, MSc in Building Information

the use of technologies depend on the context

Modelling and Integrated Design), and two more

and hence, “there is no single, one-size-fits-all

experimental settings are now being arranged in

conceptualization of technology that will work

different European countries. These multiple set-

for all studies. As a result, IS researchers need

tings are not only intended to provide a wide view

to develop the theoretical apparatus that is ap-

of the subject being studied, but also fits with the

propriate for their particular types of investiga-

current recommendations for theory construc-

tions, given their questions, focus, methodology,

tion methodologies, since the manipulation and

and units of analysis.”

observation of data in many divergent ways and the juxtaposition of different conflicting realities

In order to overcome this challenge, this ongo-

and sources counteracts the tendency of reach-

ing research proposes a theoretical approach to

ing false or incomplete results, or information-

depict the impact of augmented models in design

processing biases of the investigator.

education. By following a grounded theory methodology, observations and recordings are being

This work is expected to be finished by end-2014.

collected in diverse settings on an attempt to de-

VIEW OF THE STuDENTS’ TOOLKIT DuRING THE FIRST AR WORKSHOP AT THE uNIVERSITY OF SALFORD.

54

Related literature and sources ¢¢

Achten, H., Pavlicek, J., Hulin, J., Matejdan, D. (Eds.). (2012). Digital Physicality / Physical Digitality - Proceedings of the 30th eCAADe Conference - Volume 1 / 2. Czech Technical University in Prague, Faculty of Architecture (Czech Republic) 12-14 September 2012.

¢¢

Buechner, J. (2011). Fictional Entities

Alejandro Veliz Reyes

and Augmented Reality: A Metaphysical Impossibility Result. Journal of Evolution &

I finished my Architecture degree in 2007,

Technology, 22(Minds and Machines Special

in Chile. Today I am a teaching assistant and

Issue), 53-72.

PhD student on digital architectural design at the University of Salford (United Kingdom)

¢¢

¢¢

Cunningham, A. (2005). Notes on educa-

and my current research interests are design

tion and research around architecture. The

pedagogy, augmented reality, and collabora-

Journal of Architecture, 10(4), 415-441.

tive technologies for design.

Eisenhardt, K. M. (1989). Building theories

I have been involved in research activities

from case study research. Academy of man-

since 2007, but my background is quite mixed

agement review, 14(4), 532-550.

and has involved the domains of sustainable design, buildings in extreme/cold environ-

¢¢

¢¢

Orlikowski, W. J., & Iacono, C. S. (2001).

ments, mass customization and digital de-

Research commentary: Desperately seeking

sign. My published work has led me to pres-

the "IT" in IT research—a call to theoriz-

ent on conferences and invited lectures in

ing the it artifact. Information systems

Chile, United Kingdom, Italy, Belgium, Slove-

research, 12(2), 121-134.

nia, Czech Republic, Canada, among others.

Oxman, R. (2008). Digital architecture as a challenge for design pedagogy: theory, knowledge, models and medium. Design Studies, 29(2), 99-120.

55

56

SCARLET Project Š University of Manchester

a study iN scarlet MATT RAMIREZ

MIMAS - uNIVERSITY OF MANCHESTER

laura skilton

jean vacher

Learning and Teaching Co-ordinator

Curator, Crafts Study Centre

Mimas, The university of Manchester

university for the Creative Arts

rose lock

Marie-therese gramstadt

Special Collections Supervisor

Educational Technologist, Crafts Study Centre/

university of Sussex

Research Associate, Visual Arts Data Service university for the Creative Arts

introduction Augmented Reality (AR) was identified in the 2011

established learning methods and styles. If the

Horizon Report1 as a key technology trend with po-

focus for the student is the technology, the learn-

tential impact on education. The report provided

ing experience can be diluted, this can inevitably

the catalyst for the SCARLET (Special Collections

lead to dissatisfaction and resistance.

using Augmented Reality to Enhance Learning and Teaching) project, as a way of leveraging innovative technology with pedagogical processes.

Whenever the words â&#x20AC;&#x153;technical innovationâ&#x20AC;? is spoken in education circles, educators are under-

The benefits of using augmented reality in education

standably cautious electing to concentrate, perhaps rightly, on deep rooted pedagogical benefits

The benefits to student learning should always be

rather than short-term gimmickry. Many observ-

central to the introduction of any new technology

ers have already buried AR as a fleeting fad in

and AR is no different. It is always useful when

education due to its lack of use cases and docu-

dealing with new methods of delivery, to be armed

mented impact studies.

with a long list of tangible gains for adopting the technology. Some of the most persuasive argu-

After all, in education technology should be

ments are described below:

transparent and not an overpowering driver, particularly where the emphasis is on the teach-

Students can view and touch real manuscripts/

ing material. In addition, users do not want to

editions in conjunction with guided support

spend time adapting to a new way of learning,

from trusted sources, supporting independent

new technology should integrate seamlessly into

learning.

57

Layering AR on texts/images can encourage

rare books within the controlled conditions of

interaction (e.g. augmented 3D models that

reading rooms, isolated from much of the second-

overlay the physical image and require user

ary supporting materials and a growing mass of

touch gestures to proceed) and spark enthusi-

related digital assets. Students are used to having

asm, preparing them for solo research.

access to electronic information on demand, so this experience can be foreign and a barrier to

AR promotes ‘active’ teaching, maximizing

their use of special collections.

the opportunity for interaction, encouraging critical response and the adoption of new per-

The SCARLET project, while embracing the po-

spectives and positions. This is in opposition to

tential of AR, concentrated on delivering the

traditional didactic methods that are predomi-

benefits to student learning without being a flag

nantly teacher led.

bearer for the technology. SCARLET was led by the Learning and Teaching team at Mimas2, a na-

Users retain a very small amount of the infor-

tional centre of expertise at The University of

mation that is delivered, and a slightly larger

Manchester. A mixed team was pulled together

percentage of what is shown to them, but when

dedicated to enhancing the student experience

we become actively involved in an experience,

through the application of technology including

learners will remember and retain the majority

librarians, academics, learning technologists and

of the information presented to them.

students.

AR can harness both asynchronous (emailing

Sources for primary content were ten key edi-

tutor questions) and synchronous (discussion

tions of ‘The Divine Comedy’ by Dante (between

with peers) e-learning methods.

1472 and 1555) particularly important in terms of publishing and intellectual history and the

Abstract concepts or ideas that might otherwise be difficult for students to comprehend

world-renowned oldest fragment of the Gospel of John.

can be presented through an enhanced learning environment offering access to source his-

At the start of the project in 2011, Junaio was

torical artefacts and online research in situ.

the only AR browser to harness optical tracking functionality, linking 3D models, videos and in-

The learning curve for new users engaging

formation to images in the form of “GLUE” based

with mobile AR through browsers is relatively

channels. This coupled with an open API and

shallow enabling the learning/pedagogy to be

compatibility on Android, iOS and Nokia devices

the driver, not the technology.

would prove decisive in the decision to use Junaio. By implementing an object based AR experience, students could simultaneously experience

SCARLET (June 2011- May 2012)

the magic of original primary materials, such as an early printed book in the library, whilst enhancing the learning experience by surrounding the book with digitised content; 3D models, im-

The main challenge with the use of special col-

ages, translations, and information on related

lections in learning and teaching is that students

objects.

are required to consult archives, manuscripts and

58

evaluation A dominant theme that became evident in the evaluation was that the two academics found differing responses dependent on student user groups. Students who had little prior subject knowledge found the app most useful, providing a foundation for further investigation and research. The learning experience was most enhanced by AR when information delivered was FRAGMENT OF THE GOSPEL OF JOHN AR CONTENT

contextual and less generic. Simply adding exist-

© MIMAS, uNIVERSITY OF MANCHESTER.

ing web assets to an object is insufficient; making them unique and packaged in digestible chunks produces more positive feedback and value.

In addition, student feedback noted that AR experiences should be best used as part of a learning activity (either independent or group based) acting as enabler to achieving a key course objective (e.g. planning for essay or presentation).

AR was most successful layered over the printed marker instead of signposting to other web based resources already accessible using traditional teaching scaffolds (e.g. CMS). Throughout the project lifespan, student evaluation was critical,

DANTE AR CONTENT

directly feeding into the application using an it-

© MIMAS, uNIVERSITY OF MANCHESTER.

erative development approach.

The team went on to win the 2nd prize for the

scarlet+ (june 2012 – april 2013)

ALT learning technologist of the year team award3 and won the Innovation in HE award at Education Innovation 20134. Emphasising the need to align technology to teaching and learn-

Following the success of the SCARLET project,

ing objectives was paramount from the outset

further funding was made available to work with

to maximise student benefit and impact. Further

two other uK institutions to facilitate and guide

funding was made available through the SCAR-

them in the development of AR content. It would

LET+ project whose primary focus was to apply

also liaise with staff at these universities to

the process and framework to other institutions

transfer the skills learnt and hone the toolkit that

special collections embedding the methodology

was used as a high level framework for replicating

using a toolkit5 (university of Sussex and the

similar AR practices in education.

Craft Study Centre at the university for the Creative Arts).

59

ar visualisatioN of ukMaP data Š MiMas, uNiversity of MaNcHester.

The partner universities involved were the Spe-

which will allow students to access surrounding

cial Collections at the university of Sussex and

resources, layering anatomical information and

the Crafts Study Centre at the university for the

reinforcing learning with instructional demon-

Creative Arts7. focusing on content from 1980sâ&#x20AC;&#x2122;

strations (e.g. cannula application).

6

mass observation and 20th century crafts.

In the area of geo-spatial mapping, Mimas has This project has both developed Mimasâ&#x20AC;&#x2122; under-

collaborated with colleagues from the Landmap9

standing of implementing AR in education and

service to create an AR experience around the

embedding best practice and methodology to

uKMap dataset. This provides a wealth of rich,

other institutions. Crucial to the success was en-

multi-layered information accurately locating

suring that, as with SCARLET, a multi-disciplinary

building types, building heights and ground usage

team approach was adopted. This ensured that

to name a few. The challenge was to incorpo-

the content developed made an impact on learn-

rate this tabular data in a visual 3D model that a

ing and teaching as well as enabling AR skills to

handheld device could render efficiently. It dem-

spread across the institutions.

onstrates a visual representation of raw materials that are often extremely large in size and difficult to comprehend.

working with other subject areas In parallel with SCARLET and SCARLET+, Mimas

conclusion

have worked with the Manchester Medical School8

To conclude, the two projects have presented im-

helping students to understand pattern recogni-

portant findings in the impact of AR in education,

tion, a crucial part in decision making processes.

delivering a suite of rich materials especially

There is vast potential in this field enabling stu-

given the small amount of funding that was avail-

dents to examine x-rays/biopsies, using AR to

able. Both succeeded in providing a showcase for

identify abnormalities. In conjunction with this,

the Special Collections held at their respective

early development has started using 3D object

institutions using AR, bringing their static objects

recognition in the clinical skills laboratories,

to life. Sharing was a key element at the heart

60

of SCARLET and SCARLET+, this coupled with a strong team ethic enabling stakeholders to buy into the long-term vision. It is hoped that the legacy from these small projects will be to inspire others to undertake similar work and display the student led benefits AR can offer.

AR opens up huge possibilities for creating immersive learning activities. It is particularly effective in explaining abstract concepts visually; allowing active based learners to better absorb the transfer of knowledge. While it may not be suitable for all students and situations, when em-

Matt ramirez

ployed well it can capture the attention like few

Matt ramirez is currently working on the

other technological mediums.

technical development and support of the jisc funded scarlet+ augmented

further information:

reality project, this follows on from

http://teamscarlet.wordpress.com

his involvement in the award winning

references 1.

scarlet project. He has over 15 years experience in web design and e-learning content development for a variety of

2011 HORIZON report, section on

subject areas including medicine, it,

augmented reality: http://wp.nmc.org/

science, special collections

horizon2011/sections/augmentedreality

and business. these have used multi-

2.

Mimas: http://mimas.ac.uk

media content authoring tools such as

3.

SCARLET team are joint second in

Flex, Flash, Blender, Unity to name a

learning and teaching awards, 13th

few. Matt's role is also concerned with

September 2012: http://teamscarlet.

the research and development of new

wordpress.com/2012/09/13/scarlet-

technologies (e.g. ibook/mobile/multi-

team-are-joint-runner-up-in-learning-tech-

media development projects and haptics)

nology-awards

with the Manchester Medical and dental

Education Innovation 2013:

schools. this aims to improve the stu-

http://educationinnovation.co.uk/cms

dent experience by embracing innova-

SCARLET Toolkit: http://scarlet.mimas.

tive learning methods and providing cut-

ac.uk/mediawiki/index.php/Main_Page

ting edge support materials. read more

Special Collections at the university of

about Mattâ&#x20AC;&#x2122;s work at http://teamscarlet.

Sussex: http://www.sussex.ac.uk/library/

wordpress.com and @team_scarlet.

4.

5.

6.

specialcollections 7.

Crafts Study Centre at the university for the Creative Arts: http://www.csc.ucreative.ac.uk

8.

Manchester Medical School: http://www.mms.manchester.ac.uk

9.

Landmap: http://landmap.ac.uk 61

iNtroduciNg ar to first year graPHic desigN studeNts YOLANDE KOLSTEE

graphic design students learn how to work

we, as ar lab are thoroughly convinced that it

with the latest adobe suit, Photoshop and web-

is important to let first-year students, freshmen

site development tools. Most important, how-

and -women, experience augmented reality,

ever, are the courses in which concept develop-

because it will change aspects of their discipline

ment forms the main part. whether a concept is

as well as their ‘metier’.

devised with colour pencils, felt pens or a col-

at various other art academies we see similar

lage of other material is not the most important

projects, one of which is the scarlet (special

issue at that moment. what is important for the

collections using augmented reality to enhance

academy, and the department involved, is that

learning and teaching) project, in the uk, at

they want the students to become acquainted

the university for the creative arts.

with creative or artistic research. However, the

at the beginning of the royal academy of arts’

majority of mainstream teachers place analogue

current study year, we made a head start with

and digital techniques at the same level when

the introduction of ar via projects differing in

developing concepts; while, in effect, they of-

length and with different student groups.

ten have a (underlying) preference for pen and ink based handwork.

the idea was to work towards a special PopUp Gallery, meaning a short, one-day exhibition

when working with augmented reality — a

of the results, in cooperation with the studium

very suitable medium for graphic designers, by

generale programme, a programme covering

the way — new elements should be considered

lectures throughout the year.

in the concept and design phase. with augment-

two graphic design teachers, both young

ed reality new, very new, aspects are added to

and digitally savvy, started to work with their

the world of graphics regarding the internet and

classes of 27 students each (!) together with

social media, while also taking co-authorship

some third year students as well. the search to

and performance into account.

get to know what ar is and how it could be a

62

meaningful technique for graphic designers be-

to work as a fully functioning app. We showed

gan comprised two steps. The first was a very

them our AR projects and introduced them to

active YouTube-Google search, and the second

e.g. Aurasma — a simple and elegant tool for be-

step was to learn to play with the idea of adding

ginners in the AR app world.

extra information — both analogue and/or digi-

We were lucky to have two lecturers who gave

tal — a concept development trajectory of sorts,

their talk on AR as part of the Studium Generale

so to say. The end-results would be displayed

programme preceding the Pop-Up Gallery: AR

during our first Pop-Up Gallery on AR.

artist Sander Veenhof and data-visualisation

In the words of the students’ teachers : “Spec-

expert Niels Schrader. They were part of the

tacles, headsets, phones and tablets. These are

jury that awarded the two ‘best’ projects at the

some tools one associates with Augmented Real-

end of the first Pop-Up Gallery. Not one, but

ity (AR). This is obvious because the assumption

two winners were awarded: Jente Hoogeveen

in AR is that reality must be supplemented with

en Jorick de Quaatsteniet. Jente won with her

digital information. Or it would be just reality.

project 'Had je dat gedicht?' (roughly translated

The rationale underlying the research project

as 'How does that rhyme with you?'). Error mes-

therefore lies especially in the use of digital

sages we are confronted with on a daily basis

techniques to make this extra layer onto reality

are transformed into poems in order to take

visible. Without digital technology, no addition

some of the zing out of stressful situations.

to reality and certainly no interaction would be

Jorick won with his project 'Get your shit to-

possible.

gether' where animated QR-codes formed text.

It is exactly this assumption that we want to in-

The variety of final projects was staggering

vestigate. Is Augmented Reality only achievable

and impressive. Finally, a publication of all the

by the use of digital means? Can hidden informa-

projects was made with a spread for each stu-

tion be augmented, without the aid digital tech-

dent in which the description and intention of

niques and devices? Following that, we invited

their project were incorporated.

freshmen and -women from the Graphic Design

A special recommendation was given for the

department to get started with Augmented Re-

dyslexia project of Janne van Hooff who visu-

ality. We asked them to think about the kind of

alised how people suffering from dyslexia see

information they wanted to disclose in specific

letters and numbers. Words emerged by moving

situations. What type of problem would they

strips of black paper over the seemingly inco-

want to solve using AR? With what classification

herent rows of letters.

should it work – analogue and/or digital?

These types of projects show the broad range

With this exercise we hoped to widen the con-

of ways in which an extra layer on top of the

cept; while also seeking to come up with new

physical world poses questions about what we

applications and reasons for the use of Aug-

see. What all the students’ projects have in

mented Reality.”

common, regardless whether they are analogue

In completing the first step of the two step

or digital, is that they appear to invite specta-

project, the students gave us a huge input of ex-

tors to look at the world from another perspec-

amples they encountered: they found all sorts

tive. The same goes for those students who,

of material via social media or as search results

at first, were not completely convinced about

that they consequently discussed. The concept

using AR and how they could incorporate this

developing part, step two, turned out to be

technique in their projects. Ultimately, the idea

very exciting, for the students themselves as

was to provide an enriching experience for the

well as for us. We provided technical help when

students and to give them the tools with which

students were really eager to get their ideas

they could incorporate AR in their work.

63

64

augMeNted educatioN How ar tecHNologies exteNd our MiNds ROBIN DE LANGE

In the field of education, just as in many other fields, researchers and developers are experimenting with potential applications of Augmented Reality technologies. Many of these experiments draw on of the possibilities to explore image by Marcelo gerPe, www.creativastock.coM

virtual information spatially. An interesting example is the 4D Anatomy project by daqri (2012), where you can explore the physiology of a human being by moving the display device along a piece of paper with markers. On the screen you see a 3D model of the human body. With sliders and buttons you can set the transparency of the skin, or switch on the layer showing the nervous system for example.

Another interesting project in development is the Sesame Street app Big Birdâ&#x20AC;&#x2122;s Words (Qualcomm, 2013), which uses the latest text recognition algorithms. The (young) users of this upcoming app are asked to look for certain words in their home and aim their device at it. When the device recognizes the word, it gives points to the user. This way the user is asked to involve their environment in the process of learning words.

These examples show some of the new forms of interaction and presenting information, with which developers are trying to create new, interesting, and memorable learning experiences.

65

In this article I will argue that the developments

cal rotation in (b) is actually much faster than

in AR technologies will make digital information

the

sources much more transparently available to us.

were not only physically rotating the shapes to

In certain cases, this information may even be

fit the slot, but were also trying to determine

seen as part of our cognitive process. Because

whether the shape fits in the slot, thereby

of this change of perspective regarding external

simplifying the task. (Kirsh and Maglio, 1994)

mental

rotation.

Furthermore,

players

information sources, AR technologies could not only lead to new learning methods, but could,

It is this example of the human capacity to ma-

and in my opinion should, also trigger debates

nipulate the environment to solve problems,

about the very goals of education itself. To back-

which Clark and Chalmers employ to introduce

up this claim, I will first introduce the concept of

the Parity Principle:

the Extended Mind. “if a part of the world functions as a process

Extended Mind and the Parity PrinciplE

which, were it done in the head, we would have no hesitation in recognizing as part of the cognitive process, then that part of the world is (so we claim) part of the cognitive process.” (Clark and

Andy Clark and David Chalmers start their re-

Chalmers, 1998)

nowned paper ‘The Extended Mind’ with a thought experiment. In this experiment the reader is asked

According to the Parity Principle, the human

to consider three cases and to think about how

mind is not bound by the borders of skin and

much cognition is present in each case:

skull. To make this claim plausible, Clark and Chalmers present a thought experiment involving

(a) A person sits in front of a computer screen

the fictional characters Otto and Inga, who are

which displays geometrical shapes and

remembering how to get to the museum. Otto

is asked to fit the shapes in the displayed

has Alzheimer’s disease and uses a notebook to

‘sockets’, by rotating the shapes in his mind.

serve the function of his memory, while Inga’s

(b) The same situation as in (a), but now the

biological memory is functioning properly. Inga is

person can choose either to mentally rotate

thought to have a belief about the location of the

the shape or physically rotate it by press-

museum, before she recalls this from her internal

ing a button, the latter having some speed

memory. In the same manner, Clark and Chalm-

advantage.

ers argue, Otto can be said to have a belief about

(c) The same situation in a possible future,

the location of the museum before he actually

where the person can choose between us-

consults his notebook. Thereby, under the right

ing his neural implant that does the rotation

circumstances, the notebook can be seen as an

operation as fast as the computer does in

extension of Otto’s memory. By showing how be-

(b), or using ‘old-fashioned’ mental rotation.

liefs are not bound by the borders of the body, Clark and Chalmers show that true mental events

Now, these kinds of cases are actually not as

can extend in the environment as well.

abstract as they might seem: they describe a very well-known real-life situation, namely

All the examples of cognitive extension that

playing the video game Tetris. In their paper

Clark1 gives in his books and papers are not the

about cognitive performances while playing

typical futuristic technologies that come to mind

Tetris, Kirsh and Maglio found that the physi-

when thinking about humans merging with tech-

66

nology. Although the possibilities of Brain-Ma-

Now, wrist watches have been around for many

chine Interfaces and neural implants such as in

decades, writing utensils even for centuries. Dur-

case (c) offer very exciting new ways of commu-

ing this time these technologies have become

nicating with technology, this direct interaction

ubiquitous. They have become socially accepted

with brains is by no means necessary to become

and actually shaped culture itself. An interesting

part of the cognitive process (nor are they suf-

question is whether more modern external infor-

ficient for cognitive extension: communicating

mation sources could obtain the same status as

with technology through a Brain Machine Inter-

these age-old technologies and play a similar, ac-

face usually still takes too much cognitive effort.)

tive role in cognitive processes. Could digital in-

Our brains incorporate the world and some of the

formation sources, for example parts of the Web,

technologies therein in their cognitive processes

actually become parts of our minds?

in such an intimate way, that Clark considers us

Could digital information sources

to be “natural born cyborgs” (Clark, 2003).

In fact, the technologies which Clark considers

actually become parts of our minds?

as cognitive extensions of our cyborg minds are hardly identified as technology anymore. One ex-

To begin to answer this question and decide

ample he mentions is the use of pen and paper

whether a part of the world should (temporarily)

when doing long multiplications. To calculate the

be seen as part of the cognitive process, we can

product of two numbers, we use an algorithm

refer to the three criteria suggested by Clark:

that divides the process of multiplying arbitrary

1.

trusted;

down figures in certain locations, we use the pen to manipulate the external memory source, the

2.

The technological aid should always be available when needed;

paper. The writing utensils play a crucial role in this cognitive process and are therefore, accord-

The information retrieved from the external source should be directly endorsed and

large numbers into very simple steps. By writing

3.

The external resource should be directly

ing to the Parity Principle, actually part of this

available without difficulty. More precisely:

process.

the information access costs, a measure of the combination of time and physical and

Another example shows that it has become com-

cognitive effort, should be as low as with

mon to talk about the information that is in

an equivalent function of the brain (Smart

some of our technologies as if part of our own

et al, 2008).

knowledge. When somebody asks us on the street whether we know what time it is, and we are

From this, it is quite clear that the Web with which

wearing our watch, we often answer “yes”. Sub-

we interact on a daily basis cannot be seen as part

sequently we raise our arm, look at our watch

of our cognition. From our critical stance we do

and see what time it is. Now, according to Clark

not immediately believe most information we en-

this is not simply loosely formulated informal lan-

counter on the Web something which is, due to the

guage. You actually do know what time it is, ‘you’

open character of the Web, probably a wise at-

is only “the hybrid biotechnological system that

titude. Furthermore, the information access costs

now includes the wristwatch as a proper part”

when retrieving information from the Web is way

(Clark, 2003). (This proven transparency of the

too high. The user has to put physical and cogni-

wrist watch is what makes the development of

tive effort into navigating the browser to the right

smart watches interesting.)

page, then wait for the downloading of the page

67

and scan through the text to find the information

know the meaning of a certain word that is not

he needs. In the widespread current way of inter-

in your biological memory, and a short, clear de-

acting, the information access costs of retrieving

scription of the word pops up immediately in the

information from the Web are way too high to be

corner of your field of view, would you say that

considered as part of the cognitive process.

you know the meaning of this word?

Lowering information access costs

I can imagine that you — after you get more and more used to the device and have experienced this situation a few times already — might

However, technology is ever-evolving and many of

say ‘yes’, similar to the situation with the wrist

the present advancements will make information

watch.2 More so, in a very real sense, I think you

available to us in more reliable, quicker and more

might start to feel like you really do know it.

intuitive ways. Processors in smartphones are becoming faster, telecommunication service provid-

But what is it with AR technologies, that they

ers keep improving their networks to decrease

could lower information access cost so signifi-

download times, batteries are getting better, user

cantly? Of course Head-Up Displays (HUDs) play a

interfaces are becoming more intuitive, etc. The

great part in this, by eliminating the physical ef-

technological developments that are associated

fort of getting your smartphone from your pocket

with Augmented Reality especially have the po-

and having to hold it in within your view. When

tential to lower the information access costs for

it does indeed become ordinary to wear HUDs,

digital sources considerably. Let us consider the

information can be presented to the user at all

following scenario to get a better idea: For a few years now you have your own AR glasses. Despite of what many critics expected, it has become socially quite acceptable to be wearing a Head-Up Display at all times. An application running in the background does speech and text recognition and keeps track of words you (the biological ‘you’) don’t know the meaning of. You’ve been using this application for a few months now and it’s starting to have quite a good sense of when to present you with the meaning of a word you encounter. Of course, the applica-

The main goal of education should be to train the technologically extended cognitive system. tion knows when you’re busy driving for example and doesn’t bother you then. Now, when a friend (who is not really into new technology and rather asks a friend to help him) asks you whether you

68

times, at the exact moment when it is needed.

these digital information sources could then be

Another important aspect of AR is the use of

seen as proper parts of our hybrid minds.

information from different sensors and smart algorithms doing image and speech recognition. present information in context-sensitive ways re-

Extended cognition and education

sponding both to the environment and the user.

In the previous I have introduced the concept of

Furthermore, digital information can be placed

extended cognition and explored the possibili-

over the world, which is of course the main idea

ties of digital technology for cognitive extension.

of AR3. By doing so, you can interact with digital

Adopting this philosophical perspective can have

information in similar ways to how you interact

huge consequences for a field like education. One

with the physical world, creating a very natural,

could argue from this perspective that the main

intuitive interface.

goal of education should be to train the techno-

By combining these, possibilities are created to

logically extended cognitive system.

These are the characteristics of AR that create the potential of making digital information much

By accepting digital external information sources

more transparently available to us. I suggest that

as a proper part of memory, it could be decided to

under certain conditions, well designed, person-

store certain information that we want to remem-

alized information sources are able to compete

ber in an easily accessible, personalized cloud of

with mental resources in terms of costs of infor-

knowledge. Instead of trying to store all informa-

mation access. According to the Parity Principle,

tion in biological memory by endless repetition,

Image courtesy of Gaia Dempsey and daqri, see http://daqri.com

69

this task of storing information could be off-loaded

¢¢

daqri. (2012). 4D ANATOMY. Retrieved from http://site.daqri.com/labs/4d-anatomy

to an external source which is constantly available to us at low information access costs. ¢¢

de Lange, R. (2011). Het Semantic Web en

The general view on the use of technology in

netwerktechnologische cognitieve uit-

education is quite different from the view ex-

breidingen. Retrieved from http://www.

pressed in this article though. For the most part

robindelange.com/wikisofie

of their education, students still only get to use some basic technologies: a pen, a piece of paper

¢¢

Dennett, D. (1996). Kinds of Minds: Toward

and maybe a dictionary or an outdated (graphi-

an Understanding of Consciousness. New

cal) calculator5.

York USA: Basic Books.

4

This critical attitude towards the use of technol-

¢¢

Dror, I. E., & Harnad, S. (2008). Offloading

ogy is very understandable. Digital technology is

Cognition onto Cognitive Technology. In I.

developing very rapidly, careful decisions have to

E. Dror, & S. Harnad, Cognition Distributed:

be made about how to use it in education. To

How Cognitive Technology Extends Our

come to these decisions, a lot of research on

Minds (pp. 1-23).

the use of technology in the learning process is needed. Furthermore, there should be an active

¢¢

ESRC/EPSRC Technology Enhanced Learning

discussion on the goals of education and what

Programme. (2012). System Upgrade - Real-

technologies students can use to reach these

ising the Vision for UK Education. London:

goals. An extended view of the mind, in which

tel.ac.uk.

external resources have an active role in the cognitive process, can offer a valuable perspective in

¢¢

Kirsh, D., & Maglio, P. (1994). On distinguishing epistemic from pragmatic action.

this discussion.

Cognitive science , 513-549.

References

¢¢

Qualcomm. (2013). Big Birds words. Retrieved from http://www.qualcomm.com/

¢¢

Blomberg, O. (2009). Do socio-technical

media/videos/big-birds-words-demo-reel-

systems cognise? 2nd Symposium on Com-

video

puting and Philosophy . ¢¢ ¢¢

Rorty, R. (1999). Education as Socializa-

Clark, A. (2003). Natural-Born Cyborgs.

tion and as Individualization. In R. Rorty,

Minds, Technologies, and the future of

Philosophy and Social Hope (pp. 114-126).

human intelligence. Oxford: Oxford Univ.

London: Penguin Books.

Press. ¢¢ ¢¢

¢¢

the Web. Network-Enabled Cognition: The

bodiment, Action and Cognitive Extension.

Contribution of Social and Technological

Oxford: Oxford Univ. Press.

Networks to Human Cognition .

Clark, A., & Chalmers, D. (1998). The Extended Mind. Analysis , 10-23.

70

Smart, P. (2010 (In Press)). Cognition and

Clark, A. (2008). Supersizing the Mind. Em-

¢¢

Smart, P., Engelbrecht, P., Braines, D., Hendler, J., & Shadbolt, N. (2008). The

Extended Mind and Network-Enabled Cognition. Retrieved June 24, 2011, from University of Southampton - School of Electronics and Computer Science: http://eprints.ecs. soton.ac.uk/16649/1/Network-Enabled_Cognitionv17.pdf ¢¢

Wheeler, M. (2011, November). Thinking Beyond the Brain: Educating and Building, from the Standpoint of Extended Cognition.

Computational Culture .

Endnotes

Robin de Lange Robin de Lange has a bachelor’s degree in

1.

The initial paper ‘The Extended Mind’

Physics and Philosophy and has followed

was written by Andy Clark and David

courses on Artificial Intelligence. He is now

Chalmers. Because Clark has written

a student at the Media Technology MSc pro-

many other papers and books on this

gram at Leiden University and is particularly

subject, I will refer to Clark further on.

interested in technologically extended cognition. For his graduation research project

2.

If the word would be jargon of a field you

he is developing an Augmented Reality ap-

are not familiar with for example, you would

plication that supports the graphical solving

probably not understand the meaning di-

of mathematical equations; thereby show-

rectly and need to look up more informa-

ing educational challenges and possibilities.

tion, thereby increasing the costs of information access.

Besides his studies, Robin has taken part in several entrepreneurial projects. Most no-

3.

This characteristic of AR of overlay

tably, he was the co-owner of a company

ing the physical world with virtual objects is

that specialized in homework guidance and

not really present in this scenario. For this

tutoring. He is a freelance video producer

reason, one might argue that the example

and science communicator. At the moment,

does not really show AR. However, it does

he is looking for funding to do a PhD within

use certain AR technologies intensively to

his field of interest.

provide context-sensitive information to the user who interacts with the world.

For more information: www.robindelange.com

4.

The information access costs of looking up a word in the dictionary go through the roof.

5.

Moore’s law seems to be failing here. The hardware in these devices stays roughly the same, even remains the same price!

71

How did we do it: Which augmented reality software does the AR Lab use?

Wim van Eck

72

tHis aPPlicatioN, coMMissioNed by tHe rce (cultural Heritage ageNcy of tHe NetHerlaNds), is develoPed by tHe ar lab usiNg vuforia aNd uNity 3d. tHe PaiNtiNg 'isaac blessiNg jacob' by tHe dutcH PaiNter govert fliNck (16151660) Has beeN augMeNted to give iNsigHt iNto tHe PaiNters Practice aNd offers scieNtific data iN a Playful way. tHe aPPlicatioN will PreMier at MuseuM catHarijNecoveNt (utrecHt, NetHerlaNds) oN May 17tH.

Often we are asked “What is the best augmented

our daily workflow. For this overview we created

reality software?” which is a difficult question to

four categories: ‘easy-to-use for mobile devices’,

answer. First of all, there are so many programs

‘easy-to-use for desktop augmented reality’,

available it is difficult to actually have tried them

‘augmented reality software for interactions’ and

all, and updates and new programs pop up all the

‘head-mounted augmented reality’.

time. Secondly, choosing which software to use really depends on what you want to achieve in the end. At the AR Lab we use a variety of augmented reality programs to realize our projects. I will give an overview of the software we use and

easy-to-use for mobile devices

why we use it. This doesn’t mean we claim that

Since the AR Lab is based at the Royal Academy

this is the best software and that there are no

of Art The Hague most of our projects are devel-

alternatives, we merely aim to give an insight into

oped together with our students, or we supervise

73

The app provides clear information for every step you have to take, removing any technical barriers (Figure 1). Markers can be generated on the spot by taking pictures with the device, and photographs and videos which are stored on the device can be used to augment the chosen scene. When more possibilities are needed there is also a free online application (www.aurasma.com/partners), you only need to register to be able to access it.

This well documented application makes it easier to precisely position your virtual objects, and also to import images and videos which are not on your mobile device, as well as 3D models. It is a pity that Aurasma only seems to give good support for 3ds Max and Maya, it’s more difficult to import 3D models from, for example, Cinema 4d.

It is very easy though to use video with transparency, a not very common option which can give great results. The tracking quality of Aurasma is Fig. 1

not the best around, fast camera movements will result in loss of tracking, but it is definitely good enough for most projects.

easy-to-use for desktop augmented reality Nowadays most software companies seem to focus on mobile devices; desktop augmented reality software is getting scarcer. Some of our students, however, still prefer to present their creations on Fig. 2

a large screen, and BuildAR (http://www.buildar. co.nz) is the software we use most often. BuildAR is based on the world-renowned ARToolKit, but

students who want to develop their own applica-

adds an easy-to-use graphical interface. It runs

tion. Many of these students don’t have a techni-

on both Windows and OSX, though the OSX ver-

cal background, so easy-to-use software is pref-

sion lacks some options and seems to be less sta-

erable. Furthermore, they often have their own

ble. When downloading BuildAR, there are quite

smartphone or tablet and are very keen to de-

some versions which can be a bit confusing. The

velop for it. Aurasma (www.aurasma.com) quick-

‘BuildAR Pro 2 Trial Download’ is the newest ver-

ly became our favourite choice for this group of

sion for Windows and features both marker track-

students. By downloading the free Aurasma app

ing and image based tracking. You can use this

from the iOS App Store or Google Play store stu-

version for free, but there is a ‘demo’ watermark

dents can immediately begin creating their first

on your video-background and you won’t be able

project; all

to save your scene (Figure 2). ‘BuildAR Pro Mac

74

they need is their mobile device.

Trial Download’ is the OSX version and only fea-

it very easy to import 3d-models including anima-

tures marker tracking; the demo-limitations are

tions from almost all 3d-packages. Vuforia’s image

the same as in the Windows version. Lastly, there

based tracking is extremely stable, the tracking

is the ‘BuildAR Free version Download (2008 ver-

quality stays good even if the tracking image is

sion)’, which is only available for Windows and

partly occluded or when there is little light avail-

is free of charge. The only limitation is a ‘HIT-

able. Besides image markers there are also frame

LabNZ’ logo, but many of our students don’t mind

markers available, which use a pattern of black

this logo so much though. BuildAR is easy-to-use;

and white cubes positioned around the image.

the new version can augment your scene with im-

Semiconductor company Qualcomm offers Vuforia

ages, video, audio and 3D models.

for free and unity has a free version as well. However, you will need to buy the unity iOS or Android

ar software for interactions

add-on to be able to export to a mobile device.

A program we often use at the AR Lab ourselves tively develop for iOS (Xcode), Android (Eclipse)

Head-mounted augmented reality

or use it as a plugin for unity (www.unity3d.com)

For head mounted based augmented reality we

allowing you to easily export to iOS and Android.

have solely used software and hardware devel-

unity (Figure 3) is a popular game engine offering

oped by Tu Delft. In issue 2 of AR[t] we intro-

excellent visual quality on mobile devices. Adding

duced our latest augmented reality headset

interactions through scripting is relatively easy,

named Marty and its tracking software. Also, this

there are many tutorials around and the unity fo-

issue of AR[t] features more information about

rum is a great source for information. unity makes

this system.

is Vuforia (https://www.vuforia.com). You can na-

Fig. 3 75

AR[t] Pick Immaterials

by Design Studio onformative and Christopher Warnow, 2011

76

Immaterials is the result of a collaboration between the Design Studio onformative (www.onformative.com) and the Computational Designer Christopher Warnow (christopherwarnow.com). The project explores how information can be integrated into physical space. Using the light painting technique, data is placed in a room. The resulting forms depict possible data sets and examine the design possibilities between technoid holograms and personal notes.

Further information: ¢¢

http://christopherwarnow.com/portfolio/?p=244

¢¢

http://www.onformative.com/work/immaterials/

77

Contributors Wim van Eck

Mariana Kniveton

Royal Academy of Art (KABK) w.vaneck@kabk.nl

Royal Academy of Art (KABK) m.kniveton@kabk.nl

Wim van Eck is the 3D animation specialist of the AR Lab. His main tasks are developing Augmented Reality projects, supporting and supervising students and creating 3d content. His interests are, among others, real-time 3d animation, game design and creative research.

Edwin van der Heide Leiden University evdheide@liacs.nl

Edwin van der Heide is an artist and researcher in the field of sound, space and interaction.

Beside’s running his own studio he’s part-time assistant professor at Leiden University (LIACS / Media Technology MSc programme) and heading the Spatial Interaction Lab at the ArtScience Interfaculty of the Royal Conservatoire and Arts Academy in The Hague.

Pieter Jonker Delft University of Technology P.P.Jonker@tudelft.nl

Pieter Jonker is Professor at Delft University of Technology, Faculty Mechanical, Maritime and Materials Engineering (3ME). His main interests and fields of research are: real-time embedded­ image processing, parallel image processing architectures, robot vision, robot learning and Augmented Reality.

Yolande Kolstee Royal Academy of Art (KABK) Y.Kolstee@kabk.nl

Yolande Kolstee is head of the AR Lab since 2006. She holds the post of Lector (Dutch for ­researcher in professional universities) in the field of ‘Innovative Visualisation Techniques in higher Art Education’ for the Royal Academy of Art, The Hague.

78

Mariana Kniveton is currently a master student at Utrecht Universtity, studying New Media and Digital Culture. Since september 2012 she has worked as an intern at the Research Department IVT and the AR Lab. After a brief stint as a cover model for AR[t] #2, Mariana took up editing duties for this current issue.

Maarten Lamers Leiden University lamers@liacs.nl

Maarten Lamers is assistant professor at the Leiden Institute of Advanced Computer Science (LIACS) and board member of the Media Technology MSc program. Specializations include social robotics, bio-hybrid computer games, scientific creativity, and models for perceptualization.

Hanna Schraffenberger Leiden University hkschraf@liacs.nl Hanna Schraffenberger works as a researcher and PhD student at the Leiden Institute of Advanced Computer Science (LIACS) and at the AR Lab in The Hague. Her research interests include interaction in interactive art and (non-visual) Augmented Reality.

Esmé Vahrmeijer Royal Academy of Art (KABK) e.vahrmeijer@kabk.nl

Esmé Vahrmeijer is the graphic designer and webmaster of the AR Lab. Besides her work at the AR Lab, she is a part time student at the Royal Academy of Art (KABK) and runs her own graphic design studio Ooxo. Her interests are in graphic design, typography, web design, photography and education.

jouke verliNdeN

Matt raMirez

Delft university of Technology j.c.verlinden@tudelft.nl

teamscarlet.wordpress.com

Jouke Verlinden is assistant professor at the section of computer aided design engineering at the Faculty of Industrial Design Engineering.

With a background in virtual reality and interaction design, he leads the “Augmented Matter in Context” lab that focuses on blend between bits and atoms for design and creativity.

Matt Ramirez has over 15 years experience in web design and e-learning content development for a variety of subject areas including medicine, IT, science, special collections and business.

lotte de reus www.lottedereus.nl

guest coNtributors bc “Heavy” bierMaNN The Heavy Projects www.theheavyprojects.com

BC "Heavy" is founder of The Heavy Projects [and its collaborative spin-off Re+Public].

Heavy creates innovative interfaces between digital design and physical worlds in ways that that provoke the imagination and challenge existing styles of art, design, and interaction.

robiN de laNge www.robindelange.com

Robin de Lange is a student at the Media Technology MSc program at Leiden university and part-time entrepreneur.

He is looking for funding to do a PhD on technologically extended cognition.

Lotte de Reus recently received her Master’s degree in Design for Interaction at the Faculty of Industrial Design Engineering at Delft university of Technology. In the future, Lotte would again like to work in the domains that combine art and technology.

alejaNdro veliz reyes university of Salford

Alejandro Veliz Reyes is a teaching assistant and PhD student on digital architectural design at the university of Salford (united Kingdom).

His research interests are design pedagogy, augmented reality, and collaborative technologies for design

aNtal ruHl www.antalruhl.com

oliver Percivall

Oliver Percivall works in I.T Project Management. He is interested in where technology could take us. This interest has resulted in the science fiction novel "The Augmented Star".

Antal Ruhl is a media artist with a background in design, science and art. He creates objects that let us rethink our environment. These objects vary from kinetic sculptures to interactive installations.

NEXT ISSuE

The next issue of AR[t] will be out in the fourth quarter of 2013.

79