augMeNted reality, art aNd tecHNology
re+Public: re+imagining Public space bc â€œHeavyâ€? biermann
Pre-digital augmented reality Maarten H. lamers vestibular stimulation
as an augmented reality layer? antal ruhl
AR[t] Magazine about Augmented Reality, art and technology
May 2013 2
How & Nosm Mural Augment, The Heavy Projects, article on page 8
Colophon ISSN 2213-2481
Contact The Augmented Reality Lab (AR Lab) Royal Academy of Art, The Hague (Koninklijke Academie van Beeldende Kunsten) Prinsessegracht 4 2514 AN The Hague The Netherlands +31 (0)70 3154795 www.arlab.nl firstname.lastname@example.org
Editorial team Hanna Schraffenberger, Mariana Kniveton, Yolande Kolstee, Jouke Verlinden
Contributors AR Lab & Partners: Wim van Eck, Edwin van der Heide, Pieter Jonker, Maarten Lamers, Maaike Roozenburg Guest Contributors: Alejandro Veliz Reyes, Antal Ruhl, Lotte de Reus, Matt Ramirez, Oliver Percivall, Robin de Lange, BC “Heavy” Biermann
Graphic design Esmé Vahrmeijer
printing Klomp Reproka, Amersfoort
CoVer Our AR[t]y cover is a work by Royal Academy of Art student Donna van West who participated in the Smart Replica’s project, see: www.donnavanwest.nl
Table of contents
30 welcome to AR[t]
Re+Public: Re+Imagining Public Space
Who owns the space 2
The Misadventures in AR
BC “Heavy” Biermann — PhD
MaRty, the new affordable headset — Part 2
From: Hanna Schraffenberger To: Lev Manovich
Pre-digital AR Maarten H. Lamers
Vestibular stimulation as an AR layer?
Alejandro Veliz Reyes
A Study in SCARLET
Pieter Jonker and Yolande Kolstee
Introducing AR to First Year Graphic Design Students
Edwin van der Heide
Nonlinear stories told by cups and saucers
44 50 56 62
How did we do it
Robin de Lange
Wim van Eck
AR[t] Pick: Immaterials Editors' Choice
64 72 76
Lotte de Reus 5
fiNe art studeNts sHow tHeir work oN tHe PoP-uP gallery
two graPHic desigN studeNts (dave PoPPiNg aNd gabor kerekes) Made a MoirĂŠ-PatterN as a decodiNg key
WELCOME... to the third edition of ar[t], the magazine about Augmented Reality, art and technology! In this issue we present articles by contribu-
In the second part of this issue, special attention
tors from all over the world who are involved in
is given to education: Experiences of bringing AR
stretching the borders of augmented reality, on
into the classroom are presented in separate ar-
the edge of art and technology. We feature both
ticles by Alejandro Veliz Reyes, Matt Ramirez and
articles with a philosophical perspective and ar-
Yolande Kolstee. The results are a wide variety
ticles with a more technical point of view.
of implementations and reflections in different creative contexts. Robin de Lange considers the
In Re+Public’s article we can read about blurring
long-term ramifications of extending both the
private properties’ boundaries, in the attempt to
mind and cognition itself with AR.
leverage AR to allow artists to make incursions into public spaces, and in ways they were previ-
Wim van Eck continues his AR tutorials in the se-
ously physically unable to do.
ries ‘How did we do it’, describing some AR software programmes which are widely available.
We are very pleased to introduce to you the improved version of Marty: the video see-through
In our new section AR[t] Pick, we share artworks
AR head-up display. These drawings & 3D model
that caught our eye. In this issue, we have chosen
will be downloadable from our website, to print
'Immaterials' — the result of a collaboration be-
out at your local 3D print facility!
tween the onformative design studio and Christopher Warnow.
Hanna Schraffenberger sets about interviewing Lev Manovich, well-known among many people
We invite you to visit our new website, via the
since the publication of his book ‘The Language
same uRL www.arlab.nl, which will provide you
of New Media’ in 2001. Maarten Lamers takes us
with all the information on our artistic and tech-
back to pre-digital AR with his story on Pepper’s
nical research — but also that of fellow research-
Ghost and at the other end of the spectrum An-
ers, and information about our experiments in
tal Ruhl explores the potential of using vestibular
the cultural domain. On this website, in the part
stimulation in order to create new AR experienc-
News Picks we post short news items on AR art-
es. Crucial to AR experiences is the concept of
ists, scientists, events and developments. Please
believability, explored by Edwin van der Heide.
feel free to contact us to tell us what caught your
Lotte de Reus discusses a spatial audio interven-
eye, which might lead to an item there. We are
tion to enhance museum exhibits.
confidant you will enjoy this issue. Should you like to contribute to issue #4, look out for our call for
A recurring topic is set out by Yolande Kolstee:
contributions, which will be posted on our web-
the legal ramifications of AR initiated in AR[t] #2.
Furthermore, we feature a short science fiction sequel by Oliver Percivall.
Yolande Kolstee, Head of AR Lab 7
outdoor advertisiNg: ar | ad takeover (Nyc, 2011), roN eNglisH, photo by will sHerMaN
re+Public: re+iMagiNiNg Public sPace
BC “HEAVY” BIERMANN — PHD
RE+PuBLIC A creative collaboration between The Heavy Projects (Los Angeles) and the Public Ad Campaign (New York City), Re+Public is dedicated to using emerging media technology, and augmented reality (AR) in particular, to alter current expectations of our public media environment generally dictated by property ownership, the ability to pay for its usage, or a willingness to break the law. Blurring private property boundaries, Re+Public seeks to leverage AR in an effort to allow artists to make incursions into public spaces in ways they were previously physically unable to do. With this goal in mind, Re+Public has developed an experimental mobile device application that digitally resurfaces three specific areas of public space: outdoor advertising, murals, and buildings. As such, this article focuses on these domains and demonstrates how Re+Public has used AR to transcend current private property boundaries, which lies at the heart of our endeavor to re+imagine public space as a more open visual commons.
OuTDOOR ADVERTISING: AR | AD TAKEOVER (NYC, 2011) AR presents unique opportunities to creatively problematize the political and economic systems that shape the awareness of both individuals and the public writ large. Seeking to understand a profit-driven governmentality and its affect on
public space, Re+Public investigated how civic
sumptive message. Specifically, users could trigger
authorities allow certain private parties to profit
web-based information related to the showcased
while preventing or discouraging other forms of
artists whose work has historically addressed com-
public media production. To this end, we envi-
mercial advertising in public space such as: Ron
sion AR as the first step in the evolution of better
English, Dr. D, John Fekner, PosterBoy, and OX.
tools of expression that can democratize public media production.
We foresee AR mobile device technology as a first step in the transformation of public space into an
The AR | AD Takeover used street level ads and bill-
arena shaped by user created content. In other
boards to trigger a curated digital art installation
words, AR is an incremental step towards showing
that displayed on mobile devices. In New York City,
the public an alternate view of their landscape,
we augmented ads in Times Square with artistic
which commercial ads do not necessarily have to
content. Our digital infiltration into public space
dominate. Once individuals experience this AR
and takeover of commercial ads created a place of
version of reality, they might start demanding a
dialogic interaction rather than a monologic con-
better version of public messaging than the billboard default.
Outdoor Murals: Bowery Wall (NYC, 2012) and Wynwood Walls (Miami, 2012) In contradistinction to the use of AR to problema-
they move in perspectival relation to the viewer.
tize the consumptive monologue of outdoor ad-
It is precisely this kind of spatial “aura” that dis-
vertising, at both the Bowery and Wynwood Walls
tinguishes AR from other types of emerging me-
sites, we used AR to rupture public space with a
dia technology. AR connects the digital with the
new kind of artistic interactivity.
physical in an intimately “present” way, whereas other technologies tend to disconnect the view-
The Bowery Mural Wall is an outdoor mural ex-
er from their immediate physical surroundings.
hibition space in Manhattan. Owned by Goldman
Viewing previous murals online, for example, re-
Properties since 1984, real estate developer and
moves the viewer from the space by placing them
arts supporter Tony Goldman started the Bowery
squarely in the absent, digital world.
Mural with Jeffery Deitch and Deitch Projects. In 2008, the mural series commenced with a
In 2009, Goldman Properties and Tony Goldman
recreation of Keith Haring’s famous 1982 mural
who was looking to transform the industrial ware-
followed by work by such recognized artists as Os
house district of Miami also conceived the Wyn-
Gemeos, Faile, Barry McGee, Aiko, Kenny Scharf,
wood Walls. Beginning with the 25th and 26th
Retna, and Shepard Fairey.
Street complex of six separate buildings, Goldman endeavored to create a center that devel-
In 2012, Re+Public used AR to resurrect murals
oped the area’s pedestrian potential.
that once existed on the wall. In other words, by pointing a mobile device at the present mural, us-
Drawing from an international pool of talent, art-
ers are able to see the former murals, in situ, that
ists who have contributed to the Wynwood Walls
the current artist has painted over. While users
include Os Gemeos, Invader, Nunca, Saner & Sego,
could certainly view the previous murals online,
and Swoon among many others. During Art Basel
AR permits users to see these murals as if they
2012, Re+Public was commissioned by the Wyn-
were actually back on the wall, in the space, as
wood Walls to create an AR experience. In addi-
outdoor Murals: bowery wall (Nyc, 2012), resurrected keitH HariNg
tion to resurrecting a Shepard Fairey mural that he
sion of creative content. In addition to an immer-
recently painted over with a new mural in tribute
sive garden with a bridge, stream, and flowing
to the recently deceased Tony Goldman, Re+Public
waterfall, we created a 3D Kabuki theater that al-
created 3D, interactive environments for four
lowed users to walk into a digitized version of the
other murals by How and Nosm, Ryan McGinness, Aiko, and Retna. Additionally, we worked directly with MOMO and collaborated on an AR version of his indoor mural at the Nicelook Gallery on site. In considering our deployment at the Wynwood
AR is an incremental step towards showing the public an alternate view of their landscape, which commercial ads do not necessarily have to dominate.
Walls in particular, we wrestled with this new
mural where all of the elements were separated in
type of work and wondered if it constituted a new
Z-space. With the How and Nosm, we created an-
mode of art. Without pretending to discern any
other immersive, rather abstract 3D environment
immediate resolution, it is arguable that the AR
and permitted users to both pull apart and recon-
assets represent original works in that they con-
struct the mural elements. With the Retna, we
tain a sufficiently new visual expression of ideas,
built the mural shapes in 3D and animated them
over and above those embodied in the 2D mural. In
to extend out of the wall and placed them both
the case of each traditional mural, the AR overlay
on the ground and above the wall. Finally, with
used the 2D paintings as feature tracking markers
the McGinness, we made it appear as if the paint
and source material to produce an original expres-
colors were draining out of the mural.
Retna Mural Augment, full view
Retna Mural Augment, screenshot
AuGMENTED ARCHITECTuRE: PEARL PAINT, WILLIAMSBuRG ART & DESIGN BuILDINGS (NYC, 2012) BRADBuRY BuILDING (LA, 2012)
cal aura that may have drawn the viewer to the building in the first place. In other words, the AR assets should maintain some logical connection to the building or space upon which we have attached them.
In addition to using both outdoor ads and murals as the markers that trigger our AR deployments, we
Whether it is outdoor ads, murals, or entire build-
have also experimented with using buildings, and
ings, Re+Public seeks to continue to deploy AR
their unique architecture much in the same way.
in an effort to democratize access to our shared
To this end, we digitally resurfaced, or “skinned”
visual environment and alter the current expec-
physical buildings in urban centers by overlaying
tations of urban media in accomplishing our core
3D content onto the physical environment.
mission of re+imagining public space. It is vital to the health of any city that its inhabitants are
ultimately, in our attempt to use AR to re+imagine
able to participate, in some meaningful way, in
public space, we really see the city as a canvas
the visual urban messaging systems that surround
that allows for a multiplicity of voices to enter
them. With the coming advancements in wear-
into our visual landscape, rather than the cur-
able computing, the digital overlay will become a
rent commercial hegemony. Working with mu-
much more seamless and natural part of our daily
ralist MOMO, we chose three buildings that had
existence. It is our hope that these early entrants
a particular cultural significance and, using AR,
will help create experiences that consider art and
made it possible for MOMO to put his art on build-
design as an important part of the way the public
ings that he could not have accessed in his tradi-
adopts this technology.
tional 2D format. Converting MOMO’s 2D designs into digital 3D models optimized for mobile, we placed his art on both the Pearl Paint and the Williamsburg Art & Historical Center buildings in New York City. Both structures have a long “art history” in the city and AR allowed us to blur the lines between private and public space. In Los Angeles, we converted the Bradbury Building, site of many interior shots in the film Blade Runner (1982) into a futuristic version of itself. In this way, instead of placing converted art on the structure, we created the first example of what we refer to as “city visions.” Specifically, we used AR to provide an artistic rendering of the re+imagined building by projecting it into a Blade Runner style future. This city vision type of deployment potentially provides more practical architectural and urban planning uses and maintains our notion that the AR experience should be spatially relevant in order to maintain the physi-
williaMsburg art & Historical ceNter buildiNg augMeNted witH MoMo urbaN art
bc “Heavy” biermann Deriving his pseudonym from his penchant for philosophical discussion, BC “Heavy” Biermann possesses an interdisciplinary background that comprises technology, academia, and the arts. With a PhD in Humanities [Intermedia Analysis] from the universiteit van Amsterdam, BC has worked as both a university professor and a tech developer in Anaheim, Prague, and Saint Louis. Since 2007, BC has internationally presented his academic work, which explores augmented reality, art and semiotics in public space. As a kind of synthesis between scholarly inquiry and emerging media, BC founded The Heavy Projects [and its collaborative spin-off Re+Public] to investigate how the fusion of creativity and technology can uncover new modes of relaying ideas. Building upon existing technological and theoretical frameworks, Heavy creates innovative interfaces between digital design and physical worlds in ways that that provoke the imagination and challenge existing styles of art, design, and interaction. After finally giving up his painfully amateur skateboarding career due to a bum right knee… BC plans to use his extra time continuing to examine meaningful ways to fuse tech + creativity.
outdoor Murals: wyNwood walls (MiaMi, 2012), resurrected sHePard fairey, photo by jordaN seiler 14
Marty, the new affordable headset — Part 2 Pieter Jonker AND Yolande Kolstee
In AR[t] 2 we announced the new MARTY video-
hence there is extra knowledge on scale. In mark-
see-through headset as a design based on the
erless systems, natural features like dominant cor-
Sony HMZ T1. It was designed by Niels Mulder
ners or lines in the scene are tracked, no assump-
from studio RAAR on assignment of the TU-Delft,
tions on scale can be made and two cameras that
partner of the AR Lab, with the aim to do re-
see that same feature are necessary.
search on co-operative AR. One of the aims of the AR Lab is to stimulate AR in the world by putting
And finally: Yes we were already decades ago
our developments in public domain. When the
inspired by fighter pilots with heads-up display
new AR Lab website will be in the air at about the
and of course Steve Mann, the geek that walked
same time as this AR[t] 3 magazine is published,
around with an AR display. But also lower back-
we will post the design files as well as a photo
pain, cramped shoulders from long computer days
series how to assemble the Marty, so anyone with
and the sighing, “why can laying on bed or in an
access to rapid prototyping facilities can repro-
armchair not the ceiling or the white wall, the air,
duce Marty for his/her own scientific or artistic
or anything at any time be my display?”
research. Keep an eye on our website as we will also link to the software to do 3D pose tracking
based on natural feature tracking. The industrial design of Marty is meant to do exWe are well aware that the Sony HMTZ T2 is on
periments by researchers, artists and designers
the market, which has another face mask. We are
in the area of AR and stimulate industry to come
aiming at adapting Marty to version 2.0 that is
up with affordable AR equipment. It is, however,
based on the HMZ T2. Due to a lack of manpower
subject to copyright, meaning that no large scale
and funds this might, however, take a while. In
industrial production can be based on this design
the mean time be our guest and contribute to AR
without the consent of the copyright owner. For
to come up with your own solutions. If they are
more information: email@example.com.
valuable we will publish about it in our next AR[t] magazine.
We are also aware of Google glasses and other
Bridging the multiple reality gap: Applica-
equipment and we welcome those developments!
tion of Augmented Reality in new product
However, still many companies do not understand
development. Anna P. Chatzimichali, Wim
that you need two camera’s to see virtual objects
Gijselaers, Mien S. R. Segers, Piet Van den
in 3D and two camera’s to track (salient) key-
Bossche, Hetty van Emmerik, Frido E. Smul-
points in 3D in order to track your head pose while
ders, Pieter P. Jonker, Jouke C. Verlinden In
walking around with your headset in any unknown
proceedings of the IEEE International Con-
environment. AR-toolkit like markers can do with
ference on Systems, Man and Cybernetics,
one camera as the size of the marker is known and
Anchorage, Alaska, USA, October 9-12, 2011
MARTY, THE NEW AFFORDABLE HEADSET. Image by FERENC MOLNAR
ON OuR NEW WEBSITE (WWW.ARLAB.NL) WE WILL SOON POST A PHOTO SERIES ON HOW TO ASSEMBLE THE MARTY HEADSET.
---------- Forwarded message ---------From: Hanna Schraffenberger <Hanna@arlab.nl> Date: 2013/4/3 15:53 Subject: interview AR[t] magazine
Dear Lev, Maybe you remember me from Facebook. I work at the Augmented Reality Lab in The Hague and I am one of the editors of the AR[t] magazine. When I read your article The Poetics of Augmented Space, I realized that I would like to interview you about Augmented Reality for the AR[t] magazine. A short time ago, I finally also read The Language of New Media. As a consequence, I’d like to interview you even more. So I hope you’ll agree to an interview for the magazine? Best regards, Hanna P.S. After my last few interviews, my supervisor (Edwin van der Heide) told me that I could/should be more critical towards my interview partners. So I’ll challenge myself to challenge you. P.P.S. Maybe we can print my questions in issue 3 and your answers in issue 4? 18
Manovich, Lev. The language of new media. The MIT press, 2001. image source: www.jewishphilosophyplace.wordpress.com
To: Lev Manovich
where to go and what to look at. In contrast to ‘typical’ visual AR, the user is presented with au-
what is augmented reality?
ditory information that relates to the immediate surrounding space. Personally, I would call this
To begin with, I would like to ask you what you
Augmented Reality. Wouldn’t you?
consider Augmented Reality (AR) to be. In The Poetics of Augmented Space you describe AR as ‘the laying of dynamic and context-specific infor-
mation over the visual field of a user’. It would be great if you’d address the topic once more. Firstly, because our readers might not have read
what is special about ar compared to other forms of augmented space?
your article. And secondly, because I think that this point of view unnecessarily limits AR to the
In your article The Poetics of Augmented Space
you discuss the concept of Augmented Space. Augmented Space refers to all those physical spaces
In The Poetics of Augmented Space, you mention
that are overlaid with dynamic information such as
Janet Cardiff’s audio walks as great examples of
shopping malls and entertainment centers that are
laying information over physical space. These
filled with electronic screens and all those places
walks are designed for specific walking routes.
where one can access information wirelessly on
While navigating the environment, one gets to
phones, tablets or laptops. Besides AR, you men-
listen to a mix of edited sounds that blend in with
tion several other technological developments in
the sounds of the surroundings, as well as spo-
the context of Augmented Space, among which,
ken narrative elements and instructions such as
for example, monitoring, ubiquitous computing,
tangible interfaces and smart objects. Is AR just
that in AR, something virtual augments something
one of many related recent phenomena that
real. More specifically, the virtual augments that
play a role in overlaying the physical space with
to which it relates. In our view, space is one of
information? What’s special about AR compared
the possibilities, but likewise, we have considered
to other forms of Augmented Space?
things like augmented objects, augmented humans, augmented perception, augmented content
What else can be augmented?
and augmented activities. What is augmented depends on what the additional content relates to.
Something I really like about your article is that
I am curious whether you’d agree. Do you think
you see augmentation as an idea and a practice
that all forms of augmentation bring along an
rather than a collection of technologies. However,
augmentation of space or influence our experi-
so far, you have only discussed the augmentation
ence of the immediate surrounding space?
of space. I was wondering whether you have considered other manifestations of augmentation as well. I don’t think augmentation is limited to a
Information and space — one coherent gestalt?
space or an environment. I’d even say that often it’s not the space that is augmented, but some-
In The Poetics of Augmented Space you raise a
question that intrigues me a lot. Do the real space and the dynamically presented information add up
For example, you mention software that performs
to one single coherent phenomenological gestalt
tasks according to the mood, pattern of work, fo-
or are they processed as separate layers?
cus of attention or interests of their user. However, I am doubtful whether our experience of a
I am a bit of a sound-person and it has always fas-
space is affected by this kind of information. Let’s
cinated me that sometimes the sounds of a radio
imagine that my phone registered that I have been
seem to mix in with environmental sounds. For
sitting still for a long time and reminds me to take
example, the ticking of a red streetlight might
a short break to stretch my legs. This information
perfectly mix in with the rhythm of the song that is currently playing. Listening to a radio play, an
In AR, something virtual augments something real.
event could sound so real and so nearby, that I’d turn around, just to find, that nothing is happening there. But of course, most often, the sound of the radio just exists as a separate, independent layer of content. The voice of the newsreader doesn’t mix
relates to one individual in the space (me), to the
with the voice of my colleague, nor does it relate
activity the person is performing (sitting still), but
to my environment. Most of the time, a song is just
I don’t think it has anything to do with the sur-
a song, and has nothing to do with the surround-
rounding space. Hence, I might consider it an aug-
ing space. So judging from my experience of listen-
mentation of the activity (not moving, sitting still)
ing to the radio, information and the surrounding
or an augmentation of the user (me), but I don’t
space can be perceived as one single mixed thing
consider it an augmentation of space.
as well as independently. But besides these two options, there are more possibilities. For example, the
Edwin (my supervisor) and I have recently given
newsreader might tell me about a traffic jam and
this topic a lot of thought and we were fascinated
thereby inform me about my immediate physical
by the questions: “What is actually augmented in
space. Here, the information and my spatial sur-
Augmented Reality? What else can (we imagine
roundings aren’t perceived as a single gestalt, but
to) be augmented?” We came up with the answer,
nevertheless, there is a relationship between both.
I think the same is true for Augmented Space.
equally to renaissance paintings and to mod-
Often, information and space might be related,
ern computer displays. When we imagine a
even when they don’t add up to one phenomeno-
typical AR scenario in which virtual objects
logical gestalt. So some questions I’d like you to
are integrated into a real scene (e.g. a virtual
answer with respect to Augmented Space are:
bird is sitting on a real tree) there is no sec-
when are information and space perceived
ond space. It’s the same physical space, which
independently from each other — would you
contains both virtual and real elements. is
still call these occurrences augmented space?
this a fundamental change in visual culture?
when are information and space perceived as separate but related layers? and when and why
ar & the quest for realism
do they add up to one single gestalt? The quest for realism in computer graphics
is something that has always bored me. You note that new technological developments illustrate how unrealistic the previous ex-
One of the main questions I want to ask you is:
isting images were. At the same time they
what makes augmented reality special? I have
remind us that current images will also be
posed that question with respect to other forms
superseded. I was wondering: How does ar
of augmented space. I’d like to ask it again with
fit in the widespread aspiration towards
respect to the history of new media.
realism? On the one hand, visual AR could be considered a huge step back. The 3D models
Personally, I don’t think of AR as a recent phe-
that are usually integrated in real space don't
nomenon. Of course, there are more and more
come close to the level of photorealism we
so-called AR applications, AR technologies and
know from cinema. On the other hand, the
new media works that work with AR. However,
virtual leaves the realm of virtual space and
when we consider the concept of AR, we find
enters our real physical environments — with
examples that date back centuries. An example
respect to that the images might be experi-
of ancient AR is the Pepper’s Ghost trick (which
enced as more realistic than ever…
is discussed by Maarten Lamers on page 24). It uses a second room, glass and special lighting in
will ar take the quest for realism to a new
order to let objects seem to appear, disappear
level? I can imagine, when striving for real-
or morph into each other in an otherwise real,
ism, the virtual things that appear to exist in
our physical space should not only look like real things — ideally they also feel like them,
But even if the concept isn’t new, current manifes-
smell like them, taste like them and behave
tations of AR might still bring something new and
like them. will photorealism be traded in
special to the table. if we look at contemporary
for a form of realism that encompasses all
ar and compare that with other forms of new
senses? do you think new media will de-
media, what’s special about it and what isn’t?
velop towards a more multimodal form?
ar & the second space
ar & cinema
From The Language of New Media, I understood
In The Language of New Media, you relate dif-
that throughout media history, the screen was
ferent forms of new media — e.g. Virtual Real-
used to separate two absolutely different spaces.
ity, websites and CD-ROMs — to cinema. How
For example, this function of the screen applies
about the relation between ar and cinema?
I’m certainly not a cinema expert, but I guess
lationship to something real. Could we say that
most of what we see in visual AR has been pres-
when working with AR, artists and designers
ent in cinema for a long time. For example, AR
create a database for an existing interface?
research is very concerned with registering virtual objects in real space. As far as I understand it,
I have one more question about databases. In The
this can be seen as an analogy to compositing in
Language of New Media you write about the ele-
films: an attempt to blend the virtual and the real
ments of a database:
into a seamless whole ‘augmented’ reality. Do you agree?
“If the elements exist in one dimension (time of a film, list on a page), they will be inevitably or-
You oppose compositing to montage: while com-
dered. So the only way to create a pure database
positing aims to blend different elements into a sin-
is to spatialise it, distributing the elements in
gle gestalt, montage aims to create visual, stylis-
tic, semantic, and emotional dissonance between them. Do we have montage in AR as well? (You
In AR, virtual elements are distributed in real
give the example of montage within a shot, where
space. Can we understand this as a pure database?
an image of a dream appears over a man’s sleeping
What are the consequences of working with
head. The same could easily be done in AR.)
spatialized elements? What are the inherent limitations and possibilities when working with
Does visual AR use similar concepts as cinema?
this form? (I can imagine it has consequences, e.g.
Does cinema use other techniques to create
for storytelling? As you point out, we cannot as-
fictional realities that are not (yet) used in AR?
sume that elements will form a narrative when
Does AR use techniques that might be adapted
they are accessed in an arbitrary order...)
by cinema in the future?
AR as spatialized databases
AR & future research With The Language of New Media, you did not only One of the main claims in The Language of New
provide a theory of new media; you also pointed
Media is that at their basis, all new media works
your readers towards aspects of new media that
are databases. You argue that what artists or de-
were still relatively unexplored at that time and
signers do, when creating a new media work, is
you suggested directions for practical experimen-
constructing an interface to such a database.
tation. Are there certain aspects of Augmented Reality you consider especially interesting for
Let’s apply this database theory to a typical AR
future experiments and explorations?
scenario in which virtual objects (seem to) appear in a real environment. We can see this as
a database filled with virtual objects. The database might hold a virtual chair, a virtual pen and
a virtual painting. These virtual objects are displayed as part of a real room when a user views
Manovich, L. (2001). The language of new media. The MIT press.
Manovich, L. (2006). The poetics of aug-
the augmented environment with a smartphone.
mented space. Visual Communication, 5(2),
(Technically speaking, we could say the real world
serves as a database index for those virtual elements.) What is the interface to access the database? Is it my phone? What does the artist create? I think it is usually the virtual content and its re-
â€œ[...] the only way to create a pure database is to spatialise it, distributing the elements in space.â€?
Image courtesy of Lev Manovich
Lev Manovich, The Language of New Media
rten a a M
Pre-digital augMeNted reality MAARTEN H. LAMERS In the first issue of this magazine, I mentioned how the “Mozzies” mobile game of 2004 was my earliest encounter with augmented reality. i lied... by 26 years! in 1978 my parents took me and my brother to disneyland. besides meeting Mickey Mouse, i was most impressed by the augmented reality of Pepper’s ghost. with no computer required... 24
.net lbourke rke, pau Paul Bou Image
cou rtesy of D avid http Llewellyn ://ravenmanor , .com
Sitting beside my brother, a mechanical fun-
ment parks and museums today, but also as
house car drove us through Disney’s “Haunted
part of modern optical see-through AR tech-
Mansion” ride. Scary stuff, of which I can re-
nology. In Microsoft Research’s HoloDesk (see
member only one thing: the car stopped, ro-
tated 90 degrees to the right, facing a large mir-
project, its use is apparent. However, in cur-
ror, and on the seat between us appeared the
rent head-up see-through displays Pepper’s
scariest moving ghost ever! Instantly our heads
Ghost technology is less apparent but nonethe-
turned, facing each other. Thank God, the ghost
less used in the same fashion.
wasn’t really between us. The mirror showed us
To me it is interesting that we still rely on John
some weird illusion, most realistically.
Pepper’s idea to add digital content to our op-
Naturally, I asked my dad how the illusion
tical reality. Actually, digital technology lets us
worked. He explained something about glass,
define, render and interact with virtual con-
darkness and reflecting light. In effect, he ex-
tent. But good old-fashioned Pepper’s Ghost
plained what is known as Pepper’s Ghost: a
projection is what augments our reality with
simple but clever technique that creates holo-
graphic scenes in 2 or 3 dimensions. Combining
Who would have guessed that such basic
this with a large mirror, Disney augmented the
illusionary tricks are crucial to what we now
reality that we hold our mirrored image to be:
consider cutting-edge technology? In fact,
pre-digital augmented reality.
if you know other pre-digital augmented
Pepper’s Ghost technique was first described
reality techniques, send a short description
in the 16th century and later refined by John
to firstname.lastname@example.org, and help me put AR in
Pepper around 1860. It is still used in amuse-
Image by Kristin Smith
vestibular stimulation as an augmented reality layer? Antal Ruhl Over the past few years the field of Augmented
intensity and the direction of the current. This
Reality has started to include non-visual forms
technique has been around for a long time and
of augmentation to their scope. When I became
many studies have been done in order to learn
interested in galvanic vestibular stimulation I re-
more about the human balance system, but never
alized that this could become a whole new form
has it been used as an extra sensory input. To find
out whether GVS can be used as a new form of
Galvanic Vestibular Stimulation (GVS) stimulates
AR, I constructed a device and developed a series
the human vestibular system (balance organ) by
of experiments. In order to use GVS as a form of
means of electric stimulation. Two electrodes,
augmented reality, I needed a connection to the
usually an anode and a cathode, are placed on the
physical world. Since I also wanted to investigate
mastoids (the bones behind the ear). Applying a
what the effect would be to our balance system,
specific current to those mastoids will result in an
I used my body orientation to alter my balance.
altered balance perception related to both to the
GALVANIC VESTIBuLAR STIMuLATION CAN BE uSED TO ALTER BALANCE PERCEPTION
Setup To do so, I mounted an accelerometer on top of
own orientation. There were two possible modi
my head-mounted GVS device in order to measure
for the control unit. Modus A could counteract my
its orientation. This provided me with the setup
balance, while modus B could amplify it. In other
as shown in Figure 1.
words, if I was tilted to the left in modus A, it
The electrodes are incorporated in a pair of head-
would push me back to the center, in modus B, it
phones to make sure they are pressed against the
would push me further to the left.
mastoids properly. The headphones are in fact
Testing this simple setup immediately revealed
merely rings around the ears. The middle part is
some potential AR applications of vestibular stim-
left uncovered to retain the full use of usersâ€™ au-
ulation. While testing, I realized that, when put
ditive orientation, and to make sure this wouldnâ€™t
in modus A, it felt like I was moving through a liq-
influence the results. The data from the acceler-
uid or a thick syrup-like medium. The GVS device
ometer is sent to the control unit. The control
counteracted all my movements, so it took more
unit calculates the appropriate intensity and di-
effort to move around. In modus B, it had the op-
rection of the current in real-time and provides
posite effect; it felt like my resistance was really
feedback to the electrodes.
low since the device backed up every movement
I made. Without even using external data to augment an extra layer to reality I created an altered
The first step of my research project was measur-
world based on data from the physical world. Is
ing the effect of an altered balance, based on my
this a form of augmented reality or is it an alter-
The tests that I have done focused on enhanced performances in everyday life situations. For example, I was wondering whether I can diminish or amplify motion sickness, improve my balancing skills etc. During the testing phase, I have used this device for over two weeks to see if there was any progress. Tests involved bus rides, walking to targets on the street, walking over a balance beam, playing Wii Balance Board games and many other examples. Although we have to be very careful about drawing conclusions from these results (given the self-experimentation and singlesubject constraints), I found that in certain cirnate reality? Be that as it may, it does fall within
cumstances my balance actually improved while
the definition of mediated reality, which is the
using this device. This might be of use for people
broader field of manipulating users’ sensory per-
with an impaired balance system or in those situ-
ception through a wearable device, and letting
ations when accurate balance is crucial.
them experience their surroundings in a modified fashion. Using only these two simple orientation-
Can this be used as an interface?
based modi, we can already create an enriched
I demonstrated a system in which the real world
environment, but for what use? The examples
environment, (in this case body orientation,) is
point towards simulations of the physical world,
measured, modified and used as an input to alter
which could be used to train divers or astronauts
the actuators (in this case the electrodes which in
who work in other environments. But you can also
turn alter my own balance). This creates a broad
think about fighter pilots who fly in simulators
range of possibilities. But can we also use it in a
that don’t alter pilots’ balance when flying upside
more traditional AR environment? In other words,
down. This system could enhance a simulated en-
could we use this system to add computer gener-
vironment by distorting the balance organ based
ated data to an extra sensory layer of our envi-
on the simulators’ virtual orientation.
ronment? The easy answer is: Yes! It’s not hard to imagine that you can enrich game play with wave motions for example. Or simulate the G-forces in a racing game’s sharp corners. However, before we can use this, there are some practical considerations to keep in mind. Using my self-constructed device, I experienced some burn marks of the skin around the mastoids. The skin’s resistance is an important factor in getting the electric pulses down to the desired area. Using electrode gel and covering the electrodes with sponges soaked in a salt-water solution did the trick, for now. But a more practical solution must be found. Other issues were the bright light flashes I experienced while testing, and nausea; both are not very userfriendly. But these issues might all be overcome with the right execution. Another, perhaps more
Figure 1: Galvanic Vestibular Stimulation device
troublesome, experience I had was the adapta-
tion effect. You might remember that study from the 1960’s where a test subject was wearing reversal goggles. After a few days the brain adapted to the newly displayed environment and reversed the image back to normal. After removing the goggles, the subject saw the world upside down because the body had adapted to this new situation. The same effect occurred with my balance organ, but at a much quicker pace. While my body needed some time to get used to my ‘new and improved’ balance, it did so quite convincingly. And it did so in two ways. When I used a constant pulse on the electrodes (a constant current in the same direction) the effect diminished quite soon. But when I used an alternating pulse, and only changed the pulse-width to affect my balance, the effect was constant. Apparently I was much more sensitive to a change in stimulation than to the actual stimulation itself. Secondly, when I was wearing the device for about fifteen minutes, and removed it after that time, my body had to adapt to this ‘new’ non-stimulated situation. I saw that, when I was wearing the device while it was set in modus A, it gave me the experience as if I was moving through a liquid. After removing the GVS device I experienced the exact opposite. So when I turned it off, it felt like I had switched it to modus B; thus, giving off the impression of moving through a low friction environment. Your body is extremely good at adapting to new situations and environments, which are very important design-considerations if we want to use a GVS device as an interface. While using GVS for augmented reality purposes is still in a research stage, the possibilities are endless if you use your imagination. This article is based on research presented at Chi Sparks 2011 (Chi Nederland 15th conference), June 23, 2011, Arnhem, The Netherlands. The
Antal Ruhl Antal Ruhl is a media artist with a background in design, science and art. Antal studied Industrial Design at the The Hague University of Applied Sciences, Media Technology at Leiden University and Design and Media Arts at UCLA, Los Angeles. Antal creates objects that let us rethink our environment. These objects vary from kinetic sculptures to interactive installations. Sometimes they are purely conceptual or formal and sometimes they serve a more commercial purpose, but they always share the goal to intrigue people. “Using the technological possibilities at hand we can enrich our work and create an engaging object. Interactive and playful objects are much better in holding someone´s attention while they convey their message”. His work can be described as visually and technically attractive with a focus on natural and physical phenomena. Antal worked at distinguished design companies in Amsterdam and Barcelona, has had several (national and international) performances and exhibition, published a scientific paper, travelled the world and currently works a freelance artist/ designer. Antal has also started a company to develop interactive installation for brands, festivals and event. Creative in Motion is a creative brand activation studio: www.creativeinmotion.nl
presented paper, Experiments with Galvanic
For an overview of his work or to contact
Vestibular Stimulation in Daily Activities:
Antal please visit:
An Informal Look into its Suitability for HCI,
can be found here: www.antalruhl.com/media_tech/paper.pdf
Believability Edwin van der Heide
Believability is something we deal with on a con-
nevertheless draws us in without questioning its
tinuous basis. When we receive an e-mail that
realness or truthfulness. The story is believable,
claims to be from our bank with the request to
while not real, and we enjoy it. An important fac-
enter our account information on ‘their’ website
tor for a story to be believable is that we can
in order to upgrade its security, we enter a situ-
relate ourselves to it (or the story relates to us).
ation in which the content appears to be believable. We’ve learned to get suspicious and need to
The examples from the bank’s e-mail as well as
verify the credibility of the story. The e-mail is
the novel show us that believability is actually
believable, appears to be authentic, it pretends
independent from something being real or fake;
to be real, but is in fact fake. When we read a
or in other words, independent from something
novel, the story can be entirely fictional and it
being real or ‘virtual’. What we also learn from
Spatial Sounds (100dB at 100km/h) at Wood Street Galeries, Pittsburgh, USA, 2009 - Image courtesy Studio Edwin van der Heide
the example of the novel is that things that hap-
(2007) that fascinated me: The believability of a
pen in the story don’t have to be possible in real-
Robot. The reason that it fascinated me is that it
ity. In a novel we can meet creatures from Mars,
made me not only think about believability in the
time-travel, or never go to bed. We can ‘simply’
context of robots, but it also triggered me to think
imagine these things and believe them.
about the believability of an artwork and, more specifically, the believability of the behavior of
When I was writing my paper about the interactive
an interactive artwork (as opposed to the believ-
art installation Spatial Sounds (100dB at 100km/h)
ability of a robot’s behavior). I realized that the
for the Third International conference on Human-
believability (of the behavior) of an artwork was
Robot Personal Relationships (HRPR) I was intro-
not (yet) seen as a fundamental topic and might
duced to a topic addressed by Kerstin Dautenhahn
deserve its own study and experimentation.
Kerstin Dautenhahn has an interest in socially in-
reason that in Japan a lot of machines talk or in-
telligent robots: “A robot companion in a home
clude animations to, for example, welcome and/
environment needs to ‘do the right things’, i.e.
or thank the user for using it. It is interesting to
it has to be useful and perform tasks around the
question whether these machines indeed make us
house, but it also has to ‘do the things right’,
believe they show affection, and if so, if it lasts
i.e. in a manner that is believable and accept-
or wears off.
able to humans.” I tried to imagine an example and came up with the idea of a robot that makes
There is another possible difference between
and brings you a cup of coffee. Soon after I had
the coffee machine and the robot making coffee.
that idea I had to think about a full automatic
Robots are often made to look and behave like
coffee machine that grinds the beans and steams
humans (the humanoid) and the coffee machine
fresh milk for each individual cup of coffee that
isn’t. Does this then mean that the representa-
it makes. How do the two differ from each other?
tion is a requirement for believability? No, a ro-
With the robot we can imagine that he read our
bot (or interactive installation) doesn’t have to
mind and therefore made a coffee for us. We
represent something (else). It can be an abstract
imagine that the robot has a certain amount of
work that is believable on its own. In the HRPR
intelligence and, perhaps, even ‘feels’ affec-
paper about Spatial Sounds (100dB at 100km/h)
tion for us. In the case of the coffee machine,
I put it like this: “The installation can be seen as
we don’t imagine any intelligence and we simply
a non-verbal abstract robot and does not imitate
think of it as a machine without affection for us.
an animal or human-like look or behavior. It is a
If we, however, imagine that the coffee machine
machine-like object but does not resemble exist-
made that coffee especially for us things start
ing machines. Nevertheless, it allows us to some-
to change. It’s by imagining that we can change
how identify ourselves with it. Spatial Sounds
what we believe and thereby turn a machine into
(100dB at 100km/h) is an example of a believable
a believable affective robot. This might be the
robot in the sense that the visitors believe they
understand the behavior of the installation and
There are certain things we believe in that, once
find it worthwhile to interact with. The aspect
we discover they are fake, completely lose their
of believability is so strong that people accept
believability, while we keep on believing other
the installation as a real being and want to inter-
things that we know are fake.
act with it over and over” (van der Heide, 2011). Interesting to read in this context is Alwin de Rooij’s (2010) graduation research project for the
Media Technology Master program on ‘Abstract Affective Robotics’.
Dautenhahn, K. (2007). Socially intelligent robots: dimensions of human–robot interac-
I’m curious how the abovementioned thoughts
tion. Philosophical Transactions of the Royal
apply to augmented realities. What aspects make
Society B: Biological Sciences, 362(1480),
combinations of, and interactions between, the
real and the virtual in augmented reality believable? We have learned that something doesn’t
van der Heide, E. (2011). Spatial Sounds
have to be true to be believable. We have also
(100dB at 100km/h) in the Context of Human
learned that something doesn't have to be possi-
Robot Personal Relationships. In: Lamers M.
ble in order to be believable. Besides that, we’ve
H., Verbeek F. J., Human-Robot Personal
learned that something doesn’t have to represent
Relationships (HRPR 2010), LNICST Vol. 59,
something; it can be abstract and nevertheless
be believable. The former makes me believe that believability forms an interesting perspective to
de Rooij, A. (2010). Abstract Affective Ro-
think about what we can imagine in augmented
Spatial Sounds (100dB at 100km/h) at DAF Tokyo, Japan, 2006 — Image courtesy Studio Edwin van der Heide
Nonlinear stories told by cups and saucers Smart Replic as with responsive 3D audio Lotte de Reus, Jouke Verlinden, Maaike Roozenburg
copy of the original, but to analyse, communicate
In museum exhibitions historical objects are usu-
and enhance those qualities of the historical arte-
ally shown by visual display, in a showcase with
fact that are the most meaningful to us, now.
extra textual information added to it. Museum
Here we present the design project of Lotte de
visitors can never touch the objects, let alone
Reus in connection with this paradigm shift. Com-
use them. As a result, visitors ‘scan’ the displayed
pleted as a graduation project, it presents an au-
objects from a distance, something that needs
ditory environment to augment the artefact in an
reconsideration in our present time where the
unobtrusive and non-linear way. The objects that
‘experience’ is essential. To provide a way around
are central in this project are seven teacups and
this situation, the so-called ‘smart replica’ was
saucers that are part of the collection of Museum
proposed in the previous issue of Ar[t] (Roozen-
Boijmans Van Beuningen. These are currently on
burg, 2012): a new kind of reproduction, in the
display in a new exhibition on design and pre-in-
shape of a 3D print, that stretches the boundaries
dustrial design. Depicted in figure 1, each of these
of the replica’s concept as an autonomous object
teacup and saucer sets represents a milestone in
based on a historical artefact. New methods of ac-
the Dutch history of porcelain. Starting with the
cess and new digitization strategies based on the
first import of porcelain from China in the sev-
study of the relationship between the ‘bits’ and
enteenth century by the Dutch East Indies Com-
‘atoms’ are being developed. We hypothesize that
pany; followed by the invention of Delft’s blue as
by using these 3D imaging techniques the value of
an attempt to copy Chinese porcelain, and ending
our cultural heritage can be increased. In other
with the small scale production of porcelain in the
words, the goal is not to make the most realistic
active enrichment but not entertainment?
went on to the next; leaving the exhibition behind, full of untold stories. Our hypothesis is that more
Museums and their collections house a specific
active means will lead to a more comprehensive
group of objects with their corresponding historical
museum experience; thereby increasing the op-
data. They draw audiences in through exhibitions
portunity to reflect and learn even after the visit.
and related facilities, including museum shops,
An important part of the collection are the non-
restaurants and so forth. The artefacts shown and
linear narratives, networks of information as-
experienced in an exhibition construct a powerful
sociated with the object, consisting of stories,
medium which instils visitors with thoughts they
locations, materials, rituals, the collection’s past
perhaps have not yet fostered. Knowledge and
and the like. Augmented matter — the mixture
culture become more accessible because of the
between bits and atoms — allows novel interac-
actual presence of the objects. In the vision of the
tion techniques to embody these networks of in-
curator, paying a visit to an exhibition enables the
formation. For this design project our aim was to
visitor to have an intimate encounter with the ar-
convey the following qualities of interaction: in-
tefacts on display (van Dongen, 2012). These types
trigue, understanding, satisfaction, and integrity.
of encounters could correspond with respectively
These qualities feed the resulting research ques-
a museum or a theme park.
tions: How can museums anticipate and facili-
Figure 1. THE SEVEN CuPS AND SAuCERS picture by HuGO BES
In the spectrum of recreational activities, the ex-
tate the active assemblage of old and new sto-
hibitions of Boijmans Van Beuningen can be char-
ries – and how do these stories refer back to the
acterised as enriching experiences: an encounter
replica’s original? How can digital databases be
in which the visitor is conscious of the artefact
employed in linking smart replicas to their collec-
and the (hi)story it represents. This means that a
tions? On a philosophical level: does the original
museum requires the visitor to reach out for in-
still attract interest? In the case of pre-industrial
formation, while the information passively waits
utensils such as the teacups and saucers this
for the visitor act upon it. The engagement of
question is very relevant.
visitors is limited as they are not experts on the particular subject of the exhibition. Because of responding information, it takes effort on the part
concept of tangible interaction with responsive audio
of the visitors to maintain concentration. In the
For the seven teacups and saucers that have been
project discussed here, a focus group that visited
chosen to represent the history of the Dutch
the museum exactly proved this problem case in
quest for porcelain, the ECR-model is used to
hand. –Visitors’ overall interaction with Boijmans’
classify the valuable information (Wong, 2012).
exhibitions can mainly be described as scanning.
This storytelling model categorises the informa-
The participants walked into a room, started at a
tion in three layers: engagement, context and
random showcase, looked at an object briefly, and
reference, as shown in Table 1.
the passive character of the objects and their cor-
a. Ambient soundscape (audio).
characteristics Spatialised audio,
Draw the visitors’
depending on user
attention; the moment
location / viewing
when a visitor has some
angle. Triggered by
proximity to object.
emotional or intellectual response to the artefact.
b. Narration 1, information on the role (audio).
Close proximity to
c. Identification data (text, visual).
d. Narration 2 — information on specifics that can directly
object. Draw the visitors’ attention; the moment when a visitor has some immediate sensory, emotional or intellectual response to the artefact. Reference:
be related to the object (audio). e. Background information (text and images).
handling an object. Exhibition app,
Gives the visitor the
available after the
opportunity to draw
conclusions and connect to related resources. It provides more detailed and interpretive information about the work. Table 1. Proposed dialogue between visitor and exhibition.
In the first phase, visitors are attracted to par-
tor focuses on the features of the teacup or sau-
ticipate in the experience by other visitors who
cer (“audio specifics”). In this stage the visitor
are listening to the 3D audio clips. These audio
is encouraged to turn and explore the object,
clips give the impression that they are experi-
connecting the narrative with what is visualised
encing more: it is as if their auditory attention
about the object. The visitor can put down an
has doubled their visual attention (Erens, 2012).
object and pick up another object, while retain-
Together with an app that is made available, the
ing the soundscape.
visitor is intrigued and triggered to participate in
The app contains more background informa-
tion on porcelain and the objects. When the
When the visitor is in the proximity of an object,
visitor returns home, she can consult the app
a 3D ambient soundscape that fits in with the his-
at her convenience; to browse the additional
tory of the specific teacup and saucer will appear
information for example.
— once the first soundscape has been heard, it attracts the visitor to go to other objects accompanied by a spatial soundscape. Arriving in close
Enabling spatial audio
proximity of the teacup and saucer, the spatial
As discussed in Table 1, every teacup and saucer
audio of a narrator starts. The narrator tells about
holds a series of three audio clips. The audio clips
the role the specific objects (“audio story”).
are played when the visitor shows interest through
When the visitor picks up a replica, the narra-
their actions; proximity to an object and the han-
dling of an object. Auditory feedback enriches the
visitors’ experience actively, without visual clut-
The core of the concept was tested during the
ter and conserves traditional values on how art
Object design fair (Rotterdam, 7-10 February
should be experienced. Furthermore, audio clips
2013). Information about the teacups and saucers
allow temporal cues; it works more associatively,
was presented via text or audio clips. Ambient
and speaks directly to the visitor (Erens, 2012). The
soundscapes were on or off, yielding four permu-
soundscape is the auditory equivalent of an ambi-
tations. 140 people interacted with a selection of
ent image; non-visual immersive content, typically
the configurations presented in pairs, they were
with so-called earcons which represent specific
asked to choose between the two displays and
objects or events. The technology to spatialise au-
support their reasoning.
dio was developed two decades ago; see Burgess and Verlinden (1993) for example. The character of
The essential observations include:
3D audio relates to the idea that the sounds seem
Participants seem to enjoy the ambient
to come from sound sources placed anywhere in
soundscape: it triggers the imagination and
a space (a surround sound effect). When listening
the recognition that the objects used to be
through headphones, the brain places the sources
utensils and not art object as they are now.
of the audio clip in your head. An example of a high quality 3D audio clip is “In your head” by Big
Presenting the information via responsive au-
Orange. We used AudioStage to produce our audio
dio facilitated the visitors to consciously turn
clips with a visual interface, cf. Figure 2.
the object to find the image that the narrator
To connect this aspect to the exhibition visitor,
position and orientation tracking of the human head as well as the objects at hand is required.
Presenting the information via audio, whilst
Here optical or magnetic tracking principles also
the visitor is holding the object, does not
make sense, as they are fit for indoor use. With
clutter the visual sense.
the use of tangible replicas and 3D audio clips, scanning behaviour can be transformed into an
Participants appeared to be pleased with the
responsive auditory system, even those visitors who preferred text to audio clips.
Figure 2. SPATIAL AuDIO INTERFACE OF AuDIOSTAGE. 37
Conclusion Once the teacups and saucers were objects of daily-
will lead to a more meaningful experience. Secondly,
life and their form, weight, substance, texture, co-
understanding is nurtured because the encounter is
lour, decoration make sense primarily in the context
intuitive and the information presented by the arte-
of their functions and relations to other objects, as
fact responds to the visitor’s body language. Thirdly,
well as the people who used them. Combining ambi-
a satisfactorily quality is propelled, by the layered
ent soundscapes, tangible interaction with physical
structure of the narratives, which can be browsed in
replicas and the connection between information
a non-linear mode by the visitor. Through the app,
and corresponding visuals, triggers an immersive
the experience is saved, and can be connected to
encounter in which this sensibility is restored: the
various forms of social networking websites and
passive, one-sided encounter with the objects now
becomes an active two-sided encounter.
Lastly, the experience fits in with the integrity val-
The proposed system is by no means the first audi-
ues of the museum Boijmans Van Beuningen – the 3D
tory guide for exhibits; it rather extends the existing
scanned and printed, moulded physical replicas af-
strengths with emerging technologies such as indoor
ford what Dutch historian Johan Huizinga has called
tracking and spatialised soundscapes. Furthermore,
a “historical sensation”, the feeling as though you
it is more or less compatible with existing gear al-
are somehow “in touch” with the past (Ankersmit,
ready employed by many museums.
In essence this proposal presents a new type of rela-
Future work includes experimenting and research-
tionship between visitor and object that has interac-
ing the effect of the design in the environment of
tion qualities equal to a human conversation. Firstly,
this poses an intriguing quality that pulls the visitor in by using 3D audio clips and ambient soundscapes, creating curiosity and making visitors want to engage with the artefacts. Because the visitor has a direct intellectual sensory dialogue with the object, this
1. Experimentation with augmenting untouchable artwork with ambient soundscapes. 2. Implementation of responsive audio tours in the current exhibition context.
Figure 3. Impression of the preliminary evaluation.
3. Prototyping indoor tracking and interaction sensing possibilities, with special attention to
More information ¢¢
phones required): http://www.youtube.com/
4. Creating guidelines on how the concept could be implemented to suit different kinds of ob-
jects in the museum (or maybe even utensils in
by Cilia Erens, Professor Joris Dik, Dr. Wolf Song and
In your head; a high quality 3D audio example: http://www.big-orange.nl
constructive collaboration. Furthermore, we would like to acknowledge the valuable advice and support
Interactive installations regarding preindustrial utensils: http://new.pentagram.
We would like to thank Alexandra van Dongen, curator at the Museum Boijmans Van Beuningen for her
Software to render binaural output by visually placing audioclips in 3D: http://www.longcat.
our everyday context).
Video presentation of this project (head-
Official Smart Replica blog: http://smartreplicas.blogspot.nl
Ir. Wim Verwaal. Many thanks to DOEN foundation for funding the Smart Replicas project (including the graduation project of Lotte de Reus) and many thanks to Mareco prototyping for their contribution in the form of 3D micro prints.
Ankersmit, F.R. (2005). Sublime historical experience, Stanford University Press.
Burgess, D. A., Verlinden, J.C. (1993). An architecture for spatial audio servers. Proceedings of Virtual Reality Systems Conference (Fall ‘93).
Erens, C. (2012). Interview on soundscapes, Interviewed by Lotte de Reus. Amsterdam, 14 December.
Halbertsma, M. (1995). Themaparken, dierentuinen en musea. Rotterdam: Erasmus Centrum voor Kunst- en Cultuurwetenschappen.
Ministerie van Onderwijs, Cultuur en Wetenschap (2005). Bewaren om teweeg te brengen. Museale strategie, Woerden: Drukkerij Zuidam.
The Futures Channel, (1999). Conversation with Curtis Wong. [online] Available at: http:// www.thefutureschannel.com/conversations_ archive/wong_conversation.php
Van Dongen, A., (2012). Interview on the mu-
Lotte de Reus Lotte de Reus recently received her Master’s degree in Design for Interaction at the Faculty of Industrial Design Engineering at Delft University of Technology. In July 2012 she started her graduation project for the Smart Replicas project; she was driven by a fascination for porcelain and the wish to create effective storytelling experiences. During the project she was pleasantly surprised by the world of audio, something she had not yet encountered in her studies. New technologies such as augmented audio, immersive soundscapes and 3D audio now hold a new, special interest for her. In the future, Lotte would again like to work in the domains that combine art and technology. Her portfolio can be found at www.lottedereus.nl.
seum context, Interviewed by Lotte de Reus. Rotterdam, 21 November 2012.
Image: Revealing Identity, Guido Lagerweij
Who owns the space 2 discussing legal issues of AR Yolande Kolstee
WHAT IS PRESENT IN AR SPACE? As mentioned in ‘Who owns the space’ in AR[t] 2, we keep a close look on the developments concerning the invasion of our physical space with virtual information, and discuss some legal aspects thereof, using information from, amongst other sources, the blog Wassom.com. In AR[t] 2, I contended that we as passers-by might not be aware of any virtual content at all and, when we don’t have the right app (software) and/or mobile (hardware), we don’t have the means with which to recognize the virtual content floating in the air around us. Although, in theory, everybody has the same access to virtual content, de facto, this is not the case. For some this might be a bit frustrating, giving off the impression as if people are gossiping about them in a different language or at a very low tone; unsure whether or not they are they are talking about you… Sander Veenhof, Dutch AR artist, made a lot of people aware that AR could be everywhere, and actually is everywhere and that this is a special characteristic of AR for as a new medium for artists- and artistic purposes. For many people, seeing his art work Biggar and playing with it for the first time, helps them to realize that this AR art piece encapsulates the entire earth with its 7.463.185.678 virtual cubes whose colours can be changed by simply tapping on a mobile phone. The only thing you need to do
to actually see Biggar, is to download this app via
What is at stake is, what happens when this in-
Layar. This virtual sculpture has true omnipres-
formation about our eye-movements is stored in
ence; it floats above holy places in Rome or in
databases, analysed and even sold to marketing
the Himalayas, as well as above industrial places,
above warzones and above peaceful places… Brian Wassom, blogger and specialist in AR Law, predicted five issues related to AR Law for 2012,
What is recorded ?
ranging from public resistance to ‘adult’ AR (AR
With the announced introduction of Google glass,
porn that is) via licensing, and negligence, to AR
we see a concern regarding the possibility to re-
cord without the public knowing this.
“Stop the Cyborgs!”
On March 11,2013, Seattle’s 5 Point Cafe became the first
ban Google Glass, the highly anticipated augment-
Mr Wassom recognized he was not right in all his
ed reality device set to be released later this year.
predictions, scoring 1,5 out of 5, but for him it was
The “No Glass” logo that the café published on its
more a matter of time than a matter of substan-
website http://the5pointcafe.com was developed
tial incorrect prediction. However, in February
and released (under a Creative Commons license)
2013 an adult magazine has distributed around
by a new London-based group called “Stop the Cy-
1,000 billboards with Augmented Reality markers
borgs.” The group is composed of three young Lon-
throughout many cities in the Netherlands. When
doners who decided to make a public case against
downloaded, the app distributed via Layar works
Google Glass and other similar devices.
by pointing your smartphone at one of these post-
On the Stop the Cyborgs site, the group raises a
ers, and the girls will come to life allowing you to
significant concern: namely, that there’s no obvi-
watch them in a video right there on the street.
ous way to know when the device is on or what
This adult content will certainly appeal to a lot
it’s actually doing (recording or not).
of people and this will definitely result in another large group of people coming into contact with AR. Since most of the public are not familiar with
TWO WAY TRAFFIC
AR marketing activities like this, it will have a
For 2013, again, some predictions were made
significant informative impact.
and trends were spotted. Concerns raised range from on the one hand, inserting a lot of informa-
tion from the physical world, maybe voluntarily via players of AR games, into the virtual world,
Another type of warning from Mr Wassom is relat-
and, on the other hand, entering information
ed to the information derived from tracking our
about the physical world (‘burglary information’
eye-movements to detect were we are looking at.
as we called this in the last issue of AR[t]) into
Real immersive AR depends on knowing exactly
the virtual world, without any prior screening or
what our eyes are looking at; this information is
approval from an examination committee.
used to position the virtual content there where our eyes are looking at. Natural feature tracking systems are in rapid development; we can enter
Prevalence of AR
a room, let our camera survey the space, detect
John Moe (the host of Marketplace Tech Report and
the salient points, and make a virtual grid on
handles web content for the program), wrote in
which we position our virtual objects or scenes.
2011 that “Augmented Reality has been the Next
Using the eye-movements of the person wearing
Big Thing for a while now, although it never man-
AR glasses gives extra accuracy.
ages to become the Actual Current Big Thing”.
Images courtesy of GoldRun, see www.goldrunner2013.com and www.snapsapp.com
This quote gives us a direction for discussing the
We might come to the preliminary insight that
abovementioned AR related legal issues. The
as for now, our actual privacy laws are sufficient
intertwinement of AR in our day-to-day lives is
enough to deal with privacy concerns in aug-
actually quite slow — especially compared to the
mented reality. An incorrect accusation or incor-
revolutionary predictions. We are getting used to
rect information remains incorrect, regardless
it at a nice pace. As an example, I can take our AR
whether it is posted on Twitter, Facebook or in a
Lab, which has been working in the augmented
space around us, via an AR app.
reality field since 2006, and still for some, even
To add correct or incorrect virtual information to
closely related people within our Academy, the
a virtual object or a space, is as simple as assign-
concept of AR is completely new.
ing correct or incorrect information to any sub-
“But if the Google glass will become prevalent”
ject, object or space in our physical world, and
as one the people from ‘Stop the Cyborg’ argues
will be considered as such.
in a conversation with a journalist of arstechnica.
For the time being, we don’t need special AR
com “and suddenly everyone is wearing it and
criminal legislation, our current laws might be
this becomes as prevalent as smartphones—you
can see it becomes very intrusive very quickly. It’s not about the tech, it’s about the social culture around it”.
Further Reading Critical blog on law and social media:
FALSE STAYS FALSE Recording without the recorded person know-
http://wassom.com Critical site on wearable technology: http://stopthecyborgs.org
ing this, is possible with our smartphones and is widely spread among youtube videos. However, we have privacy laws to handle this- and if those laws are not adequate enough, we have to change those laws. 43 43
OLIVER PERCIVALL, The Augmented Star, Book Cover, Illustrator: Paul Hadcock
The Misadventures in Augmented Reality OLIVER PERCIVALL
I FIRST LEARNED ABOuT AuGMENTED REALITY AROuND TWO YEARS AGO. I DISCOVERED ITS MAIN uSES WERE AS A NAVIGATION TOOL uSED IN CITY CENTERS, POINTING PEOPLE FOR EXAMPLE TO PLACES OF INTEREST OR NEAREST TuBE STATIONS. THE THOuGHT OF CONCEALED TEXTuAL MARKERS STRuCK A STORY IDEA IN MY MIND. WHAT IF THERE WAS SOME AuGMENTED REALITY TEXT MOST OF uS WERE NOT SuPPOSED TO SEE AND YOu HAD A DEVICE THAT WAS ABLE TO READ IT… WOuLD YOu FOLLOW ITS INSTRuCTIONS? THIS IDEA FASCINATED ME INTO WRITING A SERIES OF STORIES TITLED THE MISADVENTuRES IN AuGMENTED REALITY. PART ONE FOLLOWS OuR ACCIDENTAL EXPLORER LYNDEN AS HE POSTS HIS ADVENTuRES TO AN ONLINE FORuM. 45
Part One My uncle was one clever soul. He could probably be best described as a recluse, the shy inventor type. But nearly nine years ago now he disappeared from his hometown and has never been seen since. I don’t think he’s lost, not wanting to be found is my guess and I think I know why. Using his ingenious engineering skills he somehow created a VAARR (Very Advanced Augmented Reality Reader) It works just the same as other regular Augmented Reality apps I am led to believe but with one unique difference… It has a setting that once activated, shows unseen Augmented Reality text none of which is detected by any other device. He created two actually. One was entrusted to me. And because of this I have a story to tell. A few years ago I wasn’t exactly a technology fanatic although admittedly I had more than a passing interest in new gadgets. But that was nearly three years ago now before I set foot within the Augmented Star. Now there is probably nothing I don’t know about
Images courtesy of GoldRun, see www.goldrunner2013.com and www.snapsapp.com
the integrated software algorithms used in Augmented Reality. Life in this newly discovered domain is a far cry from home. There’s no one thing that makes it uniquely different. It’s everything. The smell, the temperature, the noise levels, the beings, everything is entirely unfamiliar when you first arrive. But once you start exploring all the hidden signs and directions it’s impossible to stop. You’re immediately plunged into its rich diverse landscapes and to find your way around or locate anything at all you must be in possession of the special device — The VAARR or Very Advanced Augmented Reality Reader. But be warned: Augmented Reality text as we know it was not developed within The Augmented Star so it makes it pretty hard to find your way out. It’s not easy to write long paragraphs on this VAARR. My uncle made it possible to post directly online from this device. So my blog entries below are concise.
The Ultimate Frequency Forum This forum was created for anyone to post to that feels lost, those that do not wish to be found or find themselves beyond the confines of earthly boundaries.
Blog Entry #1
Apr 12 2011
‘It’s not about gazing into the future, it’s about seeing the present through a unique lens.’ That’s the first message you see when you arrive in The Augmented Star. Every time. And you never arrive at the same place twice. I’ve come here with a strict agenda. To find my uncle.
Blog Entry #2
Jul 12 2011
OK it took me a couple of hours to get acquainted here this time. There’s just so much to get distracted by. From learning a new food recipe to breathtaking advertising campaigns, the Augmented Reality here really is something else. On the downside crossing the street can be something of a perilous activity. It puts things in perspective when vehicles move from 0 to 600 MPH in less time than it takes to chalk a pool cue.
Blog Entry #3
Oct 12 2011
Just noticed a glitch with this online forum that I would like to point out to the moderators. The first post I made earlier was date stamped Apr 12 2011 but the post I made just a couple of hours after that is stamped July 12 2011. Three months later? I think not!
Blog Entry #4
Jan 12 2012
It’s funny because since I’ve been wandering the streets here the last few hours I’ve regretted never going to Cambridge University. Although I feel like I have been. It appears there is nothing you can’t learn here by waving the VAARR around. The whole place is an encyclopedia of knowledge. They promote learning a Science degree in Artificial Intelligence can be completed in three months here. Knowledge is ﬂuidly brought to astonishing life everywhere by Augmented Reality. I just learned that a thimbleful of a neutron star would weigh over 100 million tons. Must be something in the air here as my clothes are becoming dirty and ragged really quickly. I feel I’m ready for that science degree now!
o. l. Percivall i have always had an interest in technology and gadgets. after leaving school i took a basic computer studies diploma, which eventually led to a career in various i.t support roles leading up to Project Management. the mysterious side of technology especially intrigued me including the possibilities of where it could take us in ‘what if’ scenarios. so equipped with a reasonable understanding of augmented reality and enjoying a challenge, i set myself the complex task of plotting my science fiction novel and created an alternative fictional world that became the augmented star. the augmented star is now available on the http://tiny.cc/augmented-star
amazon kindle store, jsut follow the Qr code on the left!
Blog Entry #5
Apr 12 2012
The risk of being struck by a falling meteorite for a human is one occurrence every 9300 years. Wow that’s a curious fact. I wonder what the odds are for walking into an Augmented Universe like this one. I saw an advert on a billboard for an automobile just now. Using my VAARR triggered a full Augmented Reality breakdown of its features from performance figures to finance options and then it invited you to take a virtual test drive in a car simulation game. The AR campaigns here really resonate with customers in a way that most other ad platforms fall miserably short.
Jul 12 2012
Turns out my VAARR is a pretty valuable commodity here. It can read over 150 types of Augmented Reality text and other types too that alternative devices can’t. After an exhausting and bloody battle earlier today, I have learned some creatures here would even kill to own one.
Blog Entry #7
Oct 12 2012
Just received some news that was pretty hard to comprehend. I’ve met a lot of people here within The Augmented Star and most of them know my uncle but apparently he’s on the run from some bad people. Someone is coming. I’ve gotta move quickly. It feels like I’m on the run myself. I’ve come to realize now that I may never be able to leave the Augmented Star.
End of part one
Images COuRTESY OF GOLDRuN, SEE WWW.GOLDRuNNER2013.COM AND WWW.SNAPSAPP.COM
Blog Entry #6
Augmented modelling in the design studio:
augmented pedagogies Alejandro Veliz Reyes University of Salford
Digital physicality / Physical digitality From very early stages of the design process
Digital physicality, Physical digitality. One of the
(sketches and diagrams) up to detailing and
main grips in which this topic is grounded is that
construction, models are built by using several
the growing academic concern on the dialogs be-
different — yet interrelated techniques. The
tween physical and digital realms in architecture
construction of a physical model by computer-
is becoming blurry, since the distinction between
controlled machinery requires, unavoidably, a
both cannot be fully depicted in a world where
digital counterpart. Likewise, scanning and re-
digital information is continuously embedded
verse engineering methods allow a fluent interac-
into ‘real world’ situations, such as ubiquitous
tion between models which embeds information
computing systems or environmentally responsive
and knowledge as the design process proceeds. As
technologies. This recent conference had, so far,
a result, a complex orchestration of users, devel-
the biggest amount of published articles in the
opers, tools and techniques merge into tailored
history of the eCAADe Association Conference,
modelling flow works according to each project’s
and further evidence of this trend is the grow-
requirements. This design-technology synchronic
ing participation of architects and designers in
co-evolution has been historically aligned with
professional events such as Ars Electronica, SIG-
the zeitgeist of the architectural practice, in
GRAPH or the ISMAR community.
which new technological developments shape the way we express ourselves, and define new design
This work claims that beyond physical or digital
forms and organizational/social schemes.
modelling techniques, augmented reality now
...augmented reality now gives us the chance to build hybrid, augmented models.
gives us the chance to build hybrid, augmented models. Augmented models are those which blend a physical and a digital counterpart in a resulting synchronic manner, hence taking advantage of the benefits of physical modelling methods (dexterity skills, manipulability) as well as digital (accuracy,
Indeed, this section is titled with the topic of the
modifiability, among others). Also, in addition to
latest conference of the Association of Education
the technical definition of augmented reality, the
and Research on Computer Aided Architectural
concept of augmentation has a major role in this
Design in Europe (eCAADe) held at the Czech
work. Even if the semantics of augmentation sug-
Technical University in Prague (September 2012):
gests an ‘increment on size or amount’, it also
traiNiNg sessioN oN augMeNted reality autHoriNg aNd augMeNted ModelliNg, scHool of tHe built eNviroNMeNt, uNiversity of salford.
suggests some enhancement. As a result, this ap-
diagrams and sketches. This scenario might not be
proach to the augmentation of reality fits with
unknown for any architecture student, since the
the major aim of educational research which is to
studio teaching scheme has been largely acknowl-
enhance and improve educational processes and
edged as the core practice-based module in which
methods, thus naming this work as ‘Augmented
both design (composition, planning, representing)
and high-order cognitive skills (critical thinking, analysis, synthesis) are mainly developed in archi-
MODELS IN THE DESIGN STuDIO
tectural education programs around the world. Actually, the architecture studio and its interactions as a subject matter is a quite complex challenge.
“It looks better like this”, says an architectural
As stated by Allen Cunnigham in 2005, after a cen-
design instructor while chopping a piece of card-
tennial adaptation and evolution the studio teach-
board off a building model. In the same design crit
ing scheme and “project-based education around
session, students, instructors and observers engage
architecture employing the studio system is the
in a dialog based on the use of plans and sections,
most advanced method of teaching complex prob-
rendered views of a project, physical mock-ups,
lem solving that exists”.
The usefulness of models within the design stu-
from a scientific research perspective, that is fol-
dio is clear. Beyond the fact that its construction
lowing the guiding principles of generalizability,
itself entails the development of technical skills,
communicability and transferability of that re-
models also embed design information and knowl-
edge, affecting organizational dynamics (the design critique or peer to peer collaborative work),
The deep impact of new digital tools in design
the creation of students’ ‘toolkits’ or the final
pedagogy has been explored recently by design
presentation of the design solutions, among other
theorists, such as Dr Rivka Oxman. The particu-
benefits. During this research being conducted
larity of the design studio as a research setting is
at the university of Salford (uK), the extent of
spiced by theoretical underpins that can poten-
the impact of augmented models in this complex
tially lead the path to depict this impact. For ex-
studio-system is yet to be depicted. Augmented
ample, it has been stated that the studio teach-
models will be used, therefore, as a way to un-
ing is usually an unstructured process, in which
derstand how new technologies impact design
perceptions and interpretations of information
education and how can we describe that impact
and models play a major role in the students’
draft versioN of aN augMeNted Model for iNterior desigN. furNiture caN be re-arraNged iN a PHysical Model of a House rooM (studeNts’ work).
progression in the courses, mostly based on ‘design dialogs’ between students, and students and instructors. Also, digital tools have the potential to not only re-shape the toolkits being used for design, but also mediate in the way design methods are structured, offers new ‘digital materials’ to work with or changes the very nature of the design problems to be faced in different courses. It is not clear, however, how this impacts occurs. The interactions within the studio that make use of representations and models to design are well established ‘rituals’ such as peer-to-peer collaborative activities or the design critique, but the nature of each studio differs from each other. Variables such as the experience of the instructors, the background of the students, the nature of the design problems to be faced or the institutional standpoint turn the studio into highly context-dependant modules. As a result those variables are usually highly controlled and the study of the impact of different technologies is commonly constrained to the description of technical challenges to be solved, the development of new systems/software or metrics of student satisfaction, rather than on the provision of a theoretical account of their impact into this complex teaching/learning process. The lack of a theory that describes how technology re-shapes the studio results in very limited knowledge re-usability and in turn, into very caged and localized pedagogical frameworks that do not allow cross-institutional or cross-disciplinary collaboration, to evaluate the constant infusion and evolution of new digital tools for educational purposes or to re-use a pedagogical approach and its associated knowledge.
A THEORETICAL APPROACH TOWARDS AuGMENTED PEDAGOGIES
IMAGES OF THE TRAINING SESSIONS ON AuGMENTED REALITY AuTHORING AND AuGMENTED MODELLING. STuDENTS OF THE PROGRAMS MSC DIGITAL ARCHITECTuRAL DESIGN AND MSC BuILDING INFORMATION MODELLING AND INTE-
Indeed, there is not fixed methodology to study
GRATED DESIGN, SCHOOL OF THE BuILT ENVIRONMENT,
the impact of digital technologies in design
uNIVERSITY OF SALFORD.
education. Rather, each study requires an own
standpoint in terms of validity and fitness to
scribe the resulting studio dynamics by using aug-
the research problem and the subject matter.
mented models. Several trainings on augmented
As Wanda Orlikowski and Suzanne Iacono (2001)
reality and augmented modelling have been
state on their work on information systems the-
made at the university of Salford (MSc Digital
ory research, this corresponds to the fact that
Architectural Design, MSc in Building Information
the use of technologies depend on the context
Modelling and Integrated Design), and two more
and hence, “there is no single, one-size-fits-all
experimental settings are now being arranged in
conceptualization of technology that will work
different European countries. These multiple set-
for all studies. As a result, IS researchers need
tings are not only intended to provide a wide view
to develop the theoretical apparatus that is ap-
of the subject being studied, but also fits with the
propriate for their particular types of investiga-
current recommendations for theory construc-
tions, given their questions, focus, methodology,
tion methodologies, since the manipulation and
and units of analysis.”
observation of data in many divergent ways and the juxtaposition of different conflicting realities
In order to overcome this challenge, this ongo-
and sources counteracts the tendency of reach-
ing research proposes a theoretical approach to
ing false or incomplete results, or information-
depict the impact of augmented models in design
processing biases of the investigator.
education. By following a grounded theory methodology, observations and recordings are being
This work is expected to be finished by end-2014.
collected in diverse settings on an attempt to de-
VIEW OF THE STuDENTS’ TOOLKIT DuRING THE FIRST AR WORKSHOP AT THE uNIVERSITY OF SALFORD.
Related literature and sources ¢¢
Achten, H., Pavlicek, J., Hulin, J., Matejdan, D. (Eds.). (2012). Digital Physicality / Physical Digitality - Proceedings of the 30th eCAADe Conference - Volume 1 / 2. Czech Technical University in Prague, Faculty of Architecture (Czech Republic) 12-14 September 2012.
Buechner, J. (2011). Fictional Entities
Alejandro Veliz Reyes
and Augmented Reality: A Metaphysical Impossibility Result. Journal of Evolution &
I finished my Architecture degree in 2007,
Technology, 22(Minds and Machines Special
in Chile. Today I am a teaching assistant and
PhD student on digital architectural design at the University of Salford (United Kingdom)
Cunningham, A. (2005). Notes on educa-
and my current research interests are design
tion and research around architecture. The
pedagogy, augmented reality, and collabora-
Journal of Architecture, 10(4), 415-441.
tive technologies for design.
Eisenhardt, K. M. (1989). Building theories
I have been involved in research activities
from case study research. Academy of man-
since 2007, but my background is quite mixed
agement review, 14(4), 532-550.
and has involved the domains of sustainable design, buildings in extreme/cold environ-
Orlikowski, W. J., & Iacono, C. S. (2001).
ments, mass customization and digital de-
Research commentary: Desperately seeking
sign. My published work has led me to pres-
the "IT" in IT research—a call to theoriz-
ent on conferences and invited lectures in
ing the it artifact. Information systems
Chile, United Kingdom, Italy, Belgium, Slove-
research, 12(2), 121-134.
nia, Czech Republic, Canada, among others.
Oxman, R. (2008). Digital architecture as a challenge for design pedagogy: theory, knowledge, models and medium. Design Studies, 29(2), 99-120.
SCARLET Project ÂŠ University of Manchester
a study iN scarlet MATT RAMIREZ
MIMAS - uNIVERSITY OF MANCHESTER
Learning and Teaching Co-ordinator
Curator, Crafts Study Centre
Mimas, The university of Manchester
university for the Creative Arts
Special Collections Supervisor
Educational Technologist, Crafts Study Centre/
university of Sussex
Research Associate, Visual Arts Data Service university for the Creative Arts
introduction Augmented Reality (AR) was identified in the 2011
established learning methods and styles. If the
Horizon Report1 as a key technology trend with po-
focus for the student is the technology, the learn-
tential impact on education. The report provided
ing experience can be diluted, this can inevitably
the catalyst for the SCARLET (Special Collections
lead to dissatisfaction and resistance.
using Augmented Reality to Enhance Learning and Teaching) project, as a way of leveraging innovative technology with pedagogical processes. Whenever the words â€œtechnical innovationâ€? is spoken in education circles, educators are under-
The benefits of using augmented reality in education
standably cautious electing to concentrate, perhaps rightly, on deep rooted pedagogical benefits
The benefits to student learning should always be
rather than short-term gimmickry. Many observ-
central to the introduction of any new technology
ers have already buried AR as a fleeting fad in
and AR is no different. It is always useful when
education due to its lack of use cases and docu-
dealing with new methods of delivery, to be armed
mented impact studies.
with a long list of tangible gains for adopting the technology. Some of the most persuasive argu-
After all, in education technology should be
ments are described below:
transparent and not an overpowering driver, particularly where the emphasis is on the teach-
Students can view and touch real manuscripts/
ing material. In addition, users do not want to
editions in conjunction with guided support
spend time adapting to a new way of learning,
from trusted sources, supporting independent
new technology should integrate seamlessly into
Layering AR on texts/images can encourage
rare books within the controlled conditions of
interaction (e.g. augmented 3D models that
reading rooms, isolated from much of the second-
overlay the physical image and require user
ary supporting materials and a growing mass of
touch gestures to proceed) and spark enthusi-
related digital assets. Students are used to having
asm, preparing them for solo research.
access to electronic information on demand, so this experience can be foreign and a barrier to
AR promotes ‘active’ teaching, maximizing
their use of special collections.
the opportunity for interaction, encouraging critical response and the adoption of new per-
The SCARLET project, while embracing the po-
spectives and positions. This is in opposition to
tential of AR, concentrated on delivering the
traditional didactic methods that are predomi-
benefits to student learning without being a flag
nantly teacher led.
bearer for the technology. SCARLET was led by the Learning and Teaching team at Mimas2, a na-
Users retain a very small amount of the infor-
tional centre of expertise at The University of
mation that is delivered, and a slightly larger
Manchester. A mixed team was pulled together
percentage of what is shown to them, but when
dedicated to enhancing the student experience
we become actively involved in an experience,
through the application of technology including
learners will remember and retain the majority
librarians, academics, learning technologists and
of the information presented to them.
AR can harness both asynchronous (emailing
Sources for primary content were ten key edi-
tutor questions) and synchronous (discussion
tions of ‘The Divine Comedy’ by Dante (between
with peers) e-learning methods.
1472 and 1555) particularly important in terms of publishing and intellectual history and the
Abstract concepts or ideas that might otherwise be difficult for students to comprehend
world-renowned oldest fragment of the Gospel of John.
can be presented through an enhanced learning environment offering access to source his-
At the start of the project in 2011, Junaio was
torical artefacts and online research in situ.
the only AR browser to harness optical tracking functionality, linking 3D models, videos and in-
The learning curve for new users engaging
formation to images in the form of “GLUE” based
with mobile AR through browsers is relatively
channels. This coupled with an open API and
shallow enabling the learning/pedagogy to be
compatibility on Android, iOS and Nokia devices
the driver, not the technology.
would prove decisive in the decision to use Junaio. By implementing an object based AR experience, students could simultaneously experience
SCARLET (June 2011- May 2012)
the magic of original primary materials, such as an early printed book in the library, whilst enhancing the learning experience by surrounding the book with digitised content; 3D models, im-
The main challenge with the use of special col-
ages, translations, and information on related
lections in learning and teaching is that students
are required to consult archives, manuscripts and
evaluation A dominant theme that became evident in the evaluation was that the two academics found differing responses dependent on student user groups. Students who had little prior subject knowledge found the app most useful, providing a foundation for further investigation and research. The learning experience was most enhanced by AR when information delivered was FRAGMENT OF THE GOSPEL OF JOHN AR CONTENT
contextual and less generic. Simply adding exist-
© MIMAS, uNIVERSITY OF MANCHESTER.
ing web assets to an object is insufficient; making them unique and packaged in digestible chunks produces more positive feedback and value. In addition, student feedback noted that AR experiences should be best used as part of a learning activity (either independent or group based) acting as enabler to achieving a key course objective (e.g. planning for essay or presentation). AR was most successful layered over the printed marker instead of signposting to other web based resources already accessible using traditional teaching scaffolds (e.g. CMS). Throughout the project lifespan, student evaluation was critical,
DANTE AR CONTENT
directly feeding into the application using an it-
© MIMAS, uNIVERSITY OF MANCHESTER.
erative development approach.
The team went on to win the 2nd prize for the
scarlet+ (june 2012 – april 2013)
ALT learning technologist of the year team award3 and won the Innovation in HE award at Education Innovation 20134. Emphasising the need to align technology to teaching and learn-
Following the success of the SCARLET project,
ing objectives was paramount from the outset
further funding was made available to work with
to maximise student benefit and impact. Further
two other uK institutions to facilitate and guide
funding was made available through the SCAR-
them in the development of AR content. It would
LET+ project whose primary focus was to apply
also liaise with staff at these universities to
the process and framework to other institutions
transfer the skills learnt and hone the toolkit that
special collections embedding the methodology
was used as a high level framework for replicating
using a toolkit5 (university of Sussex and the
similar AR practices in education.
Craft Study Centre at the university for the Creative Arts).
ar visualisatioN of ukMaP data ÂŠ MiMas, uNiversity of MaNcHester.
The partner universities involved were the Spe-
which will allow students to access surrounding
cial Collections at the university of Sussex and
resources, layering anatomical information and
the Crafts Study Centre at the university for the
reinforcing learning with instructional demon-
Creative Arts7. focusing on content from 1980sâ€™
strations (e.g. cannula application).
mass observation and 20th century crafts. In the area of geo-spatial mapping, Mimas has This project has both developed Mimasâ€™ under-
collaborated with colleagues from the Landmap9
standing of implementing AR in education and
service to create an AR experience around the
embedding best practice and methodology to
uKMap dataset. This provides a wealth of rich,
other institutions. Crucial to the success was en-
multi-layered information accurately locating
suring that, as with SCARLET, a multi-disciplinary
building types, building heights and ground usage
team approach was adopted. This ensured that
to name a few. The challenge was to incorpo-
the content developed made an impact on learn-
rate this tabular data in a visual 3D model that a
ing and teaching as well as enabling AR skills to
handheld device could render efficiently. It dem-
spread across the institutions.
onstrates a visual representation of raw materials that are often extremely large in size and difficult to comprehend.
working with other subject areas In parallel with SCARLET and SCARLET+, Mimas
have worked with the Manchester Medical School8
To conclude, the two projects have presented im-
helping students to understand pattern recogni-
portant findings in the impact of AR in education,
tion, a crucial part in decision making processes.
delivering a suite of rich materials especially
There is vast potential in this field enabling stu-
given the small amount of funding that was avail-
dents to examine x-rays/biopsies, using AR to
able. Both succeeded in providing a showcase for
identify abnormalities. In conjunction with this,
the Special Collections held at their respective
early development has started using 3D object
institutions using AR, bringing their static objects
recognition in the clinical skills laboratories,
to life. Sharing was a key element at the heart
of SCARLET and SCARLET+, this coupled with a strong team ethic enabling stakeholders to buy into the long-term vision. It is hoped that the legacy from these small projects will be to inspire others to undertake similar work and display the student led benefits AR can offer. AR opens up huge possibilities for creating immersive learning activities. It is particularly effective in explaining abstract concepts visually; allowing active based learners to better absorb the transfer of knowledge. While it may not be suitable for all students and situations, when em-
ployed well it can capture the attention like few
Matt ramirez is currently working on the
other technological mediums.
technical development and support of the jisc funded scarlet+ augmented
reality project, this follows on from
his involvement in the award winning
scarlet project. He has over 15 years experience in web design and e-learning content development for a variety of
2011 HORIZON report, section on
subject areas including medicine, it,
augmented reality: http://wp.nmc.org/
science, special collections
and business. these have used multi-
media content authoring tools such as
SCARLET team are joint second in
Flex, Flash, Blender, Unity to name a
learning and teaching awards, 13th
few. Matt's role is also concerned with
September 2012: http://teamscarlet.
the research and development of new
technologies (e.g. ibook/mobile/multi-
media development projects and haptics)
with the Manchester Medical and dental
Education Innovation 2013:
schools. this aims to improve the stu-
dent experience by embracing innova-
SCARLET Toolkit: http://scarlet.mimas.
tive learning methods and providing cut-
ting edge support materials. read more
Special Collections at the university of
about Mattâ€™s work at http://teamscarlet.
wordpress.com and @team_scarlet.
4. 5. 6.
Crafts Study Centre at the university for the Creative Arts: http://www.csc.ucreative.ac.uk
Manchester Medical School: http://www.mms.manchester.ac.uk
Landmap: http://landmap.ac.uk 61
iNtroduciNg ar to first year graPHic desigN studeNts YOLANDE KOLSTEE
graphic design students learn how to work
we, as ar lab are thoroughly convinced that it
with the latest adobe suit, Photoshop and web-
is important to let first-year students, freshmen
site development tools. Most important, how-
and -women, experience augmented reality,
ever, are the courses in which concept develop-
because it will change aspects of their discipline
ment forms the main part. whether a concept is
as well as their ‘metier’.
devised with colour pencils, felt pens or a col-
at various other art academies we see similar
lage of other material is not the most important
projects, one of which is the scarlet (special
issue at that moment. what is important for the
collections using augmented reality to enhance
academy, and the department involved, is that
learning and teaching) project, in the uk, at
they want the students to become acquainted
the university for the creative arts.
with creative or artistic research. However, the
at the beginning of the royal academy of arts’
majority of mainstream teachers place analogue
current study year, we made a head start with
and digital techniques at the same level when
the introduction of ar via projects differing in
developing concepts; while, in effect, they of-
length and with different student groups.
ten have a (underlying) preference for pen and ink based handwork.
the idea was to work towards a special PopUp Gallery, meaning a short, one-day exhibition
when working with augmented reality — a
of the results, in cooperation with the studium
very suitable medium for graphic designers, by
generale programme, a programme covering
the way — new elements should be considered
lectures throughout the year.
in the concept and design phase. with augment-
two graphic design teachers, both young
ed reality new, very new, aspects are added to
and digitally savvy, started to work with their
the world of graphics regarding the internet and
classes of 27 students each (!) together with
social media, while also taking co-authorship
some third year students as well. the search to
and performance into account.
get to know what ar is and how it could be a
meaningful technique for graphic designers be-
to work as a fully functioning app. We showed
gan comprised two steps. The first was a very
them our AR projects and introduced them to
active YouTube-Google search, and the second
e.g. Aurasma — a simple and elegant tool for be-
step was to learn to play with the idea of adding
ginners in the AR app world.
extra information — both analogue and/or digi-
We were lucky to have two lecturers who gave
tal — a concept development trajectory of sorts,
their talk on AR as part of the Studium Generale
so to say. The end-results would be displayed
programme preceding the Pop-Up Gallery: AR
during our first Pop-Up Gallery on AR.
artist Sander Veenhof and data-visualisation
In the words of the students’ teachers : “Spec-
expert Niels Schrader. They were part of the
tacles, headsets, phones and tablets. These are
jury that awarded the two ‘best’ projects at the
some tools one associates with Augmented Real-
end of the first Pop-Up Gallery. Not one, but
ity (AR). This is obvious because the assumption
two winners were awarded: Jente Hoogeveen
in AR is that reality must be supplemented with
en Jorick de Quaatsteniet. Jente won with her
digital information. Or it would be just reality.
project 'Had je dat gedicht?' (roughly translated
The rationale underlying the research project
as 'How does that rhyme with you?'). Error mes-
therefore lies especially in the use of digital
sages we are confronted with on a daily basis
techniques to make this extra layer onto reality
are transformed into poems in order to take
visible. Without digital technology, no addition
some of the zing out of stressful situations.
to reality and certainly no interaction would be
Jorick won with his project 'Get your shit to-
gether' where animated QR-codes formed text.
It is exactly this assumption that we want to in-
The variety of final projects was staggering
vestigate. Is Augmented Reality only achievable
and impressive. Finally, a publication of all the
by the use of digital means? Can hidden informa-
projects was made with a spread for each stu-
tion be augmented, without the aid digital tech-
dent in which the description and intention of
niques and devices? Following that, we invited
their project were incorporated.
freshmen and -women from the Graphic Design
A special recommendation was given for the
department to get started with Augmented Re-
dyslexia project of Janne van Hooff who visu-
ality. We asked them to think about the kind of
alised how people suffering from dyslexia see
information they wanted to disclose in specific
letters and numbers. Words emerged by moving
situations. What type of problem would they
strips of black paper over the seemingly inco-
want to solve using AR? With what classification
herent rows of letters.
should it work – analogue and/or digital?
These types of projects show the broad range
With this exercise we hoped to widen the con-
of ways in which an extra layer on top of the
cept; while also seeking to come up with new
physical world poses questions about what we
applications and reasons for the use of Aug-
see. What all the students’ projects have in
common, regardless whether they are analogue
In completing the first step of the two step
or digital, is that they appear to invite specta-
project, the students gave us a huge input of ex-
tors to look at the world from another perspec-
amples they encountered: they found all sorts
tive. The same goes for those students who,
of material via social media or as search results
at first, were not completely convinced about
that they consequently discussed. The concept
using AR and how they could incorporate this
developing part, step two, turned out to be
technique in their projects. Ultimately, the idea
very exciting, for the students themselves as
was to provide an enriching experience for the
well as for us. We provided technical help when
students and to give them the tools with which
students were really eager to get their ideas
they could incorporate AR in their work.
augMeNted educatioN How ar tecHNologies exteNd our MiNds ROBIN DE LANGE
In the field of education, just as in many other fields, researchers and developers are experimenting with potential applications of Augmented Reality technologies. Many of these experiments draw on of the possibilities to explore image by Marcelo gerPe, www.creativastock.coM
virtual information spatially. An interesting example is the 4D Anatomy project by daqri (2012), where you can explore the physiology of a human being by moving the display device along a piece of paper with markers. On the screen you see a 3D model of the human body. With sliders and buttons you can set the transparency of the skin, or switch on the layer showing the nervous system for example. Another interesting project in development is the Sesame Street app Big Birdâ€™s Words (Qualcomm, 2013), which uses the latest text recognition algorithms. The (young) users of this upcoming app are asked to look for certain words in their home and aim their device at it. When the device recognizes the word, it gives points to the user. This way the user is asked to involve their environment in the process of learning words. These examples show some of the new forms of interaction and presenting information, with which developers are trying to create new, interesting, and memorable learning experiences.
In this article I will argue that the developments
cal rotation in (b) is actually much faster than
in AR technologies will make digital information
sources much more transparently available to us.
were not only physically rotating the shapes to
In certain cases, this information may even be
fit the slot, but were also trying to determine
seen as part of our cognitive process. Because
whether the shape fits in the slot, thereby
of this change of perspective regarding external
simplifying the task. (Kirsh and Maglio, 1994)
information sources, AR technologies could not only lead to new learning methods, but could,
It is this example of the human capacity to ma-
and in my opinion should, also trigger debates
nipulate the environment to solve problems,
about the very goals of education itself. To back-
which Clark and Chalmers employ to introduce
up this claim, I will first introduce the concept of
the Parity Principle:
the Extended Mind. “if a part of the world functions as a process
Extended Mind and the Parity PrinciplE
which, were it done in the head, we would have no hesitation in recognizing as part of the cognitive process, then that part of the world is (so we claim) part of the cognitive process.” (Clark and
Andy Clark and David Chalmers start their re-
nowned paper ‘The Extended Mind’ with a thought experiment. In this experiment the reader is asked
According to the Parity Principle, the human
to consider three cases and to think about how
mind is not bound by the borders of skin and
much cognition is present in each case:
skull. To make this claim plausible, Clark and Chalmers present a thought experiment involving
(a) A person sits in front of a computer screen
the fictional characters Otto and Inga, who are
which displays geometrical shapes and
remembering how to get to the museum. Otto
is asked to fit the shapes in the displayed
has Alzheimer’s disease and uses a notebook to
‘sockets’, by rotating the shapes in his mind.
serve the function of his memory, while Inga’s
(b) The same situation as in (a), but now the
biological memory is functioning properly. Inga is
person can choose either to mentally rotate
thought to have a belief about the location of the
the shape or physically rotate it by press-
museum, before she recalls this from her internal
ing a button, the latter having some speed
memory. In the same manner, Clark and Chalm-
ers argue, Otto can be said to have a belief about
(c) The same situation in a possible future,
the location of the museum before he actually
where the person can choose between us-
consults his notebook. Thereby, under the right
ing his neural implant that does the rotation
circumstances, the notebook can be seen as an
operation as fast as the computer does in
extension of Otto’s memory. By showing how be-
(b), or using ‘old-fashioned’ mental rotation.
liefs are not bound by the borders of the body, Clark and Chalmers show that true mental events
Now, these kinds of cases are actually not as
can extend in the environment as well.
abstract as they might seem: they describe a very well-known real-life situation, namely
All the examples of cognitive extension that
playing the video game Tetris. In their paper
Clark1 gives in his books and papers are not the
about cognitive performances while playing
typical futuristic technologies that come to mind
Tetris, Kirsh and Maglio found that the physi-
when thinking about humans merging with tech-
nology. Although the possibilities of Brain-Ma-
Now, wrist watches have been around for many
chine Interfaces and neural implants such as in
decades, writing utensils even for centuries. Dur-
case (c) offer very exciting new ways of commu-
ing this time these technologies have become
nicating with technology, this direct interaction
ubiquitous. They have become socially accepted
with brains is by no means necessary to become
and actually shaped culture itself. An interesting
part of the cognitive process (nor are they suf-
question is whether more modern external infor-
ficient for cognitive extension: communicating
mation sources could obtain the same status as
with technology through a Brain Machine Inter-
these age-old technologies and play a similar, ac-
face usually still takes too much cognitive effort.)
tive role in cognitive processes. Could digital in-
Our brains incorporate the world and some of the
formation sources, for example parts of the Web,
technologies therein in their cognitive processes
actually become parts of our minds?
in such an intimate way, that Clark considers us
Could digital information sources
to be “natural born cyborgs” (Clark, 2003). In fact, the technologies which Clark considers
actually become parts of our minds?
as cognitive extensions of our cyborg minds are hardly identified as technology anymore. One ex-
To begin to answer this question and decide
ample he mentions is the use of pen and paper
whether a part of the world should (temporarily)
when doing long multiplications. To calculate the
be seen as part of the cognitive process, we can
product of two numbers, we use an algorithm
refer to the three criteria suggested by Clark:
that divides the process of multiplying arbitrary
down figures in certain locations, we use the pen to manipulate the external memory source, the
The technological aid should always be available when needed;
paper. The writing utensils play a crucial role in this cognitive process and are therefore, accord-
The information retrieved from the external source should be directly endorsed and
large numbers into very simple steps. By writing
The external resource should be directly
ing to the Parity Principle, actually part of this
available without difficulty. More precisely:
the information access costs, a measure of the combination of time and physical and
Another example shows that it has become com-
cognitive effort, should be as low as with
mon to talk about the information that is in
an equivalent function of the brain (Smart
some of our technologies as if part of our own
et al, 2008).
knowledge. When somebody asks us on the street whether we know what time it is, and we are
From this, it is quite clear that the Web with which
wearing our watch, we often answer “yes”. Sub-
we interact on a daily basis cannot be seen as part
sequently we raise our arm, look at our watch
of our cognition. From our critical stance we do
and see what time it is. Now, according to Clark
not immediately believe most information we en-
this is not simply loosely formulated informal lan-
counter on the Web something which is, due to the
guage. You actually do know what time it is, ‘you’
open character of the Web, probably a wise at-
is only “the hybrid biotechnological system that
titude. Furthermore, the information access costs
now includes the wristwatch as a proper part”
when retrieving information from the Web is way
(Clark, 2003). (This proven transparency of the
too high. The user has to put physical and cogni-
wrist watch is what makes the development of
tive effort into navigating the browser to the right
smart watches interesting.)
page, then wait for the downloading of the page
and scan through the text to find the information
know the meaning of a certain word that is not
he needs. In the widespread current way of inter-
in your biological memory, and a short, clear de-
acting, the information access costs of retrieving
scription of the word pops up immediately in the
information from the Web are way too high to be
corner of your field of view, would you say that
considered as part of the cognitive process.
you know the meaning of this word?
Lowering information access costs
I can imagine that you — after you get more and more used to the device and have experienced this situation a few times already — might
However, technology is ever-evolving and many of
say ‘yes’, similar to the situation with the wrist
the present advancements will make information
watch.2 More so, in a very real sense, I think you
available to us in more reliable, quicker and more
might start to feel like you really do know it.
intuitive ways. Processors in smartphones are becoming faster, telecommunication service provid-
But what is it with AR technologies, that they
ers keep improving their networks to decrease
could lower information access cost so signifi-
download times, batteries are getting better, user
cantly? Of course Head-Up Displays (HUDs) play a
interfaces are becoming more intuitive, etc. The
great part in this, by eliminating the physical ef-
technological developments that are associated
fort of getting your smartphone from your pocket
with Augmented Reality especially have the po-
and having to hold it in within your view. When
tential to lower the information access costs for
it does indeed become ordinary to wear HUDs,
digital sources considerably. Let us consider the
information can be presented to the user at all
following scenario to get a better idea: For a few years now you have your own AR glasses. Despite of what many critics expected, it has become socially quite acceptable to be wearing a Head-Up Display at all times. An application running in the background does speech and text recognition and keeps track of words you (the biological ‘you’) don’t know the meaning of. You’ve been using this application for a few months now and it’s starting to have quite a good sense of when to present you with the meaning of a word you encounter. Of course, the applica-
The main goal of education should be to train the technologically extended cognitive system. tion knows when you’re busy driving for example and doesn’t bother you then. Now, when a friend (who is not really into new technology and rather asks a friend to help him) asks you whether you
times, at the exact moment when it is needed.
these digital information sources could then be
Another important aspect of AR is the use of
seen as proper parts of our hybrid minds.
information from different sensors and smart algorithms doing image and speech recognition. present information in context-sensitive ways re-
Extended cognition and education
sponding both to the environment and the user.
In the previous I have introduced the concept of
Furthermore, digital information can be placed
extended cognition and explored the possibili-
over the world, which is of course the main idea
ties of digital technology for cognitive extension.
of AR3. By doing so, you can interact with digital
Adopting this philosophical perspective can have
information in similar ways to how you interact
huge consequences for a field like education. One
with the physical world, creating a very natural,
could argue from this perspective that the main
goal of education should be to train the techno-
By combining these, possibilities are created to
logically extended cognitive system. These are the characteristics of AR that create the potential of making digital information much
By accepting digital external information sources
more transparently available to us. I suggest that
as a proper part of memory, it could be decided to
under certain conditions, well designed, person-
store certain information that we want to remem-
alized information sources are able to compete
ber in an easily accessible, personalized cloud of
with mental resources in terms of costs of infor-
knowledge. Instead of trying to store all informa-
mation access. According to the Parity Principle,
tion in biological memory by endless repetition,
Image courtesy of Gaia Dempsey and daqri, see http://daqri.com
this task of storing information could be off-loaded
daqri. (2012). 4D ANATOMY. Retrieved from http://site.daqri.com/labs/4d-anatomy
to an external source which is constantly available to us at low information access costs. ¢¢
de Lange, R. (2011). Het Semantic Web en
The general view on the use of technology in
netwerktechnologische cognitieve uit-
education is quite different from the view ex-
breidingen. Retrieved from http://www.
pressed in this article though. For the most part
of their education, students still only get to use some basic technologies: a pen, a piece of paper
Dennett, D. (1996). Kinds of Minds: Toward
and maybe a dictionary or an outdated (graphi-
an Understanding of Consciousness. New
York USA: Basic Books.
This critical attitude towards the use of technol-
Dror, I. E., & Harnad, S. (2008). Offloading
ogy is very understandable. Digital technology is
Cognition onto Cognitive Technology. In I.
developing very rapidly, careful decisions have to
E. Dror, & S. Harnad, Cognition Distributed:
be made about how to use it in education. To
How Cognitive Technology Extends Our
come to these decisions, a lot of research on
Minds (pp. 1-23).
the use of technology in the learning process is needed. Furthermore, there should be an active
ESRC/EPSRC Technology Enhanced Learning
discussion on the goals of education and what
Programme. (2012). System Upgrade - Real-
technologies students can use to reach these
ising the Vision for UK Education. London:
goals. An extended view of the mind, in which
external resources have an active role in the cognitive process, can offer a valuable perspective in
Kirsh, D., & Maglio, P. (1994). On distinguishing epistemic from pragmatic action.
Cognitive science , 513-549.
Qualcomm. (2013). Big Birds words. Retrieved from http://www.qualcomm.com/
Blomberg, O. (2009). Do socio-technical
systems cognise? 2nd Symposium on Com-
puting and Philosophy . ¢¢ ¢¢
Rorty, R. (1999). Education as Socializa-
Clark, A. (2003). Natural-Born Cyborgs.
tion and as Individualization. In R. Rorty,
Minds, Technologies, and the future of
Philosophy and Social Hope (pp. 114-126).
human intelligence. Oxford: Oxford Univ.
London: Penguin Books.
Press. ¢¢ ¢¢
the Web. Network-Enabled Cognition: The
bodiment, Action and Cognitive Extension.
Contribution of Social and Technological
Oxford: Oxford Univ. Press.
Networks to Human Cognition .
Clark, A., & Chalmers, D. (1998). The Extended Mind. Analysis , 10-23.
Smart, P. (2010 (In Press)). Cognition and
Clark, A. (2008). Supersizing the Mind. Em-
Smart, P., Engelbrecht, P., Braines, D., Hendler, J., & Shadbolt, N. (2008). The
Extended Mind and Network-Enabled Cognition. Retrieved June 24, 2011, from University of Southampton - School of Electronics and Computer Science: http://eprints.ecs. soton.ac.uk/16649/1/Network-Enabled_Cognitionv17.pdf ¢¢
Wheeler, M. (2011, November). Thinking Beyond the Brain: Educating and Building, from the Standpoint of Extended Cognition. Computational Culture .
Robin de Lange Robin de Lange has a bachelor’s degree in
The initial paper ‘The Extended Mind’
Physics and Philosophy and has followed
was written by Andy Clark and David
courses on Artificial Intelligence. He is now
Chalmers. Because Clark has written
a student at the Media Technology MSc pro-
many other papers and books on this
gram at Leiden University and is particularly
subject, I will refer to Clark further on.
interested in technologically extended cognition. For his graduation research project
If the word would be jargon of a field you
he is developing an Augmented Reality ap-
are not familiar with for example, you would
plication that supports the graphical solving
probably not understand the meaning di-
of mathematical equations; thereby show-
rectly and need to look up more informa-
ing educational challenges and possibilities.
tion, thereby increasing the costs of information access.
Besides his studies, Robin has taken part in several entrepreneurial projects. Most no-
This characteristic of AR of overlay
tably, he was the co-owner of a company
ing the physical world with virtual objects is
that specialized in homework guidance and
not really present in this scenario. For this
tutoring. He is a freelance video producer
reason, one might argue that the example
and science communicator. At the moment,
does not really show AR. However, it does
he is looking for funding to do a PhD within
use certain AR technologies intensively to
his field of interest.
provide context-sensitive information to the user who interacts with the world.
For more information: www.robindelange.com
The information access costs of looking up a word in the dictionary go through the roof.
Moore’s law seems to be failing here. The hardware in these devices stays roughly the same, even remains the same price!
How did we do it: Which augmented reality software does the AR Lab use? Wim van Eck
tHis aPPlicatioN, coMMissioNed by tHe rce (cultural Heritage ageNcy of tHe NetHerlaNds), is develoPed by tHe ar lab usiNg vuforia aNd uNity 3d. tHe PaiNtiNg 'isaac blessiNg jacob' by tHe dutcH PaiNter govert fliNck (16151660) Has beeN augMeNted to give iNsigHt iNto tHe PaiNters Practice aNd offers scieNtific data iN a Playful way. tHe aPPlicatioN will PreMier at MuseuM catHarijNecoveNt (utrecHt, NetHerlaNds) oN May 17tH.
Often we are asked “What is the best augmented
our daily workflow. For this overview we created
reality software?” which is a difficult question to
four categories: ‘easy-to-use for mobile devices’,
answer. First of all, there are so many programs
‘easy-to-use for desktop augmented reality’,
available it is difficult to actually have tried them
‘augmented reality software for interactions’ and
all, and updates and new programs pop up all the
‘head-mounted augmented reality’.
time. Secondly, choosing which software to use really depends on what you want to achieve in the end. At the AR Lab we use a variety of augmented reality programs to realize our projects. I will give an overview of the software we use and
easy-to-use for mobile devices
why we use it. This doesn’t mean we claim that
Since the AR Lab is based at the Royal Academy
this is the best software and that there are no
of Art The Hague most of our projects are devel-
alternatives, we merely aim to give an insight into
oped together with our students, or we supervise
The app provides clear information for every step you have to take, removing any technical barriers (Figure 1). Markers can be generated on the spot by taking pictures with the device, and photographs and videos which are stored on the device can be used to augment the chosen scene. When more possibilities are needed there is also a free online application (www.aurasma.com/partners), you only need to register to be able to access it. This well documented application makes it easier to precisely position your virtual objects, and also to import images and videos which are not on your mobile device, as well as 3D models. It is a pity that Aurasma only seems to give good support for 3ds Max and Maya, it’s more difficult to import 3D models from, for example, Cinema 4d. It is very easy though to use video with transparency, a not very common option which can give great results. The tracking quality of Aurasma is Fig. 1
not the best around, fast camera movements will result in loss of tracking, but it is definitely good enough for most projects.
easy-to-use for desktop augmented reality Nowadays most software companies seem to focus on mobile devices; desktop augmented reality software is getting scarcer. Some of our students, however, still prefer to present their creations on Fig. 2
a large screen, and BuildAR (http://www.buildar. co.nz) is the software we use most often. BuildAR is based on the world-renowned ARToolKit, but
students who want to develop their own applica-
adds an easy-to-use graphical interface. It runs
tion. Many of these students don’t have a techni-
on both Windows and OSX, though the OSX ver-
cal background, so easy-to-use software is pref-
sion lacks some options and seems to be less sta-
erable. Furthermore, they often have their own
ble. When downloading BuildAR, there are quite
smartphone or tablet and are very keen to de-
some versions which can be a bit confusing. The
velop for it. Aurasma (www.aurasma.com) quick-
‘BuildAR Pro 2 Trial Download’ is the newest ver-
ly became our favourite choice for this group of
sion for Windows and features both marker track-
students. By downloading the free Aurasma app
ing and image based tracking. You can use this
from the iOS App Store or Google Play store stu-
version for free, but there is a ‘demo’ watermark
dents can immediately begin creating their first
on your video-background and you won’t be able
to save your scene (Figure 2). ‘BuildAR Pro Mac
they need is their mobile device.
Trial Download’ is the OSX version and only fea-
it very easy to import 3d-models including anima-
tures marker tracking; the demo-limitations are
tions from almost all 3d-packages. Vuforia’s image
the same as in the Windows version. Lastly, there
based tracking is extremely stable, the tracking
is the ‘BuildAR Free version Download (2008 ver-
quality stays good even if the tracking image is
sion)’, which is only available for Windows and
partly occluded or when there is little light avail-
is free of charge. The only limitation is a ‘HIT-
able. Besides image markers there are also frame
LabNZ’ logo, but many of our students don’t mind
markers available, which use a pattern of black
this logo so much though. BuildAR is easy-to-use;
and white cubes positioned around the image.
the new version can augment your scene with im-
Semiconductor company Qualcomm offers Vuforia
ages, video, audio and 3D models.
for free and unity has a free version as well. However, you will need to buy the unity iOS or Android
ar software for interactions
add-on to be able to export to a mobile device.
A program we often use at the AR Lab ourselves tively develop for iOS (Xcode), Android (Eclipse)
Head-mounted augmented reality
or use it as a plugin for unity (www.unity3d.com)
For head mounted based augmented reality we
allowing you to easily export to iOS and Android.
have solely used software and hardware devel-
unity (Figure 3) is a popular game engine offering
oped by Tu Delft. In issue 2 of AR[t] we intro-
excellent visual quality on mobile devices. Adding
duced our latest augmented reality headset
interactions through scripting is relatively easy,
named Marty and its tracking software. Also, this
there are many tutorials around and the unity fo-
issue of AR[t] features more information about
rum is a great source for information. unity makes
is Vuforia (https://www.vuforia.com). You can na-
Fig. 3 75
AR[t] Pick Immaterials by Design Studio onformative and Christopher Warnow, 2011
Immaterials is the result of a collaboration between the Design Studio onformative (www.onformative.com) and the Computational Designer Christopher Warnow (christopherwarnow.com). The project explores how information can be integrated into physical space. Using the light painting technique, data is placed in a room. The resulting forms depict possible data sets and examine the design possibilities between technoid holograms and personal notes. Further information: ¢¢
Contributors Wim van Eck
Royal Academy of Art (KABK) email@example.com
Royal Academy of Art (KABK) firstname.lastname@example.org
Wim van Eck is the 3D animation specialist of the AR Lab. His main tasks are developing Augmented Reality projects, supporting and supervising students and creating 3d content. His interests are, among others, real-time 3d animation, game design and creative research.
Edwin van der Heide Leiden University email@example.com
Edwin van der Heide is an artist and researcher in the field of sound, space and interaction. Beside’s running his own studio he’s part-time assistant professor at Leiden University (LIACS / Media Technology MSc programme) and heading the Spatial Interaction Lab at the ArtScience Interfaculty of the Royal Conservatoire and Arts Academy in The Hague.
Pieter Jonker Delft University of Technology P.P.Jonker@tudelft.nl
Pieter Jonker is Professor at Delft University of Technology, Faculty Mechanical, Maritime and Materials Engineering (3ME). His main interests and fields of research are: real-time embedded image processing, parallel image processing architectures, robot vision, robot learning and Augmented Reality.
Yolande Kolstee Royal Academy of Art (KABK) Y.Kolstee@kabk.nl
Yolande Kolstee is head of the AR Lab since 2006. She holds the post of Lector (Dutch for researcher in professional universities) in the field of ‘Innovative Visualisation Techniques in higher Art Education’ for the Royal Academy of Art, The Hague.
Mariana Kniveton is currently a master student at Utrecht Universtity, studying New Media and Digital Culture. Since september 2012 she has worked as an intern at the Research Department IVT and the AR Lab. After a brief stint as a cover model for AR[t] #2, Mariana took up editing duties for this current issue.
Maarten Lamers Leiden University firstname.lastname@example.org
Maarten Lamers is assistant professor at the Leiden Institute of Advanced Computer Science (LIACS) and board member of the Media Technology MSc program. Specializations include social robotics, bio-hybrid computer games, scientific creativity, and models for perceptualization.
Hanna Schraffenberger Leiden University email@example.com Hanna Schraffenberger works as a researcher and PhD student at the Leiden Institute of Advanced Computer Science (LIACS) and at the AR Lab in The Hague. Her research interests include interaction in interactive art and (non-visual) Augmented Reality.
Esmé Vahrmeijer Royal Academy of Art (KABK) firstname.lastname@example.org
Esmé Vahrmeijer is the graphic designer and webmaster of the AR Lab. Besides her work at the AR Lab, she is a part time student at the Royal Academy of Art (KABK) and runs her own graphic design studio Ooxo. Her interests are in graphic design, typography, web design, photography and education.
Delft university of Technology email@example.com
Jouke Verlinden is assistant professor at the section of computer aided design engineering at the Faculty of Industrial Design Engineering. With a background in virtual reality and interaction design, he leads the “Augmented Matter in Context” lab that focuses on blend between bits and atoms for design and creativity.
Matt Ramirez has over 15 years experience in web design and e-learning content development for a variety of subject areas including medicine, IT, science, special collections and business.
lotte de reus www.lottedereus.nl
guest coNtributors bc “Heavy” bierMaNN The Heavy Projects www.theheavyprojects.com
BC "Heavy" is founder of The Heavy Projects [and its collaborative spin-off Re+Public]. Heavy creates innovative interfaces between digital design and physical worlds in ways that that provoke the imagination and challenge existing styles of art, design, and interaction.
robiN de laNge www.robindelange.com
Robin de Lange is a student at the Media Technology MSc program at Leiden university and part-time entrepreneur. He is looking for funding to do a PhD on technologically extended cognition.
Lotte de Reus recently received her Master’s degree in Design for Interaction at the Faculty of Industrial Design Engineering at Delft university of Technology. In the future, Lotte would again like to work in the domains that combine art and technology.
alejaNdro veliz reyes university of Salford
Alejandro Veliz Reyes is a teaching assistant and PhD student on digital architectural design at the university of Salford (united Kingdom). His research interests are design pedagogy, augmented reality, and collaborative technologies for design
aNtal ruHl www.antalruhl.com
Oliver Percivall works in I.T Project Management. He is interested in where technology could take us. This interest has resulted in the science fiction novel "The Augmented Star".
Antal Ruhl is a media artist with a background in design, science and art. He creates objects that let us rethink our environment. These objects vary from kinetic sculptures to interactive installations.
The next issue of AR[t] will be out in the fourth quarter of 2013.
Issue 3 - May 2013.