inter[ACT]: Embodied Interaction with Post-Architectural Space

ΊINTER΋ACT: EMBODIED INTERACTION IN POSTͳARCHITECTURAL SPACE

by Kathy Yuen September 2014

A thesis submi ed to the Faculty of the Graduate School of the University at Buffalo, State University of New York in par al fulfillment of the requirements for the degree of Master of Fine Arts and Master of Architecture

School of Architecture and Planning Department of Media Study

Acknowledgements

I would like to thank my commi ee members Mark Shepard, Nick Bruscia, and Dave Pape for their pa ence, constant support, and guidance throughout the thesis process.

I would also like to thank Molly Hogle, Albis Del Barrio, and Jon King for their help with fabrica on.

Michael Lempert, Jose Pesantz, Mike Kirschner, Hendrik Heur, Oriane Daley, Gabby Printz and Adam Laskowitz for their feedback and support along the way.

ii

Table of Contents Acknowledgements Abstract Introduc on Interac on in Social Media Shi ing from Graphical Interface to Tangiable Interfaces

Precendent Study and Literature Review Case Study 01: Coded Objects Code/Space Where the Ac on is Radical Atoms

Design Methodology Experiment 01: inter[ACT] 1.0 Case Study 02: Coded Space and Datascapes Experiment 02: inter[ACT] 2.0

Cri cal Response

ii iv 1 1 3 6 6 10 13 16 18 18 25 39

68

Experiment 03: inter[ACT] 2.1

72

Conclusion Bibliography

84 86

iii

Abstract The computer screen has become the most conven onal interface as we have grown increasingly virtually connected to each other through the expansion of internet accessibility and the prolifera on of social media. Computer screens act as a dis nct boundary between the space that we physically occupy and the space with which we are virtually interac ng. There is a transgression that occurs at the boundaries of physical and virtual space as each new interface looks to digitally augment the space we occupy; we have begun to move beyond the interface of the screen. Theorists working in the field of Human Computer Interac on (HCI), such as Hiroshi Ishii and Paul Dourish, believe tangible interac on with digitally augmented ar facts will allow users to interact within physical and digital spaces simultaneously. While most of these interfaces are s ll experimental and not yet commercially available, the interac ons that are fostered by tangible interfaces intend to reveal more informa on about the system to users that can’t be shown through conven onal interfaces.

The data we embody has unprecedented spa al and social implica ons, and [inter]ACT inves gates how humans communicate in space through physically reconfigurable ar facts. In addi on to going beyond overlaying digital ac ons and informa on onto physical space, the thesis project tests how nuanced interac ons with body-scaled objects can be used as digital interfaces in real- me. More importantly, the project also tries to understand the implica ons of data transgressions from digital to physical space (and vice versa) in a social context. To answer this ques on, the thesis project takes the proposed design frameworks from Hiro Ishii and Paul

iv

Dourish in the subject ma er of embodied and tangiable interac on to implement and develop a system that enables a connec on between physical and digital ac ons.

v

This page is intenĆ&#x;onally leĹŒ blank.

vi

Introduc on The way that we interact with digital systems evolves as new usage models for technology develop. We are constantly searching for new ways to interact with machines and computers, and seek an understanding as to how it might aect society. Movies such as Minority Report, Matrix, Hackers and Tron served as great inspira ons for this thesis, as they depict new imagina ons of the ways that the body can interact with digital systems. Each of these movies show the importance of human computer interac on (HCI) and how it can redefine the way that we interact with computer and humans through new technologies.

Interac ons in Social Media Social media is a new category of digital interface which allows users to interact with each other through dierent avenues - i.e. text, images, and video - that is posted on various social media websites. This form of interac on uses the social media websites, such as Twi er, Facebook, Foursquare, Youtube, Goggle+, and Vimeo, to mediate exchanges between users. The social media websites o en use ac ons found in the physical world to describe interac ons that are happening in these digital spaces. As a result there are new acceptable behaviors a ached to those ac ons.

1

Figure 1. A Youtube video posted by Tommy Jordan that went viral in 2013.1 The internet allows mul ple people to connect to the same content at the same me, 1 YouTube. “Facebook Paren ng: For the troubled teen..” YouTube. h ps://www.youtube. com/watch?v=kl1ujzRidmU (accessed May 4, 2013).

2

thus crea ng a new interac ve public forum, where users can comment, like, and dislike content and comments. Figure 1 depicts a father, Tommy Jordan, disciplining his daughter via YouTube, where the daughter is not only subject to the disappointment of her father, but the disapproval of the 1 billion users that are strangers, acquaintances, friends, and family. At the same me, the father also gets comments a acking as well as suppor ng his behavior. This tac c was used to teach the daughter about the repercussions of taking private issues and pos ng them in a public forum for everyone to see. Not only was this posted on YouTube but it was also shared on Facebook. The ability to share this video numerous mes and via mul ple pla orms allows contents to have a farther reach than ever before, especially now that social media pla orms allow for shareable content to be distributed through mul ple linked accounts between Twi er, Instagram, Facebook, Youtube, Vimeo, and Google.

Shi ing From Graphical Interfaces to Tangible Interfaces We are constantly looking at ways in which technology can make our lives easier and more efficient, yet, at the same me, more human. With the advent of Social Media, there has been a paradigm shi in the development of efficacy in technology. Social media has changed the way that digital technology is embedded in everyday life. We no longer rely on technology solely to help people do things faster and more efficiently, but we desire technology to enrich, augment and help us share our everyday experiences.

The conven onal compu ng interface – the screen – has served its purpose in relaying informa on to the user, but is only a window that allows a glimpse of what is happening

3

virtually. There a variety of metaphors and digi zed coded objects that are used to explain compu ng func ons and opera ons. Figure 2 depicts several physical objects that have been digi zed to mimic processes that happen in the physical world. These digi zed objects only metaphorically represent the physical processes.

Figure 2. A screenshot of a typical desktop screen highligh ng several physical world objects found in digital space. A) The recycling bin is used to dispose of unwanted items similarily to a trash bin in the physical world. B) The s cky note is literal transla on of the s cky note in the real world, used to keep quick notes and a ached to the desktop. C) The file cabinet is borrowed to represent the hierarchical structure of the file system on the opera ng system.

So ware designers use mental models of exis ng physical processes and objects to help users be er understand the new interac ons that exist virtually. The trash bin interface on most personal computers iden fies a process that allows items within the system to be thrown out.

4

Items that are placed in the trash bin are not necessary thrown out. The act of placing an item in the trash bin moves the item to a loca on on the physical drive so that it can be overridden later. Similarly in the physical world, items do not disappear when placed in a trashcan, nor when transported to a landfill to be buried. Data is never deleted - just transformed and transduced into another form.

In a world so distracted with screens, can more ambient forms of compu ng engage the bodily sensors to provide similar interac ons as the screen? This thesis ques ons the ways that our bodies currently relate to the data to examine how user ac ons can be transduced into data that can actuate digital space. Interac ons with both the digital and physical world are analyzed to understand how user ac ons directly relate to the data that is generated in the two respec ve worlds. Can a tangible interface become less representa onal and more literal?

The precedent study evaluates exis ng commercial and experimental interfaces developed in the space of tangiable interac on. The literature review provides an understanding of the design strategies in which some of these interfaces were developed. The research is is examined from two scales: the first scale examines the rela onship of the body, object, and data in a localized systems and spaces; and the second scale looks at the rela onship of the body, object, and data across several interconnected systems and spaces. The research is then conceptualized to develop usage models surrounding the implementa on of embodied system between social, proximal space of si ng and the social, distributed space of social media.

5

Precedent Study and Literature Review Case Study 001: Coded Objects This study explores the space of coded objects through the survey of nine exis ng commercial, art, and research projects understand the current use of tangible interfaces.

The case studies, Grasshopper, MaxMSP, Scratch, and 3D MAX’s Stack modifier, are digital interafaces that use method of interac ons found in physical space. For example, MaxMSP, a visual programming language used for audio and video programming, has a patch style visual language that found on physical audio equipment. The case studies, Si eo, ReacTable, Topobo, and Universal Constructor, are physical coded objects that enable more nuanced interac on similar to building blocks. Each case studied are examined from two perspec ves; how a person can interact with an individual module, and how does the individual unit aggregate and interact with other units?

Digital - Coded Object

Dataflow after construction Dataflow during construction

Process

Human Performance

Dataflow in 1D

Dataflow in 2D

Dataflow in 3D

System Performance

State

1

Object Physical - Coded Object

Figure 3. The systems of interac ons are organized by the number of axes that can be interacted on. The legend for the diagramma c excercise, showing the data flow and process associated with typical interac ons found in exis ng tangible systems

6

1

Name: 3d Max - Stack Modifier Data Flow: Linear, One Directional

1

Data Structure: Operations: Geometric Modifiers Input: Geometry Output: Geometry Final Outcome: Geometry Synthesis

2

2 3

1 1

Name: Rhino Grasshopper Data Flow: Recursive, One Directional Data Structure: Operations: Geometric Modifiers and Analysis Input: Points,Lines,Surfaces,Geometry,Numbers,Time Output: Points,Lines,Surfaces,Geometry,Numbers,Time

2

Final Outcome: Geometry Synthesis and Analysis

2 3

Name: Scratch

1

1

Data Flow: Recursive, Event Based, One Directional Data Structure: Operations: Positional Modifier Input: Movement and Transformation Output: Image Location and Orientation Final Outcome: Moving Image

2

2

3

7

Name: MaxMSP Data Flow: Recursive , Two Directional

1

Data Structure: Operations: Sound or Video Modifier Input: Number, Audio/Video Signal Output: Number, Audio/Video Signal Final Outcome: Sound or Video Analysis or Synthesis

2

pical Behaviors

Possible Aggregations

Name: Sifteo Data Flow: Linear, Proximity, Two Dimensional

1 Data Structure: Operations: Press, Unit Placement, Tilt, Shake Input: Press, Unit Placement, Tilt, Shake Output: Moving Image, Orientated Image Final Outcome: Interactive Game

2

1 Name: Reactable Data Flow: Three Dimensional Data Structure: Operations: Audio Modifier Input: Unit Placement, Rotation, Button, Slider Output: Timing, Signal Final Outcome: Audio Synthesis

2

8

1

2

1

Name: Universal Constructor Data Flow: Three Dimensional Data Structure: Operations: Unit Placement Input: Message Output: Propagation Final Outcome: Exhaustive Search

2

Name: Topobo Data Flow: Three Dimensional Data Structure: Operations: Movement Input: Rotation, Unit Placement and Orientation Output: Movement Final Outcome: Kinectic Movement

9

Currently, there is limited research being done surrounding embodied interac on and tangible media. This thesis responses to the theore cal work of Rob Kitchin and Mar n Dodge’s Code/Space, Hiro Iishi’s Radical Atom, and Paul Dourish’s Where the Ac on is. Each of these theore cal works holds a different point of view about how user should engage tangible device and coded objects to enable embodied interac on. Code/Space is a recently publish book that maps the current landscape of coded objects against coded spaces. Where the Ac on is, is an older publica on that discusses the design frameworks and approaches surrounding embodied interac on which references, Tangiable Bits, a prior design framework to Radical Atoms. The Radical Atom is the most recent design framework supported by the MIT Tangiable Media Lab. Rob Kitchin and Mar n Dodge’s Code/Space are geographers that surveys the landscape of exis ng coded objects and spaces used in everyday life.

Code/Space In Code/Space, a coded object is defined as “a material object in which code has been embedded, but where this so ware is incidental to primary func oning of the object”2 . Kitchin and Dodge classify coded objects into various levels of digital capabili es ranging from objects with limited digital func onally to mostly digital func onally3 : Peripherally coded objects are objects that are augmented so that they are machinereadable but doesn’t require so ware to perform. Hard coded objects have embedded firmware 2 Kitchin, Rob. Code/space: Software and Everyday Life. Software Studies. Cambridge, Mass: MIT Press, 2011, 262.

3 Kitchin, 55.

10

that limits their func onality. Closed codeject are coded objects that func on independent of their surrounding in a closed feedback loop. Sensory codejects are objects that are equipped with sensor that enable the object to react meaningfully to environmental data. Logjects are objects that log data over me. Networked Logjects are objects that require network access and connec on to other technologies to func on.

Peripherally coded objects can s ll operate as analog objects without the digital augmenta on, but as coded objects become less like their analog counterparts, they become reliant on the digital technology to define their usage. A cellphone can’t be a cellphone if it doesn’t have any network coverage because you can’t use it to make phone calls. Even closed objects such as digital camera, sensors define the object capabili es, without the CMOS sensor the digital camera could not be a camera because it can’t take pictures. At the same me, these new digi zed objects need to have digital replacements for their analog parts. Analog camera have viewfinders to see the image that is being captured, while many digital camera had viewfinders, they also had an LCD screen that would display the captured image. It has change the experience of taking images drama cally because you don’t have to drop film off at the photo developer and the image is developed instantly. While there are new experiences that are created, old experiences aren’t forgo en.

11

Figure 4. A modified diagram of the decision tree found in Code/Space. This modified diagram emphasizes the different levels of func onal characteris cs while showing exis ng technology that sits in each of these categories.4 Peripherally coded objects can s ll operate as analog objects without the digital augmenta on, but as coded objects become less like their analog counterparts, they become reliant on the digital technology to define their usage. A cellphone can’t be a cellphone if it doesn’t have any network coverage because you can’t use it to make phone calls. Even closed objects such as digital camera, sensors define the object capabili es, without the CMOS sensor the digital camera could not be a camera because it can’t take pictures. At the same me, these new digi zed objects need to have digital replacements for their analog parts. Analog camera have viewfinders to see the image that is being captured, while many digital camera had viewfinders, they also had an LCD screen that would display the captured image. It has change the experience of taking images drama cally because you don’t have to drop film off at the photo developer and the image is developed instantly. While there are new experiences that are created, old experiences aren’t forgo en.

4 Kitchin, 55.

12

Where the Ac on is In “Where the Ac on is”, Dourish states that metaphorical approach of graphical interface “exploit our familiarity and facility with the everyday world - whether is a world of social interac on or physical ar facts.”5 Graphical interfaces abstract our experiences and understanding of the world to describe and represent digital objects, processes, and space.

In the physical world, objects are directly related to user ac on and manipula ons, but those ac ons need to be contextualized to a par cular se ng or situa on. Objects carry mul ple meaning and uses, and depending on the situa on, it could have various social implica ons. In the case of a chair, it could be repurposed as a stepping stool to stand on to grab things out of reach, a tool for working in front of a computer, an object of leisure by using watch to television, or a social object where you are si ng with some friends enjoying a meal. While using a chair as a stepping stool, working in front of computer, watching television is described as personal and non-social, an addi onal person in any of these situa ons would inherently create a social context.

In both the physical and digital world we have socially constructed objects from our experiences in the physical world. Uses for objects in the physical world are derived from social construc ons, and emerge as objects are repurposed and created. In the digital world, as it currently existed many of the interac ve objects and ac ons are just repurposed from physical world to describe a something similar. The ac ng of friending in Facebook or following in Twi er has a completely different connota on from friending or following in the physical world.

5 Dourish, Paul. Where the Action Is: The Foundations of Embodied Interaction. New Ed. The MIT Press, 2004. 17.

13

Friending someone in the requires two people ge ng to each other over a period of me, but friending in Facebook, is the act of giving access to data and content associated to an each other mutually. Rather than ge ng to know each other over me, you get to know each other in an instance depending on the amountof content provided by the individual users. Following, in the physical world, typically has a bad connota on associated with stalking and preying, because following implies that there is someone behind with or without knowledge. Following on Twi er doesn’t nearly have the same connota on, because it announces when there are new users that are following. Users know who is following, or in a sense stalking, because they will be no fied of live updates of content.

Instead of using today’s model of repurposing physical ac ons for digital models of interac on, Paul Dourish argues the combina on of social compu ng and tangible compu ng will provide embodied interac on enhancing the future human computer interac ons. Dourish outlines three ways that embodied interac on is relevant and beneficial to compu ng6 : 1. The rela onship of inten on and the ac ons is condi onal based on the situa on and context. Dourish describes that an “increased sensi vity to se ngs” will logically lead to the comprehension of how inten on of the person and ac on of the body is linked together within specific se ngs, environments, and situa ons. This embodiment of inten on and ac on will determine how computa on and se ng or 6

Dourish, 19-20.

14

context will come together.

2. The system of interac on should be created in a developed in concrete terms rather than an abstract context. In the design of interac ve systems, the user should not be removed from the context or situa on in which they are going to be interac ng in. Scien fic observa ons and field studies in the context of their ac vi es provide much more reliable account of user interac ons than generaliza ons and assump ons, crea ng an understanding of how user interac ons manifest in specific interfaces.

3. The par cipa on of different people and reasons creates different roles for object, and the object carries informa on about the different par cipants. Dourish gives the example of the pa ent cards, where the paper version actually informs much more pa ent informa on than their digital versions. The “history of ac vi es over the card and around the pa ent” is recorded through the wear and tear, annota ons, and marks by different healthcare providers.

These condi ons of interac ons are found in the way that we interact with everyday objects in the physical environment but not yet seen in any form of digital interac on. This means of interac ng with physical ar facts and world around us provides a deeper connec on to the interac on than is currently found in any compu ng system. This model of interac on takes the physical quali es of interac ng in physical space and applies it to digital space.

15

Radical Atoms Tangible interfaces rely less on interpreta ons, which “act as physical manifesta ons of computa on, allowing us to interact directly with the por on that is made tangible.”7 At the Tangible Media Lab in MIT, research has been focused around the idea that all digital informa on is able to have a physical, tangible manifesta on. They propose that the future of Human Computer Interac on will compose of Radical Atoms which are both physical and digital objects to “make informa on directly manipulable and intui vely perceived”.8

GUI

PA INTED BITS

TUI

RADICAL

TANGIBLE BITS

AT OMS THE PHYSICAL WORLD

THE DIGITAL WORLD

Figure 5. The comparison of Radical Atoms to graphical and tangible interfaces using the iceberg metaphor depicted by the Tangible Media Lab. Graphical interfaces. Graphical interfaces using screen technology is compared to “looking through the surface of the water” and the “interac ons with the forms below” are limited to the use of remote controls like keyboards, mice, and touchscreens. 9 Tangible interface allow users to par ally interact with por ons of the digital world 7 Ishii, Hiroshi, David Lakatos, Leonardo Bonanni, and Jean-Bapstiste Labrune. “Radical Atoms: Beyond Tangible Bits, Toward Transformable Materials.” Magazine Interactions 19, no. 1 (February 2012): 38.

8 Ishii, 40.

9 Ishii, 40..

16

through physical manipula on. “Radical Atoms is our vision for the future of interac on with hypothe cal dynamic materials, in which all digital informa on has physical manifesta on.”10 This type of interac on would allow users to use of objects in the physical world to directly to interact with digital world. This is an opposing model of interac on than described by Paul Dourish in “Where the Ac on is”, digital space should take on characteris cs of physical interac on. The physical objects, in the Radical Atom design framework, become more like digital objects and take on digital characteris cs.

The main focus of the research examines how the physical state of materials can be synced with digital states so that when informa on or data changes, materials can deform and change with the data. To be called a Radical Atom, the object or objects must sa sfy three condi ons. It must have the ability to “transform its shape to reflect the computa onal state, conform to constrains imposed by the environmentor user, and it must inform the user of its transforma onal capabili es”. This design framework for defines a new specific type of coded objects, where Radical Atoms are physical objects that emulate digital data through materiality. The Radical Atom should be as malleable as the data that it is trying to represent.

10

Ishii, 40..

17

Design Methodology “A child playing with blocks engages with them in quite different ways than we could provide in a screen-based virtual equivalent; so tangible coming is exploring how to get the computer ‘out of the way’ and provide people with a much more direct-tangible-interac on experience.”11 Experiment 01: inter[ACT] 1.0

Figure 6. inter[ACT] components: gridded board, colored acrylic cubes, acrylic led container This first experiment analyzes the criteria of the “Radical Atom” through the use of material proper es to transform, conform, and inform data through the three dimensional aggrega on of modular units. To measure this, the light distor ng quali es of clear acrylic material is used to create a visual system of quan ta ve measure for interac ons of physical objects. These physical units are used to think about the informa on that can be revealed to the user through their interac ons, and to explore the mechanics of tangible interfaces. Using a set of cubes with two different proper es; a hollow cube containing a led and ba ery is used to project light, and solid acrylic 11

Dourish,16.

18

cubes that is colored in either yellow, blue or red are used to manipulate the projected light. Individual cubes are not materially malleable like the “Radical Atoms�, but the aggrega on of the units allows units to interact together and transform dynamically with limited forms of transforma on.

The shape of the cube is chosen for this inves ga on because it has the most regular shape, and it is the most densely aggregated shape. It can be stacked one unit ver cally or horizontally in any direc on with the need for support. You would think that the triangular uniform prism or platonic solid would be the simplest module when construc ng geometries, since all geometries can be constructed through triangles.

The cube actually uses less surface area to build the same form, where two 5-sided geometries are needed to form one 6-sided geometry. The density of the aggrega on reduces light bleed and provides consistent measure of ac vity as the user adds or removes units from the aggrega on. The geometry of the cube also allows for regular transforma on in the X, Y, and Z axes which allows for more rigid evalua on of each user transforma on.

Data is collected through a series of itera ons by several users, and images were recorded overhead and obliquely using two webcams at a frame rate of 60fps.

These images were then later processed and used to analyze the spa al and temporal rela onships within the system of coded cubes as generated by each user. This is intended to be an analog model of how we interact with objects spa ally and the types of rela onships that are created through user ac ons of moving, placing, removing

19

objects in space. This also serves as a genera ve model of user interac on where objects have pre-defined parameters and goals of interac on.

Each cube is also coded with physical proper es of transparency, shape, color, light, reflec on, and diffusion which establishes its own individual iden ty as an object.

The lights and colors that are coded into these objects, are used to expose the spa al and temporal rela onship between the units act as a form of data visualiza on. The transparency and diffusive quali es of the material allows light to bend through the object, crea ng a finer spread of light as more units are aggregated against each other.

This informs the user of the density of the units. Light is cast three dimensionally but light output decays from the center of the emi ng bulb, this suggests an orienta on from where light is projected most strongly. The adjacency of objects informs the user of poten al future outcomes as more objects are aggregated. The temporal rela onship between the cubes is created through the user interac on. As the user adds more cubes, specific pa erns are generated through the adjacency, orienta on, and density.

These block are currently just firmware, not connected to anything in the digital world. The change in colors and filtering of light highlights the spa al rela onship of the cubes. This is a result of how the objects are coded through material, ligh ng, and color proper es. Each cube is encoded with same basic material proper es but individually encoded with either color or light. The objects are hard coded with these proper es and can’t be easily changed. These objects are not able to be ‘so ’ coded with informa on because the hardware doesn’t allow them to be encoded with so ware. They are also

20

Figure 7. Shape study with various aggrega on pa erns for each geometric unit

21

not connected to any infrastructure to allow it to communicate with other systems. It operates as a standalone tangible interface independent of any other computer systems and interfaces.

This tangible interface has no ‘digital screen’, but the aggregated units act as a display that exhibits human interac on as data. Tangible interfaces are more of embodied interface which allows you to perceive data not only by the sense of sight, but engages the sense of touch and sound as well. This gives users a be er understanding of spa al condi ons through sensory awareness. The user able to reflect and proceed in generate data at their own pace where the objects rest in a sta c state un l interacted with. The human ac on allows cubes to be reconfigured, and it is also the condi on that allows new data to generate. The data that is stored in the configura on of these blocks is human interac on.

“Radical Atoms” suggests that the direct connec on of material proper es to immaterial proper es of data will be be er way of informing users of system affordances. Interfacing with a material that is malleable, users are able to see, touch and mold the data that is coupled with the physical material. This hypothe cal material is envisioned as “digital clay”. Hard data that is not reconfigurable would be represented through physical material hardness. The experiment with acrylic cubes shows that materiality hard objects can also suggest of reconfigurable immaterial data through the reconfigura on of objects in spaces. The coupling of data with physical materials imply interac ons within physical space are no longer just influencing how we are perceived

22

23

Figure 8. Spa al proper es of aggregated physical objects in the form of orienta on, adjacency, and density.

24

in physical space. We act in space depending on how we want to be perceived. With the coupling of digital and physical objects, we are no longer discretely ac ng in one space or another, the data that is created or manipulated virtually is ed to the ac ons in physical space. What are the implica ons of connec ng both spaces together through singular ac ons?

Case Study 02: Coded Spaces and Datascapes The first experiment, inter[ACT] 1.0, defines the criteria needed to create embodied interac on through tangible interface with digital space and media, but it doesn’t define a way of contextualizing tangible interfaces with digital media or space.

Tangible interfaces and embodied interac on must be contextualized with our current data networks, which is not just the forms of communica on in which we connect with each other, but where data generate by each users interacts together. At this point, there is a realiza on that there needs to more research into how coded spaces or datascapes operate. In addi on to the survey of objects in Rob Kitchin and Mar n Dodge’s “Code/ Space”, they also surveys several exis ng coded spaces and describes these spaces in great detail in their book. Kitchin and Dodge have defined coded spaces as “a space that is transduced by so ware, but is not dependent on so ware to func on as intended. If the so ware fails, the space could s ll be transduced, but not as efficiently, effec vely, or produc vely as if the so ware had mediated the process.”12 Not only is important to understand how digital data is derived from our 12

Kitchin, 262.

25

interac ons with objects in space and each other, it is important how data moves within systems and how they transfer to other systems. This second study analyzes and iden fies key points of interes ng interac on between the human body and data correlates these points with the human body and its interac ons with other people and objects. In this study, diagramming is a method used iden fy and illustrate key points in several systems of interac on where the body acts and data is generated, modified, or removed from the system. Several case studies of systems of interac on are selected from “Code/Space” and other case studies are selected through personal observa ons of daily life in Crosby Hall. The itera on of diagramming and iden fying key points of interac on in the processes of each system is used to clearly understand each step in the system of interac on and the resul ng data transac on that occurs at each step. Each system of interac on is analyzed from the three different types of data transac ons that can occur at each point of interac on where the user of the system is able to add data, delete data, and modify data.

Each case study examines the systems of interac ons from the perspec ve of data exchange between user and the system. In each of these case studies, the mode of interac on is to act upon some form of coded objects which allows them to act upon coded space. Ac ons on these coded objects are o en required for interac ons with these coded spaces, which automa cally opts the user into the system. Without ac ons, the system doesn’t perform, and automa cally opts the user out of the system. Some

26

systems of interac on allow you to opt in or out at any point, and other systems have a more rigid opt in or out structure where the process of interac on doesn’t stop un l the system allows for it. The system is able to perform these same transac ons on user data, and some system will allow users to perform some of these transac ons on other user data. Usually the data transac ons are between users is limited due to privacy and ownership of data. This study resulted in a series of diagramma c interac on maps that describes data flow corresponding to user interac ons.

27

HVAC Systems and the Human Body An example of an exis ng system which is dependent on embodied interac on for the system to perform is the HVAC that is found in most buildings. To regulate temperature, the body relies on the skin and the nerves underneath the skin to perceive temperature in the environment. As temperatures rise, the body responds by swea ng, and this alerts the individual that he/she is ge ng hot. This individual has the ability to act on her bodily response and change the climate around her by changing the temperature in the hea ng system through digital or analog interface of a thermostat or a window. If the individual doesn’t respond to the body, more sweat will be produced un l the body cannot tolerate anymore and parts of it could fail to func on correctly due to dehydra on or heat exhaus on. Most individuals will not allow their bodies succumb to such extreme condi ons and modify their environment based on their bodily responses. The body operates similarly in cold condi ons, where the body can develop shivers and goose bumps to react to the cold. This bodily response no fies the individual that their body is cold through goose bumps and shivers. In an enclosed space, the individual could turn up the hea ng or close windows. As the body temperature regulates, the bodily responses subside. The bodily responses provide an embodied way for individuals to interact with the HVAC system.

28

Skin response Sweat

Skin response Goose Bumps

Figure 10. Diagram depic ng the possible environmental condi ons and skin response to the environment

29

Airport Check-in and Embodied Data In the scenario of the airport check-in, the process actually starts when an individual purchases a cket. This process is intended to verify the passenger and their luggage in an eďŹƒcient manner un l they board the plane. There are three hierarchical steps within this assemblage that ensure that the passenger sa sfies the condi ons of each step before con nuing to the next step. The boarding pass is encoded with me of purchase, payment, flight i nerary, and iden ty of passenger. This item and a photo id, such as a driver’s license or passport, are used to verify the passenger at all three steps of the check-in process.

30

Figure 11. Data flow diagram depic ng the squen al flow of informa on during the airport check-in process.

Once an individual arrives at the airport, he/she checks-in to retrieve the boarding pass and to check bags. The a endant at the check-in counter also checks the passengers against people on the no-fly list, if the passenger has similar creden als to any person on this list then the passenger is detained un l the passenger can verify his/her iden ty. A er receiving the boarding pass, the passenger moves to the security checkpoint. If there had been any suspicious ac vity such as buying a cket at the check-in counter or paying by cash, there might be a selec ve screening code located on the boarding pass. This informs the security agent that this person may have performed suspicious ac vity that is out of the norm. If the passenger fails

31

Figure 12. Data flow diagram depic ng the electronic banking system from a distrubuted system. the security screening, the passenger may be detained as well. Once the passenger has made it through security, he/she is verified once more at the boarding gate. A er verifica on, the boarding pass is scanned and the status of the passenger is updated.

Personal Banking and Embodied Data The banking system is a nested system where an individual’s account can be accessed in a variety of ways, simultaneously. The ATM is a tangible interface which requires an ATM card that is encoded with a magne c strip. The act of swiping the card through the card reader retrieves the accounts that are associated with a card via the bank server. This requires the ATM

32

machine to have network access so that it can connect to mul ple banking databases. Once the accounts are retrieved, the user must confirm ownership of that account by inpu ng a personal iden fica on number (PIN). A er the card and pin are verified, the ATM machine grants the user access to a variety of transac ons such as withdrawals, deposits, and transfers. The user is then able to make the necessary transac on to the account.

At the same me, the account can also be accessed virtually using network access via an online banking system or secure payment en

es, such as Paypal. The online banking gives

you similar access, but instead of using an ATM card, the data is accessed using a username and password that is associated to the accounts. A er the username and password are verified, the user may engage in transac ons of withdrawal, deposits, or transfers. Paypal uses its own

33

system of transac on tracking to secure payments and money transfer, but the accounts are coupled with bank accounts to allow for ease of transac ons. Triggering a transac on on a Paypal account, subsequently triggers a transac on on the bank account coupled with the Paypal account.

Boundaries of Data In some of these case studies, there is a dis nct boundary between physical and digital space. This typically occurs when the interface is a screen. The screen typically occupies most of our field of vision, and most screen based interfaces have audio interfaces as well. With the screen occupying two of our strongest senses, how can we not be distracted from the space that we occupy? As soon as we engage with a screen device, the screen becomes the main focus and the ac vi es in the space that that we occupy fades into the background. This experience creates a seemingly clear dis nc on between the two worlds, physical and digital, where you only occupy digital space when you are engaged with the screen. In reality, the body is s ll occupying the world through the sense of touch, taste, and smell.

Not only is there a perceived physical boundary between the digital and physical world, but there is a temporal boundary between the two spaces. In physical space, me is perceived through the changing ligh ng situa on throughout the day. Even on dark and gloomy days, a passing day can be perceived through transi on from a fairly well lit world to a dark world, due to the movement of the sun. In digital there is no such environmental measure of me, only a standard measure of me that is based on the rota on of the sun. Unlike the changing ligh ng condi ons in the physical environment, the spent in digital space is not as easily perceived as

34

Time Stamp

Message Lorem ipsum dolor sit amet,

Your deposit was successful.

Do you have any stepper motors?

Sender

Location

2-20-2013 12:43:05

Problem with connection?

kyuen3@buffalo.edu

lasko25@gmail.com

Inbox

2-15-2013 06:30:02

UB Card Transaction Receipt

kyuen3@buffalo.edU

UB Card Office

Inbox

2-23-2013 16:03:02

Recipients

Subject

RE: Stepper Motor

Peter Schmidt

Sent

kyuen3@buffalo.eud

Date

Print Name

Print Balance

Date

Print Name

Print Balance

08-24-2012

CC

50.00

2/21/2013 5:48:04

CAMPUS CASH

$10.00

2/21/2013 5:46:53

DVDs

-3.96

2/18/2013 3:18:39

BANKING.AI

-.60

2/18/2013 3:17:28

EXPLODED.AI

-.60

UBLIB PRINT ACCOUNT

ARCH. PRINT ACCOUNT

EMAIL

Username

Password

Email

Birthday

Sex

kathy.g.yuen

hdfuie4354

kyuen3@buffalo.edu

04-29-1987

Female

GOOGLE

Message

Time Stamp

Lorem ipsum dolor sit amet,

2-19-2013 10:11:05

Subject

Recipients

3/16 is grandma's 80th birthday

kathy.g.yuen@gmail.com ,herbert.z.yuen@gmail. com

Sender

Location

People

Circle

allenyuen1958@yahoo.com

Inbox

Alice Chan

Friends

Jeff Jackson

Family

Sharon Li

Lorem ipsum dolor sit amet,

2-25-2013 12:30:02

Lorem ipsum dolor 2-25-2013 16:03:02 sit amet,

force-directed graph

kathy.g.yuen@gmail.com

oriane.daley@gmail.com

Minestrone

all recipies listserve

email@allrecipes.com

Keywords

Timestamp

flu, symptoms

01-18-2013 22:10:20

home, remedy, flu

01-18-2013 22:13:26

Mike Kirschner

Friends

Silvia Lee

Friends

embodied interaction

02-18-2013 15:54:52

media artist

02-18-2013 10:34:32

Account

Onlin line Transact tion i

Campus Cash

2 00 25.00

Campus Cash

-1.00

Campus Cash

-1.00

V C Crosby 101 Snack

V Crosby 10 01 Snack

01-27-2013 12:16:19

Inbox

Amount

UB CA ARD ACCOU UNT

GOOGLE+

EMAIL

Locati tion

01-25-2013 01 25 2013 15:28:37 01-25-2013 15:30:19

Friends

Inbox

Date

GOOGLE SEARCHS

Name

Owner

Date Modified

Electronic Research.xlsx

Me

03-11-2013

GOOGLE DRIVE

Username

Password

Email

Birthday

Sex

kathy.g.yuen

hdfuie4354

kyuen3@buffalo.edu

04-29-1987

Female

FACEBOOK

Images

Tags

/me/photos/dfgd.img

Yuen

Comments

Reza Tashakori

None

User

Time Stamp

Likes Nima Vk

02-09-2013

Kathy

55 Annette Dr.

Full Time Student

32195725

Arch/Pln Comb Masters

kyuen3

3242034203

5081280303622855

Friends Nick Brusica

Jose Pesantz

Jordan Geiger

Albis Del Barrio B.

Adam Laskowitz

Nima Vk

Peter Schmidt

Peter Schmidt

Will Sedig

Will Sedig

Karen Zhu

Kathy Yuen

Boo ok k N Name

FACEBOOK K FRIENDS FRIE S

FACEBOOK POST

Au utho or Yona Fried dman

Pro o Domo

Librar ary

Location

Health Scienc nces

Date Due

NA2500 .F75 2006

2-20-2013

Fine Amount 5.00

LIBRARY

Date of Bir Birth

Birth Country

April pril 29,1987

United States

Birth State

Martial St Status

Military Stat tatus

Citzen ens nshi hip hi

National Nat ti I ID Typ Type

Nat tion t iona al ID #

United ed d St ta tates

Soc oc ci ci ial Secur urity N Nu um mb ber r

032 32 2-32-231 11

Personal Information

Date

Descript tion

02-15-2013 10:40 0:02

Library Fine

Fall 2012

C Charge

5.00

01-2 27-2013 10:2 20:22

Payment on Accoun unt

Spring 2013

Pay yment

8,451.00

011-22-2013 10 0:20:21

Campus Life Fee

Spring 2013

Charg ge

31.25

0 01-22-2013 10:20:20

College Fee

Spring 2013

Charge

42.83

Term

Transaction

Balance

STUDENT ACCONT Last Visit

Reason

Diagnose

Shots s Up to Date

Health Insurance Provider

Insura urance Group #

Insurance Policy #

02-08-2013

Cough, Fever, Runny Nose

Flu

Yes

OxFord Health Insranc ance Group

OH3242

K3242-001

HEALTH RECORDS

Polio

Tetanus

Yes

Yes

Measles Yes

Menigitis

Hepatitus tus B

Yes

Yes

VACCINES

Term

Type e

Fall 2012 12

GSEU EU Fee Mitiga gation - Fall

Award Amount 52.22

Status

Disbursement Date

Filename

Path

Accepted

08-28-2012

02_Kathy_Yuen-Pro ject2.ai

ubfs.buffalo.edu/ubfs/myfiles/k/y/kyuen3/windows/userdata/

02_Kathy_Yuen.dwg

ub bfs.buffalo.edu/ubfs/myfil iles/k/y/kyuen3/windows/userdat ta/

file.file

ubfs. s.buffalo.edu/ubfs/collab/Ar ArchPlan/share/courses/ARC C606-3.s

FIN INANCIALL A AID D

UBFS(file server)

Course Name

Course #

Room #

Time

Term

ARC 518LEC Design Methods

000-LEC(20867)

Michael Rogers

HAYES B 07

We 10:00AM - 12:40PM

Spring 2013

ARC 606LAB - Arch Design Studio 6

3-LAB(19543)

Mark Shephard

Arr

TuTh 1:30PM - 7:30PM

Spring 2013

ARC 699TUT Masters Thesis

SHP-TUT(11669)

Mark Shephard

Arr

TuTh 1:00PM - 5:50PM

Instructor

Semester SPRING 2013

Course # ARC 699TUT

Course Name

Credits

Grading

Masters Thesis

7.00

Letter Grading

SPRING 2013

DMS 598TUT

Project Supervision

2.00

Letter Grading

SPRING 2013

DMS 700TUT

Thesis Guidance

2.00

Letter Grading

GRADES

Grades

Discussion Board

Grades

Documents

Courses

UBLEARNS

CLASSES

Figure 13. Embodied datascape from my own personal interac ons within the University at Bualo interconnected with external systems of Facebook and Google.

35

not all applica ons have a real- me clock or large environmentally perceived change in me but a small clock on the corner of the screen.Another condi on of unperceivable me in digital space is caused by the inability to dis nguish me zones. In physical space, people are localized to a region and me cycle based on the sun’s movements. Time in digital space is not provided by environmnetal awareness, but a system clock located in the perpherial vision of the user.

Digital space breaks all temporal boundaries by allowing users from any me zone to be connected at the same me. There is no perceivable me scale while interac ng within digital space, as it seemingly operates independently from physical space. As half of the world is sleeping, the other half of world is awake and has the ability to interact with other user data asynchronously. An example of a system that allows for asynchronous transac ons is the banking scenario, where direct deposits such as paychecks can be automa cally made periodically with the need of the account holder’s acknowledgement. While there are dis nc ve boundary condi ons surrounding our experience of digital systems, data doesn’t always remain in their respec ve realms. Data can be carried by the user from world to world, causing an entanglement of data.

Entanglement of Data While so far we explained the two worlds of data and their boundaries, we will now look into how these two worlds “influence” each other. One world influences the other in the form of sending data to it, or influencing in the form of ac ons. Entangled data has no direct influence or ac on, it is only data that has been overlaid from one world to the other. This

36

allows the user to take data from both the digital and physical world to make decisions and take ac on.

One common example of entangled worlds is the rela on between human body and sensors arrays found on smartphones. The former falls under our defini on of physical world, while the la er conforms to the digital world. In this scenario, at any given me, the physical world is sending data to the digital world, which then stores this data for any further ac on. All cellphones have GPS (Global Posi oning System) devices, but smartphones allow users to access that GPS data to post that data via apps like Facebook, Twi er, and Foursquare. This way, we can no fy others in digital space, where we are located physically in the world at any moment. This process of pos ng data gives the user the ability to share their physical presence or self with their digital self. This is a one way transac on of data from the physical self to the digital which allows the user to shape and mold their digital self with parts and piece of their physical self. Every person shares and posts various amount of data from their physical world. Some digital selves have no relevance to their physical selves, but the digital self is s ll cra ed with concrete content from the physical world, whether it be images, video, or wri en thought. On the other hand, the no fica ons sent from the digital world services such as Gmail, Facebook, and Twi er keeps us constantly informed of the ac vi es in the digital world but of possible relevance to the physical. We call this entanglement because the physical world is not ac vely involved in any of these ac ons, but is kept aware by the digital worlds.

37

Another form of entanglement that is found in these case studies is the ability for data to move between systems and spaces, as well as interact with data produced by other users. Data is never deleted or removed, just transduced into other forms and connec ons to make it meaningful. The scenarios described in Code/Space suggests that data that captured is contained within the system and isn’t used elsewhere as suggested in the Banking and Airport Check In system. Data that is created in both systems are restricted locally in the respec ve systems, but it doesn’t only belong in a single space or system as described in Figure 13; it is able to move from system to system, space to space, and interact with other user generated data. Datascapes aren’t singular systems found in Banking or the Airport check-in but a landscape of coded spaces that are interconnected with data that is shared between coded spaces. Kitchin and Dodge describe a coded assemblage as “a confluence of several coded infrastructures and their coded objects and processes wherein they become integral to each other in produc on of par cular environments, for example, office complexes, transport systems, and shopping centers”13 , which is similar to datascape, but is localized to a specific space.

The spaces describe within Code/Space are localized to specific systems because they are systems of efficiency and follow rigid processes. While there are rigid processes found within the datascape described in Figure 13, the most interes ng point of the diagram is the point in which data intersect with other data in different types of systems, and with different user generated data. This data suggest different social rela onship 13

Kitchin, 261.

38

between the connected users.

Transgression of Data While there entanglement of data, there is s ll the inability to directly act between spaces. What is not yet found in any of these case studies is the transgression of data, where data fluidly moves between the two spaces and the ability to directly act from one world to another. While there are digital interfaces that usually consist of a screen and keypad or keyboard interface, our physical objects cannot directly generate digital data or produce ac ons in digital space. While there are exis ng coded objects, they are only interact able in the localized system in which they belong.

Experiment 02: inter{ACT] 2.0 Building on the various research of trajectories surrounding tangible and embodied interac on with objects in space, inter[ACT] 2.0 proposes a socially mediated experience that augments the physical and digital space through the transgression of data. This method of inquiry takes a tradi onal route of the ground up approach design methodology through the itera on between usage scenarios and prototyping to develop a new system of interac on to enable the transgression of data and facilitates embodied interac on. Since no system of nature exists, the system needs to be developed from the ground up. This method of inquiry begins with analysis of everyday objects and their current usage models are compared to the study 001: inter[ACT]1.0 to find similar spa al quali es that were assessed from that usage model.

39

The chair in a social context related most closely to the usage model found in inter[ACT]1.0, where the chair is an object used to assist users in social ac vity in space through the adjacency, orienta on, and density of aggregated sea ng. Tradi onally, chairs have been designed by architects as a way to resolve design problems at a small and more manageable scale which s ll relates to the body. Several famous architects, Mies Van Der Rohe, Charles and Ray Eames, and Le Corbuiser have designed chairs as aesthe c, structural, and formal studies. In this thesis project, the chair serves as tool to study interface design in rela on to social interfaces. The Oh Chair is a designer chair that has been specifically selected for this experiment because is it a chair that is abundantly found in the School of Architecture. More importantly, the characteris cs of the chair lend itself to be the ideal candidate for this experiment because the form of the chair doesn’t allow for the occupant or user to be oriented away from the chair.

The Oh chair is an ideal objects which allows the body to be easily tether the object and the data produced by interac ons with that object. A digitally augmented chair allows the engagement with the physical body in both physical and virtual space to be tested simultaneously. The architecture and social space of physical space is tested against the architecture and social space of digital space to understand where there could be interes ng intersec on points and crossovers between the spaces.

40

Figure 14. The typical aggrega on of chairs found in various types of architectural programs and socially structured spaces.

Scenarios The scenarios developed are based on the typical aggrega ons found in architectural programs: lecture hall, classroom, group mee ngs, and offices. In architectural, the program is the social se ng defined for a specific space. The program is not usually defined by the space itself, but the arrangement and the physical objects located in that space. Without these objects, the space would be an open plan which is a generic space that allows for various programs. The chair is typically the main piece of furniture that defines the use of a space.

“Physical environments are arranged so as to make certain types of ac vi es easier or (more difficult), and in turn, those ac vi es are tailored to the details of the environment in which they take place.”14 14

Dourish, 19.

41

Figure 15. Usage scenario: group session - using health data, orienta on senors, posi onal sensors, and current flow to monitor health and the spread of germs.

Figure 16. Usage scenario: work space - a space where mul ple people can work on documents and collaborate together.

Figure 17. Usage scenario: lecture hall - where there is poten al to meet and mingle with new people.

43

Based on the typical arrangement found in such architectural programs, so ware programs are developed to augment the use of the chair in the prescribed social situa on.

In each of these scenarios, the architectural program and augmented experience is defined, and the data that needs to be collected and reported is iden fied to allow for the digitally augmented experience. This project doesn’t intend to change the interac on with physical chair within any of these spaces but it adds a digital layer which allows users to interface with digital space through the chair, where the physical object becomes the digital interface.

The scenarios are developed surrounding key concepts derived from the precedent and case studies. These concepts are then examine to find common capabili es and needs of the individual systems to resolve a design that will be able to support more than one usage model.

The common capabili es between usage models find that it will be necessary that the object will have some raw data about posi on, distance, and orienta on which is a similar results of the experiment, inter[ACT] 1.0. The data associated to the object needs to be coordinated with the user of the object and other data a ached to the user from the various systems that they’ve interacted with. The objects should also be able to communicate this data with each other, and the users should also be able to communicate to the object and through the object to other users. This model of interac on s ll enables the user act through the object to provide inten on to other users of the space through the objects, but also allows the object to access data about users that is localized to the space and the situa on.

44

Figure 18. Common capabili es needed across the various developed scenarios.

Technological Aordances This sec on iden fies exis ng technologies which can be used to develop a new system based on the developed usage scenarios and the commonali es found between usage models.

Several dierent components are analyzed within each technological capability to iden ty which components will be serve the design of this new system of embodied interac on.

Based on the conceptual design and the precedent studies, various digital components are explored and evaluated as poten al components to allow for certain capabili es based on the needs of the system, in order to turn everyday objects into digital compu ng devices.

This employs an itera ve process to test the func onality of the technology against system necessi es to determine how certain technologies can facilitate parts of the system. Machine

45

readable objects, communica on between objects, loca on and orienta on awareness, and communica on to users are the most basic capabili es required to develop each scenario.

Machine readable objects In order for objects to communicate with the system, they need to be machine-readable.

There are different levels of readability that are available to coded objects which is dependent on the type of hardware embedded in each object. Not only should each object be machinereadable but they should be able to read as well, allowing two way communica on between objects. Technology such as magne c strips, RFID, MIFARE, QR codes, and barcodes allow the other machines to read the object augmented with this technology but doesn’t allow the reader to communicate back to the object. It is a one way communica on from the object to the system. Technology like this doesn’t just serve as means of communica on but it also gives the coded object an iden ty. The iden ty of these objects are associated to the user or owner of the object, but this iden tyalso makes them addressable. If the objects had addi onal means of communica on which allowed them to receive informa on, the system would know which object to address because the iden ty of each object is known through a unique iden fier embedded on the object, usually a unique series of numbers. Some mes this unique iden fier is not a sequence of number or le ers but an image. QR codes and bar codes are image based iden fiers which represent a sequence of numbers. These are read in different ways with specific types of hardware. This same type of technology that makes physical objects machinereadable, can also make the users machine-readable as well.

RFID stands for radio frequency iden fica on, and RFID tags consists of a simple circuit of an antenna. Tags are end devices that can only broadcast, and readers are able to detect and query tags. Most tags are passive and don’t require independent power. These tags can be

46

read from a distance through a reader with a radio transponder that is tuned to the frequency of the RFID. This reader will ac vely detect is a tag is range and if finds a tag with the same broadcast frequency then it query or ask the tag to report some data. Ac ve tags that have ba eries within the circuit don’t need to be passively read but ac vely broadcast con nuously.

Tags are limited to amount of data that can be carried. They are a really small form factor which can be found in the form such as cards, key chains, and s ckers. The range of this tags vary, but typically they have a range of a few cen meters to a couple of inches. Ac ve RFID cards typically have much larger broadcas ng range because it has a ba ery powered antenna crea ng an ac ve broadcast signal.

MIFARE is advanced type of RFID where it uses a special band of radio frequency at 13.56 MHz (HF) and o en called “Smart Cards”. MIFARE also works with tags and readers, but the tags are able to store more informa on with the maximum of 4 kilobytes. These tags are burned with a unique iden fica ons number like RFID tags.

Unlike normal RFID tags, the MIFARE tags are able to be read and re-writeable. Not only can these can be read but they append, delete and modify the data stored on the cards. Not only do they have the ability to store more data, but they usually have a larger range then regular RFID tags. The minimum range without interference is typically couple of inches, and because of the larger detec on, they o en known as proximity cards as well. This type of RFID tag o en used on transporta on systems to pay for fares. Using Near Field Communica on(NFC) readers, the MIFARE cards are able to pass data without contact.

Magne c strips are typically found on cards, and are o en called swipe cards because

47

it requires the user to swipe a magne c strip card through a slo ed reader. Like RFID tags, magne c strip cards have rela vely low memory and can’t store too much informa on. Objects with magne c strip require physical contact with the reader unlike RFID and MIFARE which are contactless and rely on proximity. The user needs to ac vely engage with object in order for it to be read. Magne cs strips are typically found on credit cards, ATM cards, driver licenses, and access cards.

Bar codes is another passive means of transmi ng data. They are represent number and le ers in the form of ver cal parallel lines that vary in width and spacing. These require a special reader, o en called a barcode scanner that can scan the code and decode the image.

Special hardware is no longer required as most devices have a camera that can capture the barcode image, but special so ware is s ll required to decode the image. Most consumer goods and produce have barcodes that can be use retrieve product informa on. One barcode is used for each specific model or product.

QR codes are another form of op cal machine-readable tag. Informa on is represented in the form of a two dimensional field. It has a similar read me as the bar code and can store more informa on than the one dimensional bar code. It can be read by any imaging device such as a camera but also requires special so ware to decode the informa on stored in the pa ern.

It has go en quite popular with consumer adver sing. It can be found on ads and products and it o en re-directs the user to a website with more informa on about the product on a user’s smart phone.

These new forms of technology that makes objects more machine-readable also allows

48

for new rela onship with data to be formed between user and object. The data embedded is these coded objects is not the only data that can be read. The system also logs when and where the objects have been used. By sor ng and analyzing the logged history, the system can make inferences about the object’s usage. With the ability to connect with dierent hardware and coded objects, systems can make inference about rela onships between dierent objects.

Why are they interac ng together? Are they interac ng together at specific mes? These design ques ons develop because embedding technology such as MIFARE, allows systems to have 2-way communica on with objects that are usually sta c and non-communica ve. Not that our objects can talk to each other and us, we need to think about new connec ons and rela onships that they are forming with us in terms of data mining and analysis.

Posi on and Orienta on Systems A few posi onal and orienta on systems are tested to provide absolute posi onal and orienta on data. This new system requires the objects to accurately report their posi on and orienta on, in order to quan ta vely calculate spacing and orienta on from each object to each other. Using distance formula to measure distancebetween objects and angle of coincidence to measure orienta on of objects from one another, the system could accurately perceive the spa al spacing and orienta on of each object. While these formulas could be used, accurate data needs to be gathered and reported from each device.

AHRS stands for Al tude Heading Reference System. These systems typically use integrated sensor data from accelerometers, magnetometers, and gyroscopes to calculate

49

heading and al tude. Each sensor addresses three degrees of movement. As more sensors are integrated, the more reliable the orienta on data becomes. If the raw data from any of the three sensor is unreliable then the integrate data will be unreliable. These AHRS systems don’t address absolute loca on, only rela ve loca on and orienta on. GPS is need to generate absolute posi oning data.

GPS/Indoor GPS is a global posi oning system. Global posi oning systems rely on mul ple beacons to ping the device to triangulate the signal. Most car naviga on systems depend on at least ten simultaneous connec ons to satellites to triangle the device’s loca ons.

They are o en unreliable indoors because of interference and don’t provide enough granularity to pinpoint device that are less than a hundred feet from each other. Indoor GPS is being developing around Bluetooth and WIFI technology. These indoor GPS systems would use low power Bluetooth beacon or wireless access points to ping connected devices to triangulate objects indoors. Currently there isn’t an open source package that enables indoor GPS, and most of the technology produced is proprietary.

Marker tracking relies on computer vision infrastructure depending on viewing angle, aspect ra o and viewing area. The NyARtoolkit is an open source so ware package that provides reliable marker tracking. It is a robust system that reads the perspec ve of the glyph in order to view digital models at the same perspec ve crea ng an augmented reality experience.

In order to overlay digital models on the physical glyph, the so ware packages provide access to marker data to derive posi onal and orienta on data.

50

Modes of communica on The sec on on machine readable objects already discussed physical objects can be acknowledged by digital systems, and how users can be acknowledged by the digi zed physical objects. The mode of communica on that will be addressed in this sec on is communica on between objects, and posi onal and orienta on data from the previous sec on can be posted between objects. This important because objects should be able to address each other and the system to coordinate data. While the users and objects are acknowledged, only so much data can be read using that technology. A method of communica on allows for local data that is read, stored and processed to be shared with other objects with the same communica on capabili es.

Serial communica on is the process of sending bytes and bit one a er the other.

The receiver must have a buer as it receives one byte at a me as it sent. This type of communica on is usually tethers a device to a computer via a cable.

Bluetooth is a method of wireless communica on. These devices are usually used to serve personal area networks, and connect devices that are rela vely close to each other. They are used to send short range messages, usually within 100 feet. It operates the frequency of 2.4ghz on the ultra-high frequency band of ism. Bluetooth devices usually need to be paired or synced, allowing for a mesh network to form WiFi is another method used for wireless communica on. It is an infrastructure that supports internet protocols and is typically used to access the internet. It can also be used to service a local area network. Wifi network can also be extended and repeated, so network can

51

grow or contract depending on infrastructure and placement of devices. Wireless networks are usually a star typology and authen ca on are stored on the device so that devices can automa cally connect to access points that are in range.

Actuators Without a screen, and only using the chair as a display of data, actuators are needed to respond to users and acknowledge their presence. In this system, actuators would replace the screen and simulate the same experience provided by a screen. A series of dierent actuators are tested to see which provides the most legible interface. This sec on looks for the most engaging and yet least distrac ng user interface.

Sonic Feedback is the process of genera ng audible response to the ac on of the user.

Sonic feedback can be found in the form of several tones like a busy signal on a phone. It can also be actual voice responses where there is a spoken message usually using text to speech technology.

Hap c Feedback is the process of genera ng a response which can be felt. The most common hap c feedback is found on the cellphone where no fica on are sent to the user via vibra ons. This can be when new messages are received or calls are missed. On smartphones, they can no fy users when there are message and no fica on from other applica ons other than the phone.

Visual Feedback is common form of feedback in most human computer interac ons. It

52

can be found in the form of screen technology and ligh ng. Most of the analog world is sensed through the eyes and the digital world is mainly visually sensed as well.

Ini al Prototype This ini al prototype implements the findings previously discussed in the sec ons, Scenarios andTechnological Aordances, to create a system that allows for embodied interac on with data through the method of transgression. The scenario sec on setups usage models and iden fies technological capabili es that are needed for the usage scenario to func on as prescribed. The technological aordances examines exis ng technologies to discover if it can be built, which found to be true. Using these exis ng technologies, this project aims to augment the physical experience of si ng, not by changing the experience of si ng by allowing the social physical rela onships to exist in the same way in the digital world.

Hardware To understand if the chair could be turned into a digital interface, the first ques on that was asked is how can the person si ng in the chair be iden fied? A pressure sensor on the chair could be used to assume that someone was si ng on the chair, but it wouldn’t iden fy if it was a person, dog, or bag. At the same me, sensors that require users to ac vely engage with the chair would change the exis ng experience. To really engage the body, the chair must passively sense the iden ty of the person and their ac on of si ng.

To enable the tracking of the chair, the computer vision library, ArToolKit, is selected to track markers on located on the chair. Using marker tracking solves several problems at the

53

same me. Not only does it give the object a unique iden fier, but the ArToolKit can be used to provide posi onal and orienta on data using the marker data. The markers of several size ranging from two by two, four by four, six by six, and eight by eight were tested on various loca ons of the chair. The largest marker was most easily and reliably tracked by the camera, while allowing for the largest viewable, and because it could tracked 11 feet from the camera. For themarker to be tracked, the marker had to be somewhat level and flat so that it limited distor on when viewed by the camera. Since the marker is just looking for a black pa ern within a white frame, noise would o en be generated from a background when it was mostly white or black. To reduce noise, a depth camera could be used to iden fy only markers at a certain height, although a depth camera was used, this algorithm was never used. An analog method of moun ng the camera overhead and using a floor color that would not generate noise was implemented instead. The placement in a high contrast environment allows the markers to be more easily tracked against the chairs.

54

Ul mately the eight by eight marker, located on the seat of the chair was provided the most reliable marker data. Integra on of the marker to the chair was cri cal, for the chair to act as an interface. Ini ally, spray paint was used to place the marker on each chair, but the paint would rub off the chair even though spray paint rated for plas c was used. A seat cushion is o en used in conjunc on with various types of a sea ng, and the fabric material allowed for the marker to be screen printed permanently. One issue with fabric seat cushions is that o en depress and loose its shape over me. The ini al prototypes of the seat cushion was made with various types of fabric ranging from so , co on and jersey, to s ff fabrics, denim and canvas, as well as various filling material such as foam and co on pile. The finalized design of the seat cushion is made with denim fabric and filled with co on pile. It is pocket styled with a large flap so that electronics components could hidden inside, such as the Arduino microcontroller with various sensor and actuators.

To determine the correct type of response or feedback, the medium that is chosen must provide a clear response to the ac on, as well as create a legible message whether is it sonic, hap c, or visual. If the message is not clearly and instantaneously legible then the medium is not a correct for the message that needs to be communicated to the user. To create an immersive experience, users shouldn’t need to take me to decipher the message, it should be instantly clear what the system is doing and reac on to the user. Not only were the different sensory devices tested on the chair for legibility, they were also tested in several loca ons to understand ideal placement of the actuators.

55

Figure 19. Tes ng simple pa ern recogni on with ARtoolkit to understand the limita ons of the tracking library.

A er tes ng several dierent types of actuators, the LEDs placed along the chair proved to be most compelling and subtle. It didn’t take away or make the current interac on with chair by making the interac on more complex, but it made the user aware that there was something happening as a result of their interac on with the system. Using sound was too disrup ve, crea ng a noisy environment with even a small number of chairs, taking away from the experience of si ng and talking to one another. Hap c sensors of vibra on motors also proved to be noisy, but it was also hard to understand what you were feeling. One small vibra on motor would cause the whole chair to vibrate making mul ple motors undis nguishable, this made it hard to use hap c sensors to relay any complex messages about the users and their interac ons.

Whereas the visible feedback of led lights were subtle yet bright enough to acknowledge that there was something happening in rela on to user interac on. As aggregated units, the visual eect of mul ple chairs, suggested boundary condi ons as the exterior outline

56

of each chair create a bands of light with the same color.

The final hardware setup consisted of an Arduino Decimilinove microcontroller connected to the PN532 RFID/NFC controller and 2 Darlington Transistor Arrays to control the RGB LED strip ligh ng. The PN532 and Arduino microcontroller placed in the seat cushion. The transistor arrays and RGB strip ligh ng is placed outside the cushion with lines running to the Arduino microcontroller. The RGB strip ligh ng is run along the chair and connects to the Darlington transistor array underneath the chair with a 12V ba ery to power the LED strip ligh ng.

Figure 20. Various actua on concepts. A) Chair sonic test B) Chair hap c test C) Chair visual test

So ware Hardware enables data to be collected and communicated, but so ware is needed to make sense of the data and give it meaning. There is a server that is using Processing IDE which collects the incoming data from the sensor network of the chairs and camera, processes to pass data to the correct receivers, and visualizes collected

57

data against the processed data. The so ware is developed to associate the data between the sensors to answer the following ques ons: Is someone si ng on the chair and who is that person? Is there are chair in proximity to another chair and is that chair socially aligned? This is accomplished by understanding the usages pa erns of the chair to develop metrics which can be reliably tested against.

Figure 21. The finalized placement of the markers, screen printed on a custom made seat cushion, with the ligh ng actuators running along the chair.

The act of si ng to most people is the act of placing the bu ock down to the seat of the chair through the bending of the knee which allows the chair to carry the weight of the body.

Typically, most people carry their wallet or cellphone in their back right pocket, and these two objects are used to store mul ple user iden

es. Feature phones have limited user profiles

stored on them, but cellphones are devices which can carry mul ple applica ons and their associated profiles. The wallet can also carry mul ple cards of iden fica on, each card typically

58

has an embedded magne c strip or RFID which can be read to bring up iden

es and data

associated with the card. The NFC reader that is placed inside the seat cushion simulates the reading of one of these devices that is carried everyday by reading MIFARE card that is placed in the back pocket. While reading of the MIFARE card could iden ty, if someone was si ng and who was si ng in the chair, but the state of the chair needs to be recorded so if the user le the chair. The status of the QR marker was used to iden fy the state of the chair. It was assumed that if the QR marker was tracked then it was lost that someone had sat down in the chair. If the marker wasn’t live or in the frame then came back into the frame, it was assume that someone was si ng on the chair and le the chair. The status of the chair needed to be verified with user to ensure that the chairs were in the correct state of use.

Quan fying proximal interac on Through empirical data of observa on and personal experience, there is realiza on that social alignment can be measured through the proximity of the chairs. If the chairs are a certain distance away from each other, it is safe to say that user is in a social proximity from another user. Typically, if a space is larger enough for the volume of people, then one might sit a seat apart from another to give the other person some personal space. To quan ta vely solve for linear distance between chairs, the loca on of the chair that was previously derived from the marker data is used to find linear distance between two chairs,

. Each chair is compared to each other to find sets of chairs

within a certain threshold of linear distance from each other, it is safe to say that users are socially aligned. But it was soon realized that proximity had less impact on developing social

59

alignment than the orienta on of the users. While distance is the first social cue that could be used to easily iden fy social alignment, orienta on of occupants could re-aďŹƒrm and isolate more specific social contexts, iden fying what type of interac on that they might be having.

Even if proximity is sa sfied, chairs within the same proximity might not be aligned through orienta on. Based on the review of architectural programma c analysis, there are four simple condi ons that can be used to iden fy the social alignment of people si ng in chairs. It can assumed that people that are facing each other are talking to one another, people that are facing towards each other but are not parallel are facing the each other and the same subject, and people that are facing the same direc on are facing the same subject. Any orienta on outside of these three can be assume that they are not socially aligned at all.

Figure 22. Si ng detec on and associa on of body with chair, when the marker is blocked and a RFID is safe to assume that there is a person si ng in that chair.

60

Figure 23. The derive metrics from the analysis of the architectural programs. This diagram also depicts how the simple metrics might handle aggrega on.

In this ini al prototype, the heading of the chairs derive from the raw marker data is used to solve for orienta on. Marker points are derived from the marker orienta on and the reference image, so the markers are oriented to the chairs would have a direct rela onship to the heading of the chair. The heading is derived from the two points of the marker using the arctangent with two arguments, atan2( marker[3].y - marker[0].y, marker[3].x - marker[0].x ).

This derived heading data is simplified to eight quadrants, so that it was pa erns of sea ng could be more easily iden fiable. (angle_degrees*8/TWO_PI) * TWO_PI/8 rounded to two decimal places is used in which quadrant of a circle would the heading be located if the circle was divided into eight sec ons. Using the result from the previous equa on, the quadrant in which the heading lies in is mapped the circle from PI to integer units, map(quadA, -PI, PI, 0, 8).

Socially aligned chairs and their occupants are evaluated based on the comparison of two chairs using an exhaus ve search method. Each chair is tested against each other looking for sa sfying distance and orienta on rela onships.

62

Figure 24. if ( (chair1.quadLoc == 4 && (chair2.quadLoc == 8 || chair2.quadLoc == 0)) || (chair1.quadLoc == 5 && chair2.quadLoc == 1) || (chair1.quadLoc == 6 && chair2. quadLoc == 2) || (chair1.quadLoc == 7 && chair2.quadLoc == 3) || (chair1.quadLoc == 5 && chair2.quadLoc == 1) || (chair1.quadLoc == 2 && chair2.quadLoc == 6) || (chair1. quadLoc == 3 && chair2.quadLoc == 7) )

Figure 25. Image of studio space showing distance and proximity analysis. When the chair falls within a certain threshold of distance, the space of chair, the chairs found to be social proximity to each other.

This was not an ideal implementa on, and didn’t afford enough granularity in data comparison to iden fy subtle angles of orienta on and the exhaus ve search is an efficient data structure to derive more than pairs of socially aligned chairs. The second prototype resolves this issue by finding the angle of coincidence between two chairs rather than just comparing a heading averaged to a quadrant and a graph structure which doesn’t rely on an exhaus ve search but associa ve array to store rela onships.

Connec on to Digital Space Once this social rela onship is digi zed, the so ware a empts to re-create that physical rela onship in digital space. The prototype a empts to translate the social rela onships found in physical space using Twi er, a social media applica on which allows users to post anything under 145 characters. It used to blog about life or talk about specific topics. Twi er allows users

64

to see each other post with the act of “following”. Following someone allows you to see all their post but the person that you are following also know that you are following them. But it is not mutual, following can be a one sided rela onship. In a way, following someone in physical space is very different, and o en mes it has a bad connota on that is associated with stalking and crime. But in digital space, the interac ons between users are controlled. Even though, users are basically stalking each other through the act “following”, it’s a controlled environment where messages can be report for spam, abuse, or bullying.Users can post anything they want and o en do, because they are portrayed as anonymous users, even though there are several unique iden fiers a ached to their iden ty, such as email and birthdate.

This usage scenario with Twi er is created to test the transgression of data between the two worlds, physical and digital. The associa on between physical space and digital space are enabled through the Twi er API (applica on programming interface), the a ributes of physical space are used to leverage the affordances of Twi er. To re-create the physical rela onship on Twi er, users are automa cally “follow” each other when they are found socially aligned, so their conversa on in physical space can also occur in digital space. The tethering of the two users in digital space is momentary, as soon as the physical rela onship changes, this connec on on Twi er is also severed. With this momentary connec on, the digital space takes on some of the characteris cs found in physical interac ons between people. In digital space, you are always connected and constantly ge ng updated no fica ons of user ac vity, but our analog connec ons are physical space are momentary and only occur when people come together in specific situa ons. In this usage scenario, the chair is physical object is transformed

65

into a digital object that can be interacted with physically but it also associa ng and changing digital ac on of “friending” or connec ng to another person through physical ac on rather than digital ac ons. Since “friending” is now associated with a physical interac on, it can take on the characteris cs of that physical interac on.

Result The completed prototype was shown for the first me at the final thesis review.

Previously, parts and various components were should at interim reviews, but never a complete func oning system. A er a brief introduc on of the thesis, a performance of the inter[ACT] 2.0 during the final review was facilitated with three actors, Will Sedig, Jon King, and Vincent Krause. These three actors played out several scenarios to demonstrate the social interac on that is happening both in physical and digital space at real- me. The scenarios were loosely scripted to demonstrate the three basic social configura ons that happen in physical space.

While two actors could properly demonstrate these three basic configura ons, a third actor is used to emphasize the dynamics of social spacing in the aggrega on.

The performance was also augmented with several screens to represent the transla ons that were occurring in digital space. As the performers change posi ons and moved into new sea ng aggrega ons, data visualiza ons would update in real- me. Several modes of data visualiza on were incorporated to display different layers of informa on overlaid on top of each other to convey the various informa on in the same context. The data that is collected from the chair and the subsequent data processing in all displayed to visualize the transla on of physical space to digital space. The actua on that happens in Twi er is visualized with a third

66

party applica on, TweetDeck. TweetDeck allows mul ple twi er feeds to aggregate within one window whereas feeds viewed on Twi er can only be seen from each individual account. This allowed the reviewers to correlate the data that was being visualized on screen with the twi er feeds that aggregated on the other screen.

Although the performance didn’t allow reviewers to directly interac on with the chairs, it allowed them act as par cipants. Most of the reviewers were given the same chair, Oh chair, to sit on for the dura on of the review. This allowed the reviewers to not only be viewers of the performance, but allowed them to indirectly be part of the performance, where as they could act on the chairs and orient them towards the content and each other. The reviewers were also placed in the Oh chairs to simulate the perspec ve, so that they could judge the visual response of the chair. This allowed the reviewers to be immersed in part of the system, even though they could not ac vely generate the data that was being processed.

Figure 26. An image compile from the performance with tweets that have aggregated through the performance.

67

Cri cal Response Following the brief performance, further research and background informa on was presented to aid the conversa on surrounding the thesis and the thesis project. The reviewers in a endance were: Mark Shepard, commi ee chair Nicholas Bruscia, commi ee member Dave Pape, commi ee member Teri Rueb Josephine Anstey Omar Khan Joyce Hwang Dorita Hannah Fionn Byrne Privacy A er the presenta on of the thesis project and thesis research suppor ng the project, the ini al response from several reviewers was a sense of fright. This sense of fright might have come out of a genera onal issue, where older reviewers that were engaged on the issues of video surveillance in the 1980s had more problems with the way that project was collec ng data and making assump ons for the user. The younger cri cs seemed to not mind the data collec on which might be due to the fact that we never witnessed the issue of video surveillance and we are part of a me where data post, mined, and gathered on the internet is fairly democra zed.

Data is collected in the systems, but the users have the ability to view it. While the

68

controllers of the systems have access to more data than a typical user, the system is not completely closed unlike the closed circuit surveillance cameras. The chairs actually handle the data collec on similarly to many of systems found online, where crea ng an account opts the users into the system and ac ng within the system provide more data about the user. With the chairs, you don’t have to place the cards in the right pocket, this allows you to opt out of the system, but you lose the possibility to interact with other users at this added layer interac on. In an ideal world, users would need to opt-in to both physical and digital spaces, and have the ability to con nually opt in or out of the system as they see fit.

Dorita Hannah men ons another privacy issue, where sensing a person from their bu ock might be considered an invasion of privacy, especially because of the cultural context. In Australia, where she is from, there is less censorship and more socially acceptable to be viewing chests and bo oms, whereas the US censors more parts of the body including bo oms. I don’t think this will be much of an issue in the future. There are already many wearable device, such as Nike Fuelband, Basis B1, Fitbit, and others, on the market that collect data for sleep analysis, daily movements, and heart rate. People wearing these devices need to wear them all the me to get large samples of data and these devices are expected worn at all mes of the day, even during the most private ac vi es of showering and sleeping to help the user understand how ac vi es aect their health.

What might be more important in systems that are collec ng data about its users is trust. The users must trust that the systems and the owners of those systems are not going to sell their data, and that the data is going to be securely stored so it is not accessible by everyone. It is scary to think that there could be so much data about

69

one singular person in place. If the data used to benefit the individual rather than to harm their freedoms or way of life, and the data is collected to a trusted place then the individual shouldn’t need to worry about compu ng and more powerful systems infringing on their rights. When you have mistrust and uncertain es about the system behaviors, this deters people away from systems such as Facebook and Twi er which collect massive amounts of data about their users and never deletes it.

Complexity One major cri que of the thesis and project during the review was from Teri Rueb, who said “cri cal angle is missing”. This was a hard ques on for me to answer during the review because I had become so consumed by the various components and research topics needed to realize the project that I had lost the inten on of the thesis along the way. Omar Kahn suggests a cri cal view that already exists within the thesis, but I was not necessarily aware of at the me of the review: “It has to the do with par culari es of physical sociality versus what is understood as virtual sociality. And so the way that we interact with Twi er or Facebook, the ques on of proximity there, is actually one of the frameworks, because you are not proximate. On some level those things are star ng intersect.

And the interest of Facebook and Twi er is to impose themselves in this space.

It raises an interes ng problem, where we are layering our other social network life on top of everyday life. They occupy two different spaces, space of physical sociality is proximity and the space of virtual sociality is remoteness a emp ng

70

to create proximity.” -Omar Kahn A er some reflec on, I’ve realized that the thesis is not so cri cal about the point of intersec on between physical and digital space but is cri cal about ways that the intersec ons form and occur. The research finds that physical space is overlaid on digital space when a means of representa on is needed for a new ac on, process, or feature that is created in digital space.

This is problema c because digital space borrows exis ng physical ac ons and uses them to misrepresent digital ac ons. The act of “friending” is borrowed by Facebook to describe the up-leveling of user access of content. If you are friends, you can see everything that your “friend” has shared, no ma er the level of your physical friendship. In the physical world, friendship has many levels: close friends, friends, childhood friends, and acquaintances.

These different category of friends typically know different parts of your life depending on the length of a friendship and connec on, and some mes that doesn’t get past acquaintances even though you’ve known each other for a while.

Facebook use the word “friendship” describe a digital rela onship of sharing content because you trust your friends. In some cases, the word “friendship” is taken too far, and too much data is given to “friends”, especially with apps that have the expecta on to track users all the me. Apps like Azumio’s Argus overshares informa on between friends, the app shares users’ height and weight in addi on to all their ac vity levels mapped out on a meline.

“Friends” can decipher when exactly you got up and started moving during the day because the pedometer automa cally tracked and graphed in rela on to the me, showing each step that was taken when the user was carrying the phone. The problem is the not the oversharing of

71

data but the associa on of the word “friend” or the act of “friending” to the authoriza on of data transac on.

Just as the oversharing of data makes the misuse of the word, “friending”, is even more evident, the extreme cases presented by the thesis project sheds light on the misuse of physical ac ons in digital space and the misuse of digital ac ons on physical space. This should be less problema c as more physical objects take on digital forms but the thesis shows that the direct transla on of physical ac ons to digital ac ons proves to be just as convoluted as the misrepresenta on of digital ac on. A physical object can have various meanings depending on how the object is used, i.e. a broom can be a tool for cleaning but it can also be a witch’s s ck for flying. The dis nc on can be told through how it is being held and its placement in rela on to the body.

Experiment 03: inter[ACT] 2.1 This second itera on of inter[ACT] 2.0 responses to some of the cri cism at the final review, and redesigns the hardware and so ware associated with chair. The context awareness algorithm and data structure is refined to allow for grouping of mul ple chairs rather than just pairing chairs together found in the previous prototype allowing the system to make more complex and unstructured social configura ons as on the metrics used in inter[ACT] 2.0. Not only is the so ware redesigned in this next itera on, the hardware associated with the chair is refined to be more robust so that actors are not needed in the next public installa on, and the general public could be invited to experience this new system of embodied interac on with

72

data.

The hardware and so ware refinement is made, so that the chairs and their interconnec on with digital space can be explored in more depth conceptually. This itera on creates a more passive system of interac on between users via the physical connec on between the user and the chair. The chair s ll enables the physical interac on between users as well as the digital interac on, but the user is not interface directly through the chair anymore.

The chair is now an ar fact that is engaging with the occupants of the chair digitally and physically through the act of si ng. The chair no fies the proximal users of social changes that have occurred through the act of si ng or standing.

The first itera on tested the transla on of physical ac on to digital ac on, to see if the physical ac on could directly relate to the digital ac on and conform to digital space. This tests the ability of digital space to conform to condi ons created by physical space and interac ons.

For this scenario to work, the tweets will have to conform to the speed and number of aggregated micro-interac on performed within a space.

Hardware Refinement In this version of inter[ACT] 2.0, the hardware is slightly modified to be mounted to bo om side of rather than be inserted in the seat cushion. There were too many components in the seat cushion causing to heat up over me. This new version also implements a wireless connec on to a server rather than using a wired connec on to provide power and a communica on line. The WiFly wifi module is used provide a UDP connec on to the server. The removal of this cable requires a large enough ba ery unit to support the Arduino

73

microcontroller, MIFARE reader, LED strip ligh ng, and the WiFly wifi module, which also influence the shi from the placement of the hardware components from the seat cushion to an under the chair mount. The selected ba ery was 5” x 4.5 x 7” making it impossible to fit the in seat cushion but is able to support the installa on for eight hour me span when fully charged.

Another hardware change in this itera on is the implementa on of the LED strip ligh ng.

In the previous itera on. The LEDS were hung on the by various small brackets, which was cri cize because it made the chair look like it was s ll in a prototype state rather than a finished product. In this version, a mold of the rail of the chairs were created so that the LEDS could be embedded in poured silicon to be glued to the chair.

Figure 27. The silicon pours and test molds This provides the user with protec on from exposed wire and provides a more robust system of actua on that is protect from liquids or jostling. The material of the chair didn’t provide a diffuse enough ligh ng surface for the strong LED ligh ng, and the visual effect created what seemed like a do ed line. The translucent silicon also provides a diffuse material for the light to

74

pass through crea ng be er light quality and a more consistent beam of light.

So ware Refinement The previous itera on of inter[ACT] 2.0, doesn’t reliably tracked the orienta on of the chair, because it reduced the granularity of the heading from 360 degrees to eight posi on.

The posi on of the chair didn’t always match the derived value. This version uses the raw heading data and projected line intercep on based on the object heading to calculate the social alignment of users.

Metrics were created based on raw heading data and focal point intercep on of each chair to derive the social alignment of users. People that are facing each other are measure at 180 degrees from each other, crea ng intersec ng focal points. Between 90 and 0 degrees, it can be assumed that the users are socially align to each other and to subject ma er, where user views are intersec ng. User that are si ng parallel and not focal views are not intersec ng then it can be assume that the user are not engaged with each other but the same subject ma er. If none of these condi ons are sa sfied then it is safe to assume that the users are not socially aligned at all. These derived metrics are feed into the context algorithm to group socially align users.

Brute force algorithms were used in inter[ACT] 2.0 which were ineďŹƒcient in iden fying more than two chairs that were socially aligned. inter[ACT] shi s from the brute force algorithms to more refined search algorithm using query-able data structures. Replacing the brute force comparison of distance between chairs, a spa al hash is used to query all chairs within a certain area. The spa al hash algorithm divides the viewable area of the camera into

75

gridded squares and each square represents a “bucket” or container. This algorithm locates data within the “buckets” depending on bucket size, and is usually used isolated small amount of data within a larger system like points in a three dimensional mesh. In the case of the chairs, the objects are much largerand the “buckets” needed to be scaled to accommodate the chairs while retaining the threshold of one chair space between chairs. Unlike the previous method of using brute force and distance formula, this algorithm automa cally groups mul ple chairs together by returning all the chairs within a “bucket”.

Figure 28. Data visualiza on of the spa al hash map without the overlay of the live video stream showing the distance rela onship between the objects in space.

The spa ally hashed chairs are then used to calculate the orienta on of the chairs against each other. This filters the proximity aligned chairs down to sa sfy the three condi ons of orienta on: facing chairs, chairs that are facing the same thing, and chairs that are facing

76

each other and the same thing. To result in a filter list of socially aligned chairs, a new algorithm is used in combina on of the algorithm found in inter[ACT] 2.0 with an algorithm that solves for angle of coincidence between two chairs. This octant algorithm is changed slightly in this version so that the true heading isn’t mapped from zero to eight within eight quadrants of the circle but from zero to seven, this.octLoc = (int) map(this.octA, -PI, PI, 0, 7). Using the following code, the metrics for resolving social alignment is employed:

Figure 29. Code snippet from inter[ACT] 2.1, showing the condi ons that need to be sa sfied to each metric to define social alignment.

Once the chairs are grouped by proximity, they are then filtered for social alignment. The code, first, checks to see if the chairs are facing the same direc on using the octants algorithm found in inter[ACT]2.0. The chairs are compared to check if the same octant loca on or parallel to see if it sa sfies the condi on of facing the same loca on. An equa on to

77

check for parallel headings is not used so there is a slight tolerance between perceived object heading and true object heading of +/- two degrees. If they are not facing the same direc on, it checks to see if the both chairs have the same focal point, and if the angle of coincidence is between 179 and 1 degrees. To do this, Toxiclib Geometry Library is used to project an infinite line aligned with heading of each chair to find the point of intersec on between the chairs.

The point of intersec on will be in the front of the chairs rather than behind, then both chairs have the same focal point. If both chairs have the same focal point, the angle of coincidence is found using arctan2 between the two chairs to validate the chairs is less than 180 degrees and greater than 1 degree within each other and within the same angle of coincidence from each other at the tolerance of +/- one degree. If these first two condi ons are not sa sfied then the code checks to see if the chairs are facing each other using the octants. If either chair is four octants away from the other, then the chairs are facing each other. This gives the chairs some tolerance rather than just using true heading to intersec on at 180 degrees. If any of these condi ons are sa sfied then the chairs are added as an edge in a directed graph using JgraphT.

78

Figure 30. The metrics of social rela onship used to develop a new context awareness algorithm.

Using JgraphT, a directed graph is used to store pairs of socially aligned chair within the group of proximal chairs. Each of the chairs are a ver ces within the graph and the edges are made of the pairs of socially aligned chairs. Once all the socially aligned chairs are found through the brute force itera on, the each socially aligned grouped is retrieved by chair. Once retrieved, each group is compared to find and delete all duplicate groups. As a result of this filtering the process, the remaining groups are defined as the socially aligned groups, and this filtering process actuates the LED strip ligh ng of each chair to acknowledge the resul ng grouping.

79

Using this context awareness algorithm also led the modifica on of the actua on of digital space via Twi er. Based on the response from the final review, this itera on changes the transla on between chairs and Twi er so that the act of si ng in the chair doesn’t automa cally “friend” or “follow” occupants that are found to be socially aligned, but the chair automa cally and mutually “friends” and “follow” the occupant of chair. The chair no fies the user when there are social changes that are occurring in space. As a user sits down, if that user is in proximity to another chair with an occupant, the chairs “tweet” at the occupant to no fy the user that someone has just enter their social boundary.

Figure 29. Visualizing the distance and orienta on rela onship of chair and the resul ng grouping.

80

This is a much more passive approach to connec ng individuals that are using the system, but takes into considera on how “small moves say things to other people.” This use case takes in considera on of the physical connec on between human and object in addi on to the physical connec on between users. The act of si ng creates a connec on between user and object, and at the same me, the ac on of si ng creates a connec on between the chair and user on Twi er. Here the chairs are conversing with their occupants, whereas the previous itera on used the chair to directly interface people to Twi er.

There are some limita ons found with the Twi er system due the change in method of transla on between the ac on of si ng in the chair and its correspondent ac on in digital space. Because of the system architecture found in Twi er, this new system wouldn’t work with addi onal chairs aggregated in the system. The “tweets” have a 125 character limit; with addi onal chairs, the character limit would be easily reached if more than four chairs are found to be socially aligned. Another limita on of Twi er is the amount of tweets that can tweeted within an hour. To limit the overloading of their server, Twi er API or applica on programming interface, puts a limita ons on the number of callbacks to their servers. This severely limits inter{ACT] 2.1 because the rate at which the chairs can post changes based on the microinterac ons that affect the social situa on based on their proximity and orienta on to another chair is hindered.

If one chair changes its orienta on, it might affect one or more chair, therefore aggrega ng tweets to each affected chair. The aggregated tweets build up rela vely fast because each chair tweets its own status but its status is dependent of the state of all the other

81

chairs. This exposes how micro-interac ons with physical objects can really affect space, but the limita ons of the Twi er API shows that digital space can’t support this micro-interac on that happens in physical space. As of now, servers don’t have the ability to support the number of responses needed in real- me due to processing power, bandwidth, and data storage. Not only is digital space of Twi er limited by infrastructural condi ons but the architecture also proves to be limi ng, in terms of a direct transla on between spaces. Even though the architecture of Twi er doesn’t support this type of interac on between user and object, there might be another digital architecture that can support this type of interac on, such as the Short Message Service (SMS).

82

Figure 30. Aggregated tweets from chair_one “twee ng� occupants of the chair.

83

Conclusion The installa on at the final review featured the work done throughout the semester, and it clearly demonstrated the physical and digital rela onships that were explored. The thesis installa on provided and considerate the similari es and differences between social rela onships that occur in physical, proximal space to the virtual, distant rela onships that occur in social media. The understanding of these rela onships facilitated the transgression of interac ons between digital and physical space, enabling users to use a ributes of physical objects to interact with digital space.

While some social media sites like “Grinder”and “Tinder” use proximal distance in physical space to connect digital users together to create new physical rela onships, physical objects inherently carry many meanings and those meanings are going to be carried into digital space. As digital space and physical space become increasingly intertwined, a clear understanding of both spaces will be needed to design future models of interac on involving both physical and digital space. The thesis is caught in this dilemma of agency within both digital and physical space. What does the user want from their digital connec ons and what does the user want from their analog connec ons? When would they intertwine?

The success of the installa on and thesis can be defined by the success of the design framework that is developed to model the interac on between digital spaces through the use of everyday objects. The framework is flexible enough to be used to test various models of interac on as seen in Version 1 and Version 2 of the thesis project. Although this thesis only explores the use of one physical ar fact and one digital en ty and this might seem as it doesn’t

84

provide conclusive results of a successful framework for other physical ar facts and digital en

es, the concepts developed within it aim for a complex inves ga on of these interac on

models, physical models and their rela onship with the built space.

85

Bibliography Art and Technology Hartmut Bohnacker, Benedikt Gross, and Julia Laub, eds. Genera ve Design Visualize, Program, and Create with Processing, 2012.

Manovich, Lev. “The Poe cs of Augmented Space.” Visual Communica on (2006) (206AD): 219–240.

Myron, Krueger. “25. Responsive Environments.” In The New Media Reader. The MIT Press, 2003.

Pearson, Ma . Genera ve Art. Pap/Psc. Manning Publica ons, 2011.

Shiffman, Daniel. “The Nature of Code,” November 26, 2012. h p://natureofcode.com/book/.

Negroponte, Nicholas. The Architecture Machine: Toward a More Human Environment. The MIT Press, 1973.

Human Computer Interac on Dourish, Paul. Where the Ac on Is: The Founda ons of Embodied Interac on. New Ed. The MIT Press, 2004.

Ishii, Hiroshi, David Lakatos, Leonardo Bonanni, and Jean-Baps ste Labrune. “Radical Atoms: Beyond Tangible Bits, Toward Transformable Materials.” Magazine Interac ons 19, no. 1 (February 2012): 38–51. doi:10.1145/2065327.2065337.

“MIT Media Lab Tangible Media,” n.d. h p://tangible.media.mit.edu/projects/.

86

Sterling, Bruce. Shaping Things. First Edi on. The MIT Press, 2005.

Weiser, Mark. “The Computer for the 21st Century.” Scien fic American no. September 1991 (n.d.).

Coded Space Craig Wisneski, Hiroshi Ishii, Andrew Dahley, Ma Gorbet, Sco Brave, Brygg Ullmer, and Paul Yarin. “Ambient Displays: Turning Architectural Space into an Interface between People and Digital Informa on.” Coopera ve Buildings: Integra ng Informa on, Organiza on, and Architecture (1998): 22–32.

Friedman, Yona. Yona Friedman: Pro Domo. Actar, 2006.

Frazer, John. An Evolu onary Architecture. Architectural Associa on Publica ons, 1995.

Kitchin, Rob. Code/space: So ware and Everyday Life. So ware Studies. Cambridge, Mass: MIT Press, 2011.

Maas, Winy, ed. The Ver cal Village: Individual, Informal, Intense. NAi Publishers, 2012.

Ma hews, Stanley. “The Fun Palace as Virtual Architecture Cedric Price and the Prac ces of Indeterminacy.” Journal of Architectural Educa on (2006): 39–48.

Varnelis, Kazys. “Programming a er Program. Archizoom’s No Stop City.” Praxis no. 8 (June 2006).

Whyte, Willam H. “The Social Life of Small Urban Space”. New York, NY. Project for Public Spaces, 2001.

87