Campbell Argenzio by Obuchi Lab, University of Tokyo

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

気分と連動した仮想ライブラリのプロトタイプ

Tokyo Advanced Design Studies Department of Architecture The University of Tokyo

2020 Advanced Design Studies Master's Thesis

Campbell Argenzio

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

感情をデータとして入力する設計手法の研究：

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Supervising Professor:

Campbell Argenzio Master's Thesis

ADS

Yusuke Obuchi Thesis Sub-Advisor: Prof. Hidenori Watanave Prof. Koichi Kato Additional Course Support: Shuta Takagi Yusuke Komata

the Department of Architecture at the University of Tokyo in partial fulfilment of the requirements for the degree of 3

Master of Engineering in Architecture

2020 Advanced Design Studies Master's Thesis Research Title: Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” 感情をデータとして入力する設計手法の研究：気分と連動した仮想ライブラリのプロトタイプ Editors: Campbell Argenzio Book Design: Campbell Argenzio Book Edition Design: Jan Vranovsky Printed in Tokyo, Japan Advanced Design Studies Typeface: F Grotesk by Radim Peško © 2020 Obuchi Lab, T

ADS

Department of Architecture Graduate School of Engineering The University of Tokyo Eng. Bldg. 1, Room 415 7

Bunkyō

1 Hongō ku

Tokyo 〒 113 Japan www.t-ads.org

8656

List of Figures Acknowledgements Abstract

1 Introduction 12

1.1 Research Question

1.2 Structure of the Thesis

2 Background

16 18

2.1 Complexity of Emotions

2.2 Dealing with Complexity Digital Design in Architecture 24 2.3 Cues from Other Fields

2.4 Human-Centric Digital Design

ADS

3 State of the Art

28 T

3.1 Environment and Emotions

3.2 Emotions in Architecture

3.3 Emotions in Software Engineering

3.4 Emotions Response in Psychology

3.5 Case Studies

4 Methodology

4.1 Design Approach

4.2 Virtual Reality as Architectural Practice 72

4.3 Text as a Spatial Medium: Immersive Typography 78

4.4 Prototype Description

5 Design Documentation 6 Conclusion

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Table of Contents

108

7 Appendix 110 8 References

116

List of Figures Fig. 1 Sketch of an Emotionally Responsive Environment with Slider Interface Fig. 2 Early Sketch for VR GUI

Fig. 3 Tsinghua Facial Expression database (Yang, 2020)

Fig. 4 Orientation Study: Rotation Tower, Ladybug Tools (Armijo, 2013)

Fig 5. A topological ordering of a directed acyclic graph: every edge goes from earlier in the ordering (upper left) to later in the ordering (lower right). (Eppstein, 2016) 26 Fig. 6 Archetypal Parametric System

Fig. 7 Workshop on Computer Vision for Affective Computing Banner (Dhall, 2019)

Fig. 8 Students drawing an arc with their bodies for the TOCA project (Lopez, 2016)

Fig. 9 Arc Typology assembled into form (Lopez, 2016) 31 Fig 10. Mazuch's Biophilic Design (Mazuch, 2017)

Fig. 11 Mazuch's Biophilic Design (Mazuch, 2017) 35 Fig. 12 A Windowless Radiology room equipped with screens on the ceiling and walls (IBI Group via Mazuch, 2007) 36 Fig. 13 . Instances of balance images used in the study Can Hospital Form

Trigger Fear Response? (Pati et al., 2016) 37 Fig. 14 Reading Emotions of Color Environments (Yoon & Wise, 2014)

Fig. 15 Peter Zumthor's Thermal Vals ((Zumthor, accessed ArchDaily 2009)

Fig. 16 Buildings and Emotional Affects

Fig. 17 Teshima Art Museum (ph. Benesse Art Site)

Fig. 18 Teshima Art Museum (ph. Iwan Baan, 2020)

Fig. 19 (Day, 2004)

Fig. 20 (Day, 2004)

Fig. 21 (Day, 2004)

Fig. 22 (Day, 2004)

Fig. 23 (Day, 2004)

Fig. 24 (Day, 2004)

Fig. 25 Different kinds of HBI spaces (e.g., indoor, public, personal) are imagined with respect to the three concentric circles of people, built environment, and computing, and dissecting physical, social, and spatial dimensions. (Alavi et al. 2019) Fig. 26 Al Bahar Towers, UAE (Richters, 2014) Fig. 27 Buildings and Emotional Affects

Fig. 28 Installation of Hylozoic Ground (Beesley, 2014) 50 Fig. 29 Interactivity and Motion (Beesley, 2014) 50 Fig. 30 D-Tower Website (Spuybroek, 2003)

Fig. 31 D-Tower Instalation View (Spuybroek, 2003) Fig. 32 MoBI System (Kalentari, 2019)

Fig. 33 MoBI System (Kalentari, 2019)

Fig 34. Schematic representation of a machine learning process for emotion recognition (Bota et al. 2019) Fig. 35 Facial Expression Analysis (Dantone, 2012) Fig. 36 Emotiv Products (Emotiv, 2020)

Fig. 37 Empatica Embrace Wearable (Empatica, 2020) 57 Fig 38.Theoretical Constructs of Emotoin (Fortkamp, 2005) 59

Fig. 39 Circumplex Model of Emotional Dimensions (Russell & Barrett 1999)

Fig. 40 Different Circular Models of Emotions (Russell & Barrett 1999)

Fig. 42 Digital, Slider Based Self Report User Interface (Marcus, 2017)

Fig. 43 VR Space Website Mock-Up, early study 67 Fig. 44 Overall System design

Fig. 45 Headlines Regarding Grieving During the Covid 19 Pandemic 68 Fig. 46 Research framework based on the S-O-R model (Yoon & Wise 2016)

Fig. 47 Exposing individuals to a stimulus vs designing for individuals with a heigtened emotional state Fig. 48 Park Borrowed from the EMMA project: Peaceful Park (Luis Alcaniz Raya et al. 2014) 73 Fig. 49 Park Borrowed from the EMMA project: Anxious Park (Luis Alcaniz Raya et al. 2014) 74 Fig. 50 Park Borrowed from the EMMA project: Nuetral Park (Luis Alcaniz Raya et al. 2014) 75 Fig. 51 algorithmically composed Liquid Architecture (Novak, 1991) 76 Fig. 52 Spatial Computing (Magic Leap, 2018)

Fig. 53 ad for kitchens that hides geometry in the newspaper layout (Felipe Salazar 2020) 80 Fig. 54 Giulio Camillo’s Theatre of Memory (1519-1544) 80

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Fig.41 The PANAS Scale (Watson, Clark, and Tellegen, 1988) 62

Fig. 55 Source Uknown 81 Fig. 56 An example of a Word Search (Freeprintable.com, 2020) 81 T

Fig. 58 Immersive Typography (Adam Varga, 2020)

Fig. 59 Graphic User Interface for VR Space

Fig. 60 Mapping VR Environment to Occupant's Emotions Fig. 61 VR Prototype Workflow

ADS

Fig. 57 OMSE Augmented Reality Typography 2019

Fig. 62 STEP ONE: Enter VR Space 88 Fig. 63 STEP TWO: Self Report Emotional Status

Fig.64 STEP THREE: Emotional Status Determined

Fig. 65 STEP FOUR: Environment Generated

Fig. 66 STEP FIVE: Texts populate enviornment to match mood , mood can always be changed Fig. 67 GUI Inside of VR Space

Fig. 68 GUI Inside of VR Space Interaction 93 Fig. 69 GUI Inside of VR Space Interaction 93 Fig. 70 Positive Affect Environment Mood Hopeful

Fig. 71 Positive Affect Environment Mood Hopeful

Fig. 72 Positive Affect Environment Mood Hopeful, Text placements axon Fig. 73 Positive Affect Environment Mood Hopeful

Fig. 74 Negative Affect Environment Mood Sadness

100

Fig. 75 Negative Affect Environment Mood Sadness, Text placements axon Fig. 76 Negative Affect Environment Mood Sadness

103

Fig. 77 Negative Affect Environment Mood Sadness

105

102

Fig. 78 What is Empathy? (Oliveira, 2018) 109

Thank you to Yusuke and Komata-San for their guidance in this thesis. Thank you to Professor Watanave and Professor Kato for their review and time. Thank you to Shuta Takagi for his patience and knowledge in helping me figure out C#. Thank you to my friend Ali for advising me on how to structure projects in Unity. Thank you to the Ministry of Education, Culture, Sports, Science and Technology (MEXT) for supporting me as a graduate student.

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Acknowledgements

Thank you to my mother, Ellen.

Thank you to my brother, Charles.

ADS 9

Page Description

Traditionally the built environment affects occupants’

emotions in a one directional way. Occupants are subjected to the stimulus designed by the architect, though each individual’s reaction will be a unique one. Emotions are highly subjective and difficult to quantify and analyze. This challenging work is undertaken primarily by cognitive and social scientists and psychologists. Recent technological development has enabled a broader interest in sensing, quantifying and analyzing emotions, especially within the field of Software Engineering. As digital design becomes second nature in Architecture and we are in the nascent stages of the virtual truly integrating with our everyday lives, it is important to assess architects’ role in this development. We must look to how Psychologists and

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Abstract

software Engineers utilize digital methods to deal with

by the emotional state of the end user. Virtual Reality is

ADS

Virtual Environment is built that is parametrically driven

emotions as data. As part of the research a prototypical

the perfect prototypical playground for architects wishing to engage with the more subjective aspects of creating environments as it is not encumbered by the pragmatic necessities of a real structure. The target occupants are grieving adults. The program (function) of the space is that of a “library” where the occupant enters the space in search of words and text relating to the topic of grief. The content found will be uniquely tailored to the occupant’s current emotional state, indicated via self report. Though VR is usually associated with engaging the visual, auditory, and more recently physical, this paper argues that text is the most efficiently empathetic media. The variety and intensity of emotions felt by grieving individuals are a shared universal experience. Any project that endeavors to respond to this diverse topic must reflect this shared aspect. The design does so by utilizing texts gathered from online forums, staple literary resources and user-submitted entries, creating an empathetic architectural ecosystem around the topic of grief.

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Introduction

1. Introduction 14

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Introduction

1.1

Research Question The main topic of investigation is how architecture

can be more interactive with an individual occupant’s emotions. In order to interact with emotions, we must be able to sense or read them and respond to this information. Contemporary digital design in Architecture utilizes information or data, commonly in the form of parameters that affect a geometric outcome. These tools are ready to be directed at emotional reactivity within the built environment by architects. Though the technology for immediate emotional reactivity is not yet there yet, as this paper examines, it is coming. Architects must find their place in this dialogue between buildings and individual occupant’s emotions mediated by technology. By looking to two other fields that already interact with emotions as data, this paper explores how architects and designers might re-purpose many of the well documented methods used in Psychology and Software Engineering to deal with measures of mood and emotion. The scope of this paper is not to give a thorough state of the art of these two relevant but entirely different fields. Instead, it examines many of the ongoing techniques deployed by Software Engineers and Psychologists and speculates how architects who are already working with data analysis and computation in their design processes might deal with the data set of occupants’ emotions. This research objective stems from two core beliefs. The first is that digital design in architecture should center human factors like emotions as much as geometrical complexity. The second is that as digital media becomes spatial and immersive it becomes just as relevant to the field of architecture as our real-world built environment. This research asks: how can architects deal with emotional data as an input in digital design? How can emotions be quantified, analyzed and applied in immersive, virtual spaces? How can architects interact with emotions as information? As any method being examined is best understood from both the conceptual level as well as practical application, it is a necessary part of the research to build a prototype of this idea of “Emotion as Input” in a designed environment, explored in this thesis via a Virtual Reality space. This prototype is not a proof of concept so much as an investigative illustration. If site 16

data, material parameters and generative form-finding

wishes to bring emotions to the table as well.

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

are guests at the dinner party of digital design, this author

T ADS

Fig. 1 Sketch of an Emotionally Responsive Environment with Slider Interface

Introduction

1.2

Structure of the Thesis

This thesis begins investigation by referring to the existing evidence of environmental impact on emotions, not as an extensive literature review, but as a frame of reference for the work to follow. Next the ideas of interactive environments in the context of current technological development will be examined, followed by discussion on the emotional dimensions of architecture. An overview of how emotion is being utilized in Software Engineering follows as a contemporary example of another field that has used technology to deal with emotions as input. The well-established methods for examining emotions and quantifying them in Psychology and Cognitive Science are then brought into the exploration, not as a comprehensive review but a reference as to what methods were informative for the methodology of this thesis. Relevant projects that showcase interactive environments with emotional implications or impact are referenced. Following this state of the art is a close look at the methodology used to build the aforementioned prototype. The design approach rationale is first illuminated as an overall systems design pieced together from the previous research topics, then followed by specific descriptions of the as-built prototype. The design documentation showcases two different hypothetical user outcomes and the interactive, spatial environment they encounter based on their differing emotions. An appendix of the data-base built and used in the prototype is organized in the appendix. The thesis concludes with discussion and implications.

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Fig. 2 Early Sketch for VR GUI

ADS

Background

2. Background 20

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Background

2.1

Complexity of Emotions Emotions are extremely subjective. When discussing

how emotions might interplay with architecture, we must start simply by defining them. According to the foremost textbook on the matter Understanding Emotions, emotions are defined as a “psychological state or process that mediates between our concerns (or goals) and events of our world” (Keltner, Oatley, Jenkins, 2013, p. 4). Linguistic researchers can’t determine “whether the disparities come from the difference of the quality of the emotional experience or the word choice when translating one emotion word to another language” (Karasawa, 1995, p.53). If it is this difficult to translate emotions from one language to another, how can we expect to translate emotions from experience into architecture? As architecture has often done, it is necessary to look to other fields for established methods as well as new technologies that may be borrowed and re-contextualized in architectural practice. This paper does not attempt to explain or defend the phenomena that architecture and environment have an effect on emotions, that has been done quite thoroughly elsewhere, though often within the context of healthcare design (Lawson, 2010; Mazuch & Stephen, 2005; Whitemyer, 2010). This investigation starts at the premise of emotional interactivity in the built environment and explores further how it might be recognized and utilized in digital design now that emotional analysis tools are a rapidly growing field.

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Fig. 3 Tsinghua Facial

Expression database (Yang,

2020)

Background

2.2

Dealing with Complexity: Digital Design in

Architecture In the very beginning of Scripting Cultures, Mark Burry declares in 2011 what is quite obvious in 2020 that “digital design is now fully assimilated into design practice.” (Burry, 2010). In his formative handbook Elements of Parametric design, Robert Woodbury offers the enticing point that “to the human enterprise of design, parametric systems bring fresh and needed new capabilities in adapting to context and contingency and exploring the possibilities inherent in an idea” (Woodbury, 2010, p.7). Digital design in architecture encompasses the use of computation, scripting, programming and parametric methods to deal with complexity. This complexity manifests as information and can be aesthetic, pragmatic, conceptual, based on data interactions or tied to programmatic requirements. This information can be regarding site, weather conditions or structural elements. It can exist in purely digital form or in material parameters or fabrication processes. Though computational design, scripting and parametric design are not interchangeable, all fall into the larger umbrella term of digital design and the implications, possibilities and opportunities presented by each are extremely related. In his entertaining and thoroughly examined PhD thesis, Daniel Davis draws parallels between parametric design in architecture and the field of software engineering. Davis posits that “architects assign a range of meanings to the phrase parametric model. A number of definitions have been advanced that range from style-based classifications, to software zealotism, to arguing that all design is inherently parametric” (Davis, 2013). This thesis does not seek to further the muddling of what defines digital methods in architecture by applying it to the very intricate world of emotion recognition and interaction. In the simplest terms this thesis seeks to utilize emotion as another input in architectural digital design practice, via virtual environments. According to Davis the “archetypal embodiment of the mathematical definition of parametric” consists of three main components: input parameters, explicit function and outputs (Davis, 2013). Davis reinforces this by referencing Ipek Dino who has said that

“parametric systems are principally based on algorithmic

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Fig. 4 Orientation Study:

Rotation Tower, Ladybug Tools

(Armijo, 2013)

Background

principles” because “an algorithm takes one value or a set of values as input, executes a series of computational steps that transform the input, and finally produces one value or a set of values as output” (Ipek as cited by Davis, 2013). It is important to point out that parametric models are systems built for change or variability, at least in terms of the input values and ability of parameters to respond to multiple conditions. Variations and adjustments can be made within the model. The parametric model creates what Volker Mueller calls a “Solution Space” where the designer explores several variations instead of a single solution (Mueller cited by Woodbury, 2010). This variability and responsiveness to unique conditions is what makes digital design methods so appealing to architects. Utilizing these tools requires that architects define problems in terms of inputs, functions and outputs, create and curate data streams and design responsive functionality. All of the ingredients are there for responding to the very complex data stream of human emotions, but they must first be gathered and interpreted.

Fig 5. A topological ordering of a directed acyclic graph: every edge goes from earlier in the ordering (upper left) to later in the ordering (lower right). (Eppstein, 2016) 26

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

ARCHETYPAL PARAMETRIC SYSTEM

OUTPUT

ADS

FUNCTION

INPUT

CHANGE

Fig. 6 Archetypal Parametric System

Background

2.3

Cues from Other Fields

It seems common sense that architecture must rely on interdisciplinary expertise when designing at the scale of the built environment. Just as engineering is often repurposed by architects from the purely pragmatic to serve not only structural goals but aesthetic onesarchitects should look to psychology and cognitive science when seeking to understand and implement emotions. In order to do so emotions must be tracked, quantified and then analyzed before they are ultimately utilized. In the context of digital design practice this system is most optimal when running in an adaptive feedback loop instead of a finite direction, which is currently the more common approach in Architecture (Castillo, 2019). Commonly, digital design processes in architecture are created in an Acyclic Graph that does not run continuously and has a finite end, see Fig. 5. We can see this programming logic in David Rutten’s Grasshopper3D, though plugins currently exist to increase its capacity. We might instead also look toward Human Computer Interaction and how treating buildings as interfaces would allow them to be flexible enough to become emotionally responsive. Architecture exists in the exciting yet challenging space between art and technical limitations. Bryan Lawson beautifully articulates this quality when he summarizes that architecture is “actually a design ﬁeld in which artistic processes and ways of knowing are combined with more technical engineering and scientiﬁc knowledge” (Lawson, 2010, p.97). As we seek to treat emotions as input we must combine our “ways of knowing” with the scientific knowledge of other fields that already deal deeply with this rich subject matter. The most obvious fields to borrow from are those dealing directly with emotions, Psychology and Cognitive Science. The second field to examine is another that utilizes digital methods to deal with emotions as inputs in order to interact with them: Software Engineering.

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Fig. 7 Workshop on Computer

Vision for Affective Computing

Banner (Dhall, 2019)

Background

Fig. 8 Students drawing an arc with their bodies for the TOCA project (Lopez, 2016)

2 2

2.4

Human-Centric Digital Design in Architecture 3

The emphasis on centering human emotions emerges from a line of thinking about human capacity in the context 4

of digital design established by Yusuke Obuchi in his lab at The University of Tokyo. Though an architecture department, the research blends digital fabrication, urban study and human-computer interaction to promote 3

fabrication processes that make advantage out of the difference and variety of individual human factors. The work seeks to re-establish the relationship of the human hand in creating the built environment in this rapidly evolving digital age. The research involves creating new 4

processes of fabricating buildings, but with an emphasis on the democratization of these processes through making them accessible. In the TOCA (Tool Operated Choreographed Architecture) project the arced contour motion of each individual students’ arm as they drew POSTHUMAN ENGAGEMENTS

a shape in the air was logged in a data-base called the “Catalog of Movements” shown in Figure 9. These contours could then be optimized into a total structure (López et al., 2016.). The differences of each student’s body proportions created a range of different shapes that could then be compared against each other and calculated into an ideal form. This philosophy of building fabrication systems that marry individual factors of individual participants with form finding is the ecosystem from which this thesis emerged. The major departures however are that the prototype built for this thesis is a Virtual Reality Environment and not a fabrication system, and the human aspect being emphasized is emotional experience. In discussing scripting in Architecture, Mark Burry points out that “the tool user (designer) becomes the new toolmaker” (Burry, 2011, p. 9). Using human-centric systems into architectural design as we create new tools allows us to celebrate and utilise what are commonly thought of as limiting conditions in digital processes. This engenders digital fabrication with a sense of empathy and

digital design, promoting greater well being in designed spaces, which is often the goal of the architect anyways.

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

inclusiveness and points towards a way of humanizing

T ADS

Fig. 9 Arc Typology assembled into form (Lopez, 2016)

POSTHUMAN ENGAGEMENTS

State of the Art

3. State of the Art 32

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

State of the Art

3.1

Environment and Emotions

Humanistic Architecture (Mazuch, Rona 2005), Therapeutic Design (Connellan et al, 2013), Evidence Based Design (Lawson 2010) and Sense Sensitive Design (Mazuch & Stephen, 2005) all refer to an approach to architecture that “draws on international research in the fields of psychology and sociology, biology and physiology into the effects of the environment on health” (Mazuch & Stephen, 2005). The logic follows that “environmental characteristics can have powerful healing and therapeutic benefits for their users” (Ulrich, 1991a; Scher1996, Lewy et al, 1980; Murgia &San Martin, 2002 as cited by Mazuch, 2005.) Such characteristics include color, natural lighting, views to the outdoors, physical sensations like smells, furniture arrangement and sense of scale and way-finding (Mazuch, 2005). These approaches are concerned with how an individual person takes in their environment through their sensory receptors and how this is relayed as data to the brain, which affects both physiology and psychology of that individual (Marzuch, 2005). One particularly well known study documents that “students who are fortunate to be in a school classroom with large windows and skylights perform better on tests than those Fig 10. Mazuch's Biophilic Design (Mazuch, 2017)

in rooms with very little natural light — between 7 percent and 18 percent better” (Whitemyer, 2010). Evidence Based Design exists most prominently in healthcare design because of “the existing evidence-based medicine culture” (Watkins, 2008, as cited in Whitemyer, 2010 p. 10). This wealth of research reinforces that “evidence can be a powerful tool used to inform designers’ and architects’ decisions” (Whitemyer, 2010, p.10). In Lessons From Neuroscience: Form Follows Function, Emotions Follow Form, Upali Nanda takes a neuroscience approach by examining the impact of visual stimuli in environments on emotions (Nanda, 2013). Referencing Gibson (1979) she describes that the interactions between an organism and their environment are what’s called its ‘affordance.’ She continues that as designers and architects who “seek to sculpt not just spaces, but human experience and behaviour, we are designing more than environments, we are designing affordances” (Nanda, 2013, p.2). This language of emotional affordance is a

clear way to encompass the idea of emotional interactivity

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

in environments. By defining the architect's role as someone who designs affordances that are then linked with brain response testing, the way in which architects deal with something as subjective as emotions can be formalized (Nanda, 2017). Natural Imagery is particularly well documented in reducing stress, pain perception and anxiety. Experimental studies have been conducted using various mediums including wall mounted images, videos, backlit displays and VR (Hathorn and Nanda 2008). In one study physiological data was collected via skin conductance and other indicators on participants. Those that viewed realistic simulations of nature settings had a faster recovery time than those who viewed urban ones (Nanda, 2017). Nanda, who has closely examined affordance in the built environment from a scientific and analysis perspective also reminds us that there are many cases where designers “implicitly employ formal tools

to elicit immediate emotional responses, across setting

ADS

types” (Nanda, 2017, p.13). She points especially to the aforementioned Healthcare sector and also Justice Architecture as primary examples of architecture that has targeted emotional response. Especially in healthcare design, emphasis has been placed not only on the interior functional spaces but especially on the main public lobby and how it is perceived from the outside (Nanda, 2017). In another study, Nanda and her research partners compared Amygdala activation in the brain based on exposure to Fig. 11 Mazuch's Biophilic Design (Mazuch, 2017)

State of the Art

Fig. 12 A Windowless Radiology room equipped with screens on the ceiling and walls (IBI Group via Mazuch, 2007)

curved edge contours vs. sharp corner contours using MRI technology. Part of the limbic system, the Amygdala “mediates the processing of negative emotions such as fear” (Bota et al., 2019). This examination of contour conditions in interior spaces was tested against 36 individuals across three age groups and of both sexes. Participants were shown a randomized set of 312 photographs of interiors, objects, landscapes and a set of controls. It was found that images with curves had a larger proportion of being liked than sharper images, and the percentage disliked for sharper images was significantly higher than liked images. The researchers emphasized that the potential link between the shape of contours and its influence on our fear response has “greater significance in the context of healthcare design, since patients and family members (as well as staff) encounter a high-stress environment (Pati et al., 2016). Fear, stress, anxiety and pain, all emotions present in acute care settings have a substantial effect on outcomes in healthcare settings. Considering this critical condition, Nanda and her research partners put forth that “it is imperative to create a better understanding of the association between ‘‘formal’’ design decisions and human emotional response” (p. 163). While this is especially true for health-care settings, Nanda’s research findings are relevant to all designed spaces. As contours are an environmental attribute of form 36

which designers manipulate, and it has been proven that contours in environments affect the brain’s amygdala’s

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T ADS

Fig. 13 . Instances of balance images used in the study Can Hospital Form Trigger Fear Response? (Pati et al., 2016)

State of the Art Fig. 14 Reading Emotions of Color Environments (Yoon & Wise, 2014)

response to visual stimuli, “environmental design decisions related to the built form must bear a systematic association with human emotional response to the built environment” (p.163). What other spatial aspects are responsible for having an effect on emotional well being? In Kathleen Connellan’s comprehensive overview of the research literature, Stressed Spaces: Mental Health and Architecture she references Nightingale Associates who have done substantial work in the healthcare field that has led to an understanding of materialistic and sensory aspects related to mental health. She Cites Mazuch and Stephen as combining psychotherapeutic methods with architectural practice to create healing. These architects created an ‘Emotional Mapping’ system which incorporates mapping emotional qualities of sensory information to spaces according to their intended use. They underscore the role of colors, citing that blues have been indicated to subdue aggressive individuals and that “visual monotony” can add to emotional and physiological stress. In the foundational study Relationship Between Color and Emotion: A Study of College Students, Kaya and Epps found that the color red is often considered exciting, orange as stressful, purple is dignified and that blue is associated with comfort. The researchers recognized that while some colors have been preferred regardless of age, ethnicity or culture, there

is also some evidence that color preference is culturally

because of its associations with nature. Gray was found to be associated with negative emotions like depression, sadness, loneliness and fear because of its similarities to the colors of bad weather. Kaya and Epps point out that Saito found that vivid blue was the preferred color for Asian study groups who found the color “refreshing, beautiful, and bright” and had none of the associations with sadness that Western study groups perceived. The researchers acknowledge that it cannot be confirmed whether associations with color are innate or cultural (Kaya & Epps, 2004, p. 400). Yoon and Wise developed a research framework to “explore the mind and body interaction to explain and further study the psychological and emotional effects of a color environment on people in the environment” by simulating several different colored

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

based. Green was found to be one of the most liked colors

rooms using high quality graphics renderings and then

measures of self report and also physiological sensors.

ADS

220). The students indicated their emotional state via two

exposing them to college students (Yoon & Wise, 2014, p.

The researchers’ findings supported the previously developed Color Image Scale which relates colors to image words using semantics by Kobayashi of Nippon Color & Design Research Institute. Though there is a wealth of existing literature on the effects of color on emotion and psychology, “little has been proven through empirical testing of what different color environments mean to people,” spurring this study. Yoon and Wise reiterate that there are amplified challenges when addressing colors in real-world environments (Yoon and Wise, 2014, p.220). Materiality and touch also affect patient well being in health care settings. According to Mazuch and Stephen surfaces can play a role in the recovery of patients as they aid patients to “re-engage with the materiality of the world.” An interesting point to note is that “perceptual confusion should be avoided,” such as a faux wood grain texture on a metal door as the metal would be heavier and colder than the given perception of wood. Regarding sound, excessive noise has negative effects such as increased blood pressure, cholesterol and heart rate, and it should be considered that harder materials absorb less noise when specifying materials (Mazuch as cited by Connellan, 2013, p. 139). 39

State of the Art

Fig. 15 Peter Zumthor's Thermal Vals ((Zumthor, accessed ArchDaily 2009)

3.2

Emotions in Architecture

Corbusier said in 1923: “ARCHITECTURE is a thing of art, a phenomenon of the emotions, lying outside questions of construction and beyond them. The purpose of construction is TO MAKE THINGS HOLD TOGETHER; of architecture TO MOVE US. Architectural emotion exists when the work rings within us in tune with a universe whose laws we obey, recognize and respect.” It is perhaps an obvious quality that architecture affects emotions via the somatic, embodied experience as well as the cognitive and perceptual. Some architects have been particularly successful at inciting emotions with their work. Peter Zumthor and Steven Holl create a sense of intimacy (Fortkamp 2005). Tadao Ando imbues his sites with a Spirit of Place (Hsu et al., 2015). In 2014 The Royal Academy of London held an exhibition titled Sensing Spaces: Architecture Reimagined, “about how architecture confronts us and communicates with us on an emotional and psychological as well as visual and intellectual level” (Goodwin cited by Glancey, 2014). All of this goes to say the emotionality of architecture is one of the reasons the field is so interesting. Though it is 40

important to acknowledge that currently, emotion exists

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

in architecture as a static condition expressed most often through physical qualities like materiality, light and scale. Typically, when a person enters a building their emotions are affected by that building, though the range of intensity of the effect varies widely. This concept is illustrated as a workflow in Figure 16. In Figures 19 through 24 Christopher Day explores in his influential textbook on the matter, how qualities of built space impact people. When one walks into the Teshima Museum by Artist Rei Naito and Architect Ryue Nishizawa they are enveloped by a shroud of concrete, their footsteps echoing across the curved walls. The island scenery emerges in ellipse-shaped vistas that morph in perspective as you traverse the concrete expanse. Water droplets mysteriously perspire from the ground at intervals far apart enough that you can carefully avoid them. When

mostly empty, the space feels vast and quiet, somehow

ADS

amplifying the nature surrounding the building that melds into the hillside. When the museum is more full of fellow occupants, one becomes lost in the meditative buzz of other’s passive auditory expressions like low whispers and clothes shuffling. Typical Relationship btwn Architecture and Emotions

Person

Architecture Fig. 16 Buildings and Emotional

Building Induced Emotion

Building

as an Emotional Organic User Interface "OUI"

Affects

Person

Building Responds to Emotional Status

Emotional Status

CHANGE 41

State of the Art

From a lecture to the staff of Teshima Art Museum, re-printed in the Museum Handbook: Matrix It is always beside us, and everything is born from and nourished by it. Something that makes life on earth possible and also that observes life on earth - as a vessel. Through space, as it feels

Like a vessel that accepts nature as it is

Based on the idea that anything before us is good

Man is reborn moment after moment, newborn each time

Natural vitality (tama/anima) is there and here

Somebody other than me is there, living on earth now as I do

Is it a blessing in its own sake to exist on earth?

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Fig. 17 Teshima Art Museum (ph. Benesse Art Site)

(ph. Iwan Baan, 2020)

ADS

Fig. 18 Teshima Art Museum

State of the Art

Fig. 19 (Day, 2004) A very uncomfortable plan shape to be in can be made pleasant if the acute angles are blocked off to become obtuse.

Fig. 20 (Day, 2004) Even a quietly enclosing curve still has a visual force – reflecting its structural force. The meeting of the moving against the rigid can appear unstable unless resolved by moderating neighbouring elements, angles and meetings.

Fig. 21 (Day, 2004) Non-rectangular forms are less harsh and oppressive than rectangular ones – just imagine the experience of walking around a right-angle-cornered building or one at 105°.

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Fig. 22 (Day, 2004) The hierarchy-stressing shape of the trapezium.

Fig. 23 (Day, 2004) This is not a secure space. The ceiling weighs heavily – it sags. The

walls droop.

ADS

Not so bad in canvas perhaps – but terrible in concrete. This, on the other hand, we experience as stable, enclosing, secure.

Fig. 24 (Day, 2004) There is a lot of difference between a straight line which suddenly swings into a curve and one in which the curve is implied in the straight and the firmness of the straight in the curve.

State of the Art

This poem certainly hints at the impactful experience of entering the Teshima Art Museum, qualities much like a balm to an average individuals’ existential dread. There is no doubt that spaces such as these interact with our emotions on a higher level than a typical office building or convenience store. The light, the temperature, the foliage in view, the sound of footsteps, the perception of other occupants, all of these things certainly change according to the seasons and the number of visitors present. Still, the museum stands solidly in its material expression, it does not engage directly with the occupants' inner mental turmoil. Instead, it envelopes the occupant, suggesting to them some of the wonderful concepts portrayed in Naito’s poem. All visitors are shown roughly the same stimulus. It is not that the Teshima Art Museum is lacking in any way because it does not supply different occupants with different stimuli. Instead we can think of this museum as a fabulous example and a pinnacle for designed spaces that are meant to have an emotional effect. However we must acknowledge that even given the meticulously designed qualities of a space such as this, the dialogue between building and occupant continues to be one directional, as diagrammed in Fig 16. In recent years there have been examples of materials and physical systems that are non-static and reactive, such as the kinetic sun-shading of the Al Bahar Towers by Aedas Architects. These have generally always been geared towards environmental conditions and not human emotions. It can be said that the impact of such a dynamic facade system will create an emotional response, which is not untrue. However the facade does not respond or interact with emotions directly, only with the environmental conditions of the site. In order for environments to engage more directly with their individual occupants, we can utilize a growing, new sector of the architecture field that melds data, spatial adaptation and personalized automation, something that has been called by Alavi et al. “Human Building Interaction,” shown in Fig 25 (Alavi et al., 2019). Sara Nabil similarly defines an “Organic User Interface” or “OUI” as architecture that is “not just ‘literally’ organic in form or materialism, but a user interface that is ambient, and dynamic in a way that allows buildings and architectural structures (from bridges to sculptures) to react, express or heal us through changing

their appearance or shape similar to the way humans and

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T ADS

State of the Art

animals are able to communicate by simple speechless gestures or subtle reactions” (Nabil et al., 2017, p.93). OUIs build off previous ideas on Smart Environments where non-intrusive sensing networks have leveraged the networking of IoT or Internet of Things. (Nabil et al., 2017). We can say that this new context defines architecture as an interface, where several different materialities both real and virtual are applied. “Human-Building Interaction (HBI) frames HCI research and design within built environments, seeking to sketch the scope of an interdisciplinary area situated at the interface between HCI and the domains of architecture and urban design. Questions addressed have started examining how emerging interactive experiences are “spatiotemporally immersive.” Experiences that “are not discrete or limited to moments of interaction, but persist over time, and can be enacted at different temporal scales of adaptability” (Alavi et al., 2019, p. 2). Some examples exist that illustrate how an environment might be physically interactive or be related to an individual's psychophysiological data. Existing in the purely physical realm, Hylozoic Ground is an immersive, interactive environment created by Phillip Beesley and exhibited around the world in different renditions. It consists of thousands of light-weight digitally fabricated components that incorporate sensors and microprocessors. The environment cycles through processing the air and reacting to itself and its occupants. The environment is a living system where human interaction triggers breathing, processing and motion. The fronds of the elements act as a filtering membrane and chemical reactions take place in vials nested there. The environment quite literally reacts and changes according to the occupants present. The Hylozoic Series was created in the Living Architecture Systems Group, an interdisciplinary, transnational initiative that asks “Can architecture feel, and know, and respond to their occupants? Might buildings begin, in primitive ways, to come alive?” (Beesley & Armstrong, 2014, p. 8). An example that features both a physical component and digital interface is the D-Tower by Lars Spuybroek of NOX. This tower, built in the city of Doetinchem, the Netherlands, consists of three main parts: a physical building (the tower itself), a questionnaire and a website. The tower

turns different colors based on the emotions felt by the

Person

Building Induced Emotion

Building

Fig. 26 Al Bahar Towers, UAE

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Typical Relationship btwn Architecture and Emotions

(Richters, 2014)

Architecture as an Emotional Organic User Interface "OUI" T ADS

Person

Building Responds to Emotional Status

Emotional Status

CHANGE

Fig. 27 Buildings and Emotional Affects

State of the Art Fig. 28 Installation of Hylozoic Ground (Beesley, 2014)

townspeople as indicated in the self report questionnaire via the website. The tower visualizes the emotional states of different city occupants. The project becomes a physical manifestation of the affective state of the townspeople (V2, 2001). Saleh Kalentari is a researcher who uses virtual reality to present architectural variations to participants and test their biophysical response before the buildings are built. Kalentari and his team built a prototypical platform MoBI (Mobile Brain Body Imaging System) that monitors physical and conscious reposes as users interact with a virtual environment. Kalentari’s research establishes an evidence-based design approach that utilizes data collected pre-occupancy instead of post occupancy, which

Fig. 29 Interactivity and Motion (Beesley, 2014)

“can allow designers to gain crucial, objective feedback on their work—to help resolve problems or to justify the human benefits of a design—prior to the investment in its physical construction” (Kalentari, 2019, p. 1884). Kalantari also found very specific data regarding occupants preferences; when using virtual environments “data did not differ significantly between the actual classroom and the identical virtual one” (Kalantari, 2019, p. 1889).

(https://www.d-toren.nl/nl/)

(/en/)

D-toren is een kunstwerk in opdracht van de gemeente Doetinchem dat de emoties van de inwoners van Doetinchem peilt en in kaart brengt. D-toren meet dagelijks de mate van GELUK, LIEFDE, ANGST en HAAT aan de hand van verschillende onderwerpen.

D-Toren (/nl/) Lezen (/nl/lezen) Handleiding (/nl/handleiding) Deelnemen (/nl/deelnemen) Colofon (/nl/colofon)

Inschrijven (/nl/inschrijven)

Inloggen

" #

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

(/)

Gebruikersnaam Wachtwoord

Onthoud mij



Registreer

(/nl/inschrijven) Gebruikersnaam vergeten? (/nl/inschrijven? view=remind) Wachtwoord vergeten? (/nl/inschrijven?view=reset)

ADS

Inloggen

Fig. 31 D-Tower Instalation View (Spuybroek, 2003)

State of the Art Fig. 32 MoBI System (Kalentari, 2019)

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Fig. 33 MoBI System (Kalentari, 2019)

State of the Art

3.3

Emotions in Software Engineering Emotion Recognition is Software Engineering

has been gaining traction as a profitable new sector. First defined by Rosalind Picard, what is known as Affective Computing influences, relates to or arises from emotions. It is a multidisciplinary research field bridging computer science, psychology and cognitive science (Bota et al., 2019). This research field “investigates how technology enables human affect recognition, how emotions and sentiment can be simulated to embed emotional intelligence in interfaces, and how software systems can be designed to support emotion” (Novielli & Serebrenik, 2019). Affective computing exists in several manifestations: computer vision based applications like facial expression recognition, natural language processing applications like text sentiment analysis, audio analysis such as tone of voice recognition, and psychophysiological sensing. In 2019 the Institute of Electrical and Electronics Engineers published the Sentiment and Emotion in Software Engineering as a special issue of the Institute of IEEE Software Publication in response to the growing interest of the subject in the field. A range of software engineering companies have started to consider emotions in their work. Some examples involve emotion analysis in internal processes like the hiring and support of neurodiverse developers, such as Microsoft. Others utilize emotions to enhance productivity through emotional awareness (Novielli & Serebrenik, 2019). Physical manifestations of emotions such as facial expressions, while easy to collect, “present low reliability since they depend on the user's social environment, cultural background (if he is alone or in a group setting), their personality, mood, and can be easily faked, becoming compromised” (Bota et al., 2019, p. 140991) These constraints are not as applicable to physiological tellers of emotional state. Heart Rate and perspiration among others are not easily falsified and present a more authentic assessment into a subject’s emotional state (Shu et al., cited by Bota et al. 2019). These physiological signals are almost always collected in lab settings according to a media stimulus where machine learning is applied to aid in the signal processing and classification of 54

biosignals. Fig 34 shows this typical work flow process.

(EMG), heart rate variability, galvanic skin conductance, and electroencephalography (EEG) (Yoon & Wise 2014). Most often the use of psychophysiological sensors is performed in a lab setting where subjects are purposefully exposed to different media to induce certain emotions. This allows researchers to control the stimulus media as well as environmental factors. The sensor data is cleaned, processed and classified using Machine Learning algorithms trained against existing data-bases such as the DEAP (A Database for Emotion Analysis using Physiological Signals) Database which contains data from several sensor types from 32 volunteers who were recorded while watching 40 one minute-long music videos. Few studies have attempted to extract data-sets from outside of a lab setting in everyday circumstances (Bota

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

These sensors commonly include electromyography

et al., 2019). While data is collected in real-time it must

classification systems such as the iMotions platform,

ADS

their emotional label. There exist some real time data

be processed in order to classify the sensor signals into

the PLUX Biosginals platform and the wearable device Empatica. However at the time of this writing access to these real-time platforms was prohibitively expensive, making them less accessible. The field of Affective Computing is still very young and “scientific evidence of emotion remains one of the most challenging yet important issues in affective engineering” (Yoon & Wise

Fig 34. Schematic representation of a machine learning process for emotion 55

recognition (Bota et al. 2019)

State of the Art

2014 p. 222). Emotion classification, whether based on biosignals, computer vision or natural language processing, requires artificial intelligence to parse whatever data is being collected. This comes with it’s own limitations and issues. A recent study Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements warns that the current digital methods for extracting emotion from data are not always fully accurate. The study was composed by a team of experts in neuroscience, computer science and psychological science and reviewed over 1,000 studies on facial movements and emotions and determined that the current technology is not yet reliable enough and carries a lot of bias (Baron, 2019). Furthermore the AI Now 2019 report warns that Affect Recognition using AI which is “rapidly being commercialized for a wide range of purposes—from attempts to identify the perfect employee to assessing patient pain to tracking which students are being attentive in class… is built on markedly shaky foundations” (Crawford et al., 2019, p. 50). According to the report there is little evidence that current affect-recognition products have scientific validity (Crawford et al., 2019). Citing a study out of Berkley, “in order to detect emotions with accuracy and high agreement requires context beyond the face and body” (Chen & Whitney cited by Crawford et al., 2019). However the report underscores the growing popularity of this field “the emotion-detection and -recognition market was worth $12 billion in 2018, and by one enthusiastic estimate, the industry is projected to grow to over $90 billion by 2024” (Crawford et al., 2019, p. 50). Despite its current limitations, it is clear that Affective Computing is an inevitable development in the field of Software Engineering and product design. Though we cannot directly start implementing these processes, it's pertinent that architects start to imagine the implications of effective computing in the built environment.

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Fig. 35 Facial Expression Analysis (Dantone, 2012)

Fig. 36 Emotiv Products (Emotiv, 2020) T ADS

Fig. 37 Empatica Embrace Wearable (Empatica, 2020)

State of the Art

3.4

Emotionsal Response in Psychology Emotions are a feeling state that involves thoughts,

physiological reactions and outward behaviours. However psychologists have continued to debate about how exactly the mind and body relate to each other and a unified theory of emotions remains elusive. According to Levenson, emotions are a survival instinct preserved from natural selection “due to the need for an efficient mechanism able to mobilise and organise the selection of quick responses from highly differentiate and disparate systems when environmental stimuli pose a threat to survival” (Bota et al., 2019, p. 140992). The seeming common ground among researchers is that emotion is particularly difficult to define (Fortkamp, 2005). Scientific discussion of emotions can be traced to 1872 when Charles Darwin published The Expression of the Emotions in Man and Animals (Keltner, Oatley, Jenkins, 2013). Darwin investigated the physiological and behavioural components of emotions but ultimately regarded them as ‘expressions’ of feelings felt by animals and humans. Emotions were taken for granted as subjective states of experience until the early twentieth century (Griffiths, 2003). In the 1980s psychologists began to recognize the importance of affect in human experience, prompting much research related to emotionality and mood (Watson & Clark, 1997). However, several lines of thinking emerged defining affect as it relates to emotions in different ways. Most definitions of emotions are relative to a larger theoretical explanation of why we experience emotion or what causes emotions. Fig 38 is a diagram of just some of the different theoretical frameworks addressing the origins of emotions. Seeking to clear up the confusion, a seminal study Core Affect, Prototypical Emotional Episodes, and Other Things Called Emotion: Dissecting the Elephant was published in 1999 introducing the Circumplex model of emotions. Episodes of emotion vary along different dimensions like intensity, pleasure or degree of activation. The disparate experiences of riding a roller coaster or being chased by a bear can both have a high level of activation, though the former would be a more pleasurable experience. Factor analysis and multidimensional scaling are used for analyses of self-reported emotions, facial expression 58

and vocal expressions often yield two broad dimensions

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

that can be interpreted as a level of pleasure or arousal. However categories of emotions do not cluster at the axis which is what creates the circumplex as a multi dimensional structure (Russell & Barrett, 1999). It can be summarized that two points of view encapsulate the way that scientists model emotions. The first, such as the circumplex model, consider emotions “as a continuous function and models them across multiple dimensions, such as the affect polarity, or valence, associated with the pleasantness or unpleasantness of the emotional stimuli, and the arousal that is the level of activation, ranging from calm to excited” (Novielli & Serebrenik, 2019). Other theories reflect a more discrete viewpoint that seeks to identify a smaller set of basic emotions (Novielli & Serebrenik, 2019). The attention to measuring emotion was developed by psychologist David Watson who created

the Positive Affect and Negative Affect Scale measure

ADS

prior to the circumplex model in 1988. Watson examined such factors as which emotion words and terminology best measured the positive and negative dimensions, how time frames impacted participant responses as well as different

Fig 38.Theoretical Constructs of Emotoin (Fortkamp, 2005)

State of the Art

Fig. 39 Circumplex Model of Emotional Dimensions (Russell & Barrett 1999)

formats for the scale. However many studies since that time have shown that participants experience more than one concurrent affect responses which requires that researchers use a “battery of affect items to accurately differentiate emotional responses” (Marcus et al., 2017) The slider method utilized in the prototype for this thesis is modeled from one developed by psychologists George Marcus and W. Russell Neuman. In a relatively new study from 2017, they present an updated format, stemming from the original PANAS scale. This format is digital and utilizes sliders instead of radio buttons. They found that the slider format produces a cleaner and more “discriminating array of results than does the radio button format” as it captures continuous data (Marcus et al., 2017, p. 14). They also noted the optimal placement of the slider value starting position was at the midpoint as it had the least influence on participant responses.

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Fig. 40 Different Circular

Models of Emotions (Russell &

Barrett 1999)

State of the Art Fig.41 The PANAS Scale (Watson, Clark, and Tellegen, 1988)

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Fig. 42 Digital, Slider Based

Self Report User Interface

(Marcus, 2017)

Methodology

4. Methodology 64

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Methodology

4.1

Design Approach Emotions are a rather vast and unwieldy lot. Adults can

differentiate between a minimum of 50 to 100 different emotional states (Fehr & Russell, 1984 via Karasawa 1995). In order to realistically explore how to deal with such a behemoth topic, it is important to narrow the prototype exploration to the emotions most relevant to a target end user audience. Emotional research studies are generally conducted in response to a stimulus. Environmental psychologists Mehrabian and Russell devised the S-O-R (stimulus - organism - response) paradigm to “explain the relationship among stimuli, environments, responses, and behavioral intention in the context of marketing and consumer behavior research” (Yoon & Wise 2014 p. 222). This paradigm is illustrated in Fig 46. Instead of exposing participants to an emotional stimulus and seeking to evaluate or respond to their emotional state, this prototype will instead be geared towards a demographic that is already in an emotionally heightened state. It is common practice in Architecture to tailor a building’s design to it’s target end user. In this way design decision making can be clarified and different outcomes can be weighed according to the needs of the program required. The prototype for this thesis exists within this framework of an Organic User Interface. In this case the target user of Grieving Adults was chosen. This choice was based on the following factors. Firstly, this group is historically and particularly under-served in Western popular culture and therefore digital media. Furthermore, by aiming to serve grieving individuals, the difficult task of incorporating emotions into architecture becomes grounded in its universality. At the time of this writing, a global pandemic has not only caused families around the world to lose loved ones, but has also robbed them of their abilities to gather and process their grief. Some headlines reflecting this new condition are pictures in Fig 45. In their overview of the idea of Organic User Interfaces as architecture, Nabil et al. establish that the malleable nature of these interactive environments necessitates dialing them in to a target user group. According to Nabil, it is important from the perspectives of both the HCI and Architecture disciplines as “both depend on building their design ‘concept’ on deﬁning the 66

users” (Nabil et al., 2017). In an ideal world, an immersive

Mock-Up, early study

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Fig. 43 VR Space Website

ADS

Fig. 44 Overall System design

Methodology Fig. 45 Headlines Regarding Grieving During the Covid 19 Pandemic

environment such as the built prototype would serve as a tool for grieving adults that would help them process their loss by being emotionally responsive. The end user’s emotional status is determined as it relates to the content possibilities that exist in the VR space via a graphic user interface. Many papers have reported an increase in emotions recognition accuracy with an increase in physiological sensor modalities. However there is no clear evidence of which physiological sensors are most relevant, and public data sets for emotion recognition are limited and not based on daily-living scenarios (Bota et al., 2019). Because of the current limitations “self-assessment is widely the most commonly applied methodology in affect recognition state-of-the-art research, with self-reporting questionnaires being presented to the user. “ (Bota et al., 2019) This single GUI interface is not intended to assess the end user’s emotional status on a professional level or offer a comprehensive view of the end user’s emotional health in the way that mood tracking mobile applications or wearable devices seek to. Instead, the GUI is a vehicle for a tailored experience, where the relative emotional status will determine which content is shown in the 3D space. There are three mechanisms to ensure that every individual user has a unique experience in the space that is produced just for them. The first is that the database that the VR platform draws from is constantly being updated

with new text data. The second is that the emotional

based on the S-O-R model

(Yoon & Wise 2016)

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Fig. 46 Research framework

ADS

Methodology

readings of each user will always be unique. Lastly, the positioning of the text content within the VR space is generativity programmed so that the location of the text entries in space changes every time the game is initialized. This spatial experience of text is constantly reshuffling and never presents the same order of data twice. In this way, the content of the text and the form of the text are always evolving. The prototype was built using the Unity 3D Game Engine and a Vive Virtual Reality headset and controllers. The data-base was compiled manually from several literature sources and blogs, see Appendix X . Though it was not built for this study, the intent is that the data-base would also host user contributions via a separate stand alone website, pictured in Fig 43.

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

to a stimulus vs designing for individuals with a heigtened

ADS

Fig. 47 Exposing individuals

emotional state

Methodology

4.2

Virtual Reality as Architectural Practice In the 1990s Marcos Novak proposed using Virtual

Reality to create a new type of architecture “cut loose from the expectations of logic, perspective, and laws of gravity” (Silva, 2006 p. 2). Using terms like Liquid Architectures of Cyberspace and Transarchitectures, Novak put forth spaces conceived specifically for a virtual domain, one that does not exist in a physical world. The implications of Novak’s explorations are becoming more relevant as virtual worlds become layered onto and interactive with our everyday environments. Recently there has been a growing body of research around the merging of Human-Computer Interaction (HCI) and the built environment. An extension of Ubiquitous Computing, Human Building Interaction (HBI) is an aspiration to “transition from the ‘realm of artifacts’ to the ‘realm of architecture” (Alavi et al., 2019, p. 2). This field has started to address how “merging interactive experiences are ‘spatiotemporally immersive’... ones that are not discrete or limited to moments of interaction, but persist over time, and can be enacted at different temporal scales of adaptability” (p. 2). HBI can allow architects to use data in order to enrich occupants interaction with the built environment, where as “designing spaces has historically relied on making assumptions about occupants' comfort and desires” (Alavi et al., 2019). As we start to re-define architecture as a user interface it requires a reconsideration of many design fundamentals. Nabil describes OUI environments as “immersive rather than focused, when interacting with spaces rather than devices or building-sized interfaces rather than tabletops” (Nabil et al., 2017, p.96). Though Augmented Reality and the promises of spatial computing are presently starting to take hold, Virtual Reality is a more immersive tool that better serves the cause of interactive environments that engender emotional affordance. When they acquired the Virtual Reality platform Oculus, Facebook CEO Mark Zuckerberg explained that “strategically we want to start building the next major computing platform that will come after mobile” (Zuckerberg quoted by Hassan, 2014, p. 196). Though VR is not yet universally accessible it very much hails the beginning of a new media era. In his TED Talk The Birth of Virtual Reality as an Art Form, Chris Milk expounds 72

a techno-utopian vision for the future of VR:

format is fixed at its birth. Film has been a sequence of rectangles, from Muybridge and his horses to now. The format has never changed. But VR as a format, as a medium, isn't complete yet. It's not using physical celluloid or paper or TV signals. It actually employs what we use to make sense of the world. We're using your senses as the paints on the canvas, but only two right now. Eventually, we can see if we will have all of our human senses employed, and we will have agency to live the story in any path we choose. And we call it virtual reality right now, but what happens when we move past simulated realities? What do we call it then? What if instead of verbally telling you about a dream, I could let you live inside that dream? What if instead of just experiencing visiting some reality on Earth,

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

“The tricky thing is that with all previous mediums, the

you could surf gravitational waves on the edge of a black

That's not a virtual reality anymore. And honestly I don't

ADS

each other not using words but using our raw thoughts?

hole, or create galaxies from scratch, or communicate with

know what that's called. But I hope you see where we're going “ (Milk 2016).

Fig. 48 Park Borrowed from the EMMA project: Peaceful Park (Luis Alcaniz Raya et al. 2014)

Methodology

Fig. 49 Park Borrowed from the EMMA project: Anxious Park (Luis Alcaniz Raya et al. 2014)

In 2007 Riva et.al analyzed the possibilities of VR as an affective medium, or a “medium able to elicit different emotion through the interaction with its contents” (Riva et al., 2007, p. 46). In 1989, Jaron Lamier described VR as a simulated environment where people interact via “visual, aural and haptic devices, [where] the user can experience the environment as if he/she was part of the world” (Riva et al., 2007, p. 46). Though VR is associated primarily with its hardware consisting of a Head Mounted Display (HMD) and controllers, it can also be defined in terms of human experience. VR is a media through which users experience presence in a simulated environment. Presence in the Virtual Reality context is the sense that one is occupying a different world than their literal environment, the one that is simulated digitally in the HMD (Riva et al., 2007). In order to examine the affective capacity of VR, the researchers borrowed a VR park environment from the “EMMA” project, an immersive VR world built for the purpose of applied psychology. Riva et al. created three separate versions of the park environment by changing the shadows, lights, textures, music, sound and audio-visual features in order to induce two separate emotional states. The third park was kept as a neutral control group, while the first was adjusted to induce anxiety and the second was adjusted to induce relaxation. Sixty-one undergrad Psychology students took part in the experiment by visiting all three sites in random order. Each participant completed 74

two different styles of self report, the first paper and pencil

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T ADS

Fig. 50 Park Borrowed from the EMMA project: Nuetral Park (Luis Alcaniz Raya et al. 2014)

Methodology

Fig. 51 algorithmically composed Liquid Architecture (Novak, 1991)

questionnaire and then emotional and presence ratings during the VR experience in the digital environment. The scientists’ hypothesis was confirmed when the anxious park experience was shown to reduce happiness and positive effects and increase anxiety and sadness. Furthermore the relaxing park environment reduced anger, sadness and anxiety and negative effects while increasing quietness and happiness. Riva et. al highlight the importance of presence in their experiment, “In particular our study suggests that if a medium is not able to induce a feeling of presence, the affective responses might be low independently from the emotional content provided by it” (Riva, et. al, 2007, p. 55).” Presence has also been cited by Huttner and Sussanne as playing a crucial factor in the immersive success of virtual environments. VR’s immersive capacity, ability to affect emotions and media relevance make it a prime tool for architects to start dealing with the idea of emotion as input. It remains to be seen whether emotionally reactive VR spaces should be treated as prototypes for physical spaces, the possibilities seem wide open and it is important we start investigating them in architectural practice. The digitality of VR spaces in the context of emotions allows designers to engender their environments to be uniquely reactive to individuals, this individual customization is not warranted or realistically feasible in the built environment currently. 76

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Fig. 52 Spatial Computing

(Magic Leap, 2018)

Methodology

4.3

Text as a Spatial Medium: Immersive Typography How can text become a spatial medium? There is an

ancient learning method called the “Method of Loci” (MOL) that helps the practitioner to memorize large amounts of information. This method consists of mentally walking through a familiar environment and placing each item to memorize at certain locations in the physical space. To practice, the practitioner repeats this over and over but only in their mind’s eye. When it is time to recall the content that has been memorized the practitioner mentally walks through the familiar environment again and is able to ‘pick up’ the items they previously stored there. This is based on the principle of creating mental associations between the information to be remembered and other easy to recall concepts. Information can be associated with sounds, images, and spatial cues (Huttner & Robbert, 2018, p.1). The foremost principle of the MOL is the practice of navigating in the mind through an architecture or spatial environment. The effectiveness of this method “suggests a natural human tendency to use spatial context to memorize and recall information” (Huttner & Susanne, 2017, p. 2). In her book Narrative as Virtual Reality: Immersion and Interactivity in Literature and Electronic Media, Marie-Laure Ryan examines ideas about interactive text. She “introduces the concept of "ergodic design" (building on Aarseth) in order to discuss work that includes a feedback loop in its design, causing the act of reading/ viewing to produce alterations in the work” (Wardrip-Fruin, 2004, p.336). Cognitive scientist Yair Neuman says of text that they “do not only constitute our consciousness but also constitute the medium through which we reflect on our consciousness” (Neuman, 2010). He further elaborates “literature stands in its own right as a unique medium for illuminating the human existence to include the human psychology” (Neuman, 2010). When we read the written thoughts of others we tend to project our own self, reflected through the other as a sort of translocation. This is considered a form or “projective identification” where “the particularity of the other and his uniqueness is substituted for a replica of one’s self particles” (Neuman, 2010, p. 239). Osgood et al suggest that “meaning is a process linked to linguistic and situational variables of the 78

individual... an individual’s interpretation and expression

beyond the immediate experience” (Yoon & Wise 2014, p. 221). As in our prototype we are trying to engineer a sense of empathy towards the user as we engage with their emotions we will take advantage of this mechanism of reading self into the words of another. The digital space supplies the words of others, but does so specifically to match the current emotional status of the end user as indicated by self report in the VR space. Through the ideas of MOL and self projection in the words of another we can start to see how text can provide not only spatial cognition but empathetic as well. So how are the texts supplied to the virtual space? For the sake of the prototype a database of text entries that are related to the topic of grief have been collected. They have primarily been sourced from

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

of ideas rooted in his or her own prior experience, which is

well known literature on the subject. A sample of the

in the 3-D virtual world in different configurations. The

ADS

the end user enters the VR space these texts are floating

data-base can be found in Chapter 7, Appendix. When

end user is able to walk around and through the floating text which is also animated and colored. The text in the VR world is almost like a 3D word search where the end user can pick out certain words or phrases that catch their eye. The text in the 3D space is intentionally unclear, forcing the end user to move around and through it, building immersion. In order to examine more closely they must reach out and grab or capture the words they are interested in. This process is further illustrated in Chapter 5, Design Documentation. By utilizing text resources that are dynamically matched to individual users, the prototype seeks to engage not only the visual, but the cognitive within a Virtual Reality space, which is traditionally thought of as a dominantly audio-visual medium. This Virtual Reality space is a library. A library is defined most simply as a place where someone goes when they are typically in search of written words. This Virtual Reality space is called a library because it allows grieving adults to find words that are particularly relevant to their current emotional state. The idea of what Adam Varga dubbed “Immersive Typography” regarding his Augmented Reality text experiments efficiently summarizes this idea of spatiotemporally experiencing written text. 79

Methodology

Fig. 53 ad for kitchens that hides geometry in the newspaper layout (Felipe Salazar 2020)

Fig. 54 Giulio Camilloâ€™s Theatre of Memory (1519-1544)

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Fig. 55 Source Uknown

Fig. 56 An example of a Word

Search (Freeprintable.com,

2020)

Methodology Fig. 57 OMSE Augmented Reality Typography 2019

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Fig. 58 Immersive Typography

(Adam Varga, 2020)

Methodology

4.4

Prototype Description The user enters the Virtual Reality space into a

nondescript, dark environment. They are immediately faced with a GUI (graphic user interface) in the VR space. This interface menu asks the question “Thinking about the person you lost, how do you currently feel?” This question is intended to act as a stimulus to the target user group of grieving adults. Considering their lost loved one(s) will activate an emotional response that the VR space seeks to empathize with. Below this question are five vertical slider bars with a handle in the middle. The user can indicate their current emotional level of five different emotions commonly associated with grief: sadness, anger, gratitude, regret and hopefulness. This graphic user interface is based on the slider menu format given in Fig 59. The end user will drag the different sliders on the screen to indicate what emotion they are currently feeling. We will call this interface the “Emotional Self Report” and it is shown as step two in Figure 63, Prototype Workflow. At any point the end user may return to this GUI and can change their previously reported emotional condition in order to indicate how they are currently feeling. Upon pressing the submit button, each slider’s input value is logged and analyzed in two ways. Firstly, all slider values are compared against each other, the slider with the highest rating triggers the category of content that will populate the 3D VR Space. For example, if a user has a Sadness rating of 1.0, an Anger rating of 1.5, a guilt rating of 4.0 a gratefulness rating of 0.5 and a hopefulness rating of -0.5, their current emotional status is logged as Guilt. A For Loop iterates over the values of each slider and compares it to the one previous with the function (>=) until a highest value is determined. This is stored as the SavedHighestSliderMood() and triggers the Environment Manager to load a certain scene in a switch statement. Secondly the overall input value is mapped onto certain qualities of the VR environment and acts as a multiplier, effecting aspects such as color, animation or motion speed of the text objects and the overall spatial form the text takes in the environment. Different emotional readings trigger different spatial arrangements. Each word bobs up and down slightly, almost as if floating or reacting to wind. 84

The overall environment of the space reflects the mood

or game features that create the illusion of the sky are shown for each mood determined. The ground plane material is also referential to the evaluated mood. A user who has been evaluated as Angry will see the text floating in front of them in a tall, tower-like formation. Users who are feeling grateful are greeted with grasses and flowers while users who are feeling sadness walk along a stark, marble-like surface. Only the “Active” text, indicated by the brighter color of the text is select-able. If the end user sees a phrase or sentence fragment that they find interesting they can reach out and grab it. Upon colliding with the active text, the VR glove gives haptic feedback, increasing the immersive qualities of the space. The scale of the text and architectural placement of it purposefully obstructs the readability. Upon collision, or “Grabbing” of the active

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

determined by the slider value as well. Different “skyboxes,”

text, a second menu is shown on the VR screen that states

the end user can return to the main menu screen and re-

ADS

second menu allows for full readability. At any point in time

the entire text paragraph and the author and source. This

indicate their emotional state if the current environment is not working for them

Fig. 59 Graphic User Interface for VR Space

Methodology Fig. 60 Mapping VR Environment to Occupant's Emotions

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Fig. 61 VR Prototype Workflow

Methodology

Fig. 62 STEP ONE: Enter VR Space

Fig. 63 STEP TWO: Self Report Emotional Status

Fig.64 STEP THREE: Emotional Status Determined

ment Generated

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Fig. 65 STEP FOUR: Environ-

T ADS

Fig. 66 STEP FIVE: Texts populate enviornment to match mood , mood can always be changed

Proposal

5. Design Documentation 90

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Proposal Fig. 67 GUI Inside of VR Space

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Fig. 68 GUI Inside of VR Space Interaction

Fig. 69 GUI Inside of VR Space

Interaction

Proposal Fig. 70 Positive Affect Environment Mood Hopeful

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Proposal

Fig. 71 Positive Affect Environment Mood Hopeful

Author: Cheryl Strayed Reference: Tiny Beautiful Things: Advice on Love and Life from Dear Sugar

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Active Text Selection

Emotion: Hopeful Quote: " Most things will be okay eventually, but not everything will

be. Sometimes you'll put up a good fight and lose. Sometimes

ADS

you'll hold on really hard and realize there is no choice but to let go. Acceptance is a small, quiet room."

Proposal Fig. 72 Positive Affect Environment Mood Hopeful, Text placements axon

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Fig. 73 Positive Affect

Environment Mood Hopeful

Proposal Fig. 74 Negative Affect Environment Mood Sadness

100

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

101

Proposal Fig. 75 Negative Affect Environment Mood Sadness, Text placements axon

102

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

Fig. 76 Negative Affect

Environment Mood Sadness

103

Proposal

Active Text Selection Author: Cheryl Strayed Reference: Tiny Beautiful Things: Advice on Love and Life from Dear Sugar Emotion: Sadness Quote: " Nobody will protect you from your suffering. You can't cry it away or eat it away or starve it away or walk it away or punch it away or even therapy it away. It's just there, and you have to survive it. You have to endure it. You have to live through it and love it and move on and be better for it and run as far as you can in the direction of your best and happiest dreams across the bridge that was built by your own desire to heal"

104

Environment Mood Sadness

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Fig. 77 Negative Affect

ADS

105

Conclusion

6. Conclusion 106

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

107

Conclusion

6.2

Conclusion Though it is tempting to want to interface architec-

ture directly with promising devices and data analysis we must do so with reservations. As these systems are not yet stable enough for consumer consumption, by designing with them we severely limit our audience for the time being. However there are still ways we can build emotional affordance into our projects. What might start as a screenbased UI interface could evolve into an application that would connect with a wearable device in the future. The value is in starting to consider how these innovations in technology will inevitably allow us to interface with emotions as architects. Perhaps architects should approach the subject in the same ways we already approach designing for end users via program. Instead of using a sensor to evaluate an emotion that is hypothetically in response to some kind of condition or stimulus, we should instead emphasize the underlying emotional states of groups of end users and their needs. This requires that architects, when wishing to design for emotions, pinpoint shared human experiences that have a prominent emotional dimension or aspect. Grief is just one example of such a condition. It can be said that by creating emotionally responsive architecture via technology we run the risk of the kinds of negative outcomes that other forms of digital media have shown to incur. At the time of this writing, the Covid-19 Pandemic has highlighted how at a global scale digital media affects communities, perceptions and livelihoods. There have been calls for a technological Hippocratic Oath that would engender digital processes with the same considerations for well-being that those in the medical profession must regard (Sample, 2019). As the architecture profession becomes increasingly more technological we are faced with many of the same ethical dilemmas. It is extremely important to ground our initiatives in evidence based research, such as that carried out by Nanda and Saleh, even when prototyping environments that are playful or more abstract. We can combat some of these negative outcomes through intentional empathy. Interacting with emotions in the built environment is an inherently empathetic process as it requires architects to contemplate and plan for the emotions of building occu108

pants. In his meditation on empathy through semiotics,

definition “The best definition of empathy … is the capacity to think and feel oneself into the inner life of another person” (Neuman, 2010), p.235). He expounds that the “modern cognitive understanding of empathy [is] a kind of “mind reading”, a natural capacity or “endowment” that is deeply embodied in our neurology and the way our “mirror neurons” reflect the other’s mind” (Neuman, 2010). From this definition we can infer that empathy involves first and foremost a sharing of understanding between two parties, self or other. From this definition we can reframe the architect as “self” and the building occupants as the “other” to be empathized with. This research involves examining ways that a sense of empathy, or awareness of another’s experience can be facilitated in a digital environment.

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Cognitive Scientist Yair Neuman supplies us with Kohut’s

T ADS

Fig. 78 What is Empathy? (Oliveira, 2018)

109

Appendix Conclusion

7. Appendix

110

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

111

Appendix Conclusion

Texts sources regarding grief 1. Author: C.S. Lewis Reference: A Grief Observed Emotion: Sadness Quote: " Part of every misery is, so to speak, the misery's shadow or reflection: the fact that you don't merely suffer but have to keep on thinking about the fact that you suffer. I not only live each endless day in grief, but live each day thinking about living each day in grief." 2. Author: C.S. Lewis Reference: A Grief Observed Emotion: Sadness Quote: " No one ever told me that grief felt so like fear. " 3. Author: Cheryl Strayed Reference: Tiny Beautiful Things: Advice on Love and Life from Dear Sugar Emotion: Sadness Quote: " Nobody will protect you from your suffering. You can't cry it away or eat it away or starve it away or walk it away or punch it away or even therapy it away. It's just there, and you have to survive it. You have to endure it. You have to live through it and love it and move on and be better for it and run as far as you can in the direction of your best and happiest dreams across the bridge that was built by your own desire to heal" 4. Author: Philip Beesley Reference: An Unpublished Manuscript, December 12th 2019 Emotion: Hopeful Quote: " Instead of inert and dampening qualities, could soil offer a kind of precarious dĂŠtente, a manifold that resonates and shimmers, where new forms could spring forth from older times and elder presences? The burials and deep earth that deaths seem to always invoke could bring a particular kind of comfort opposite of a halo which emanates outward: the completely dispersed and immersive living ocean--and-earthen fields of those who have died seem to now shimmer inward toward me, making inward-rolling waves, engulfing me, billowing and quivering, passing through my transparent skin and resonating within the tangles of my organs and my bundled paths. The unspeakable suffering and the unthinkable pain they each bore shimmers inside my body too, with such pressure and weight that it sometimes engulfs me completely. Those inward-swelling tides might speak of erasure, infinitely falling. Yet instead of horror the sense that they are vividly here inside and outside my own body offers the quietest kind of elation: I am living them. I can live them. I will live them."

112

Reference: n.a. Emotion: Hopeful Quote: " You will lose someone you can’t live without,and your heart will be badly broken, and the bad news is that you never completely get over the loss of your beloved. But this is also the good news. They live forever in your broken heart that doesn’t seal back up. And you come through. It’s like having a broken leg that never heals perfectly—that still hurts when the weather gets cold, but you learn to dance with the limp." 6.Author: Cheryl Strayed Reference: Tiny Beautiful Things: Advice on Love and Life from Dear Sugar Emotion: Sadness Quote:

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

5. Author: Anne Lamott

" Nobody will protect you from your suffering. You can't cry it away or eat it away or starve it

better for it and run as far as you can in the direction of your best and happiest dreams across

ADS

survive it. You have to endure it. You have to live through it and love it and move on and be

away or walk it away or punch it away or even therapy it away. It's just there, and you have to

the bridge that was built by your own desire to heal" 7. Author: Anne Lamott Reference: n.a. Emotion: Anger Quote: " But you can’t get to any of these truths by sitting in a field smiling beatifically, avoiding your anger and damage and grief. Your anger and damage and grief are the way to the truth. We don’t have much truth to express unless we have gone into those rooms and closets and woods and abysses that we were told not go in to. When we have gone in and looked around for a long while, just breathing and finally taking it in – then we will be able to speak in our own voice and to stay in the present moment. And that moment is home." 8. Author: Kristin O'Donnell Tubb Reference: The 13th Sign Emotion: Guilt Quote: " Whoever said that loss gets easier with time was a liar. Here's what really happens: The spaces between the times you miss them grow longer. Then, when you do remember to miss them again, it's still with a stabbing pain to the heart. And you have guilt. Guilt because it's been too long since you missed them last."

113

References

8. References 114

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library” T

ADS

115

References

Armijo, J. M. (2013). Orientation study: Rotation Tower [Digital image]. Retrieved 2020,

from https://www.grasshopper3d.com/group/ladybug/forum/topics/orienta

tion-study-rotate-tower Baron, J. (2019, July 18). Tech Is Already Reading Your Emotions - But Do Algorithms

Get It Right? Retrieved May 25, 2020, from https://www.forbes.com/sites/jessicabar

on/2019/07/18/tech-is-already-reading-your-emotions-but-do-algorithms-get-it-right/ Beesley, P., & Armstrong, R. (2014). Near-living architecture: Work in progress from the

hylozoic ground collaboration, 2011-2013.

Bonanno, G. A. (2001). Grief and emotion: A social–functional perspective. In M. S. Stro

ebe, R. O. Hansson, W. Stroebe, & H. Schut (Eds.), Handbook of bereavement

research: Consequences, coping, and care. (pp. 493–515). American Psychological

Association. https://doi.org/10.1037/10436-021

Bota, P. J., Wang, C., Fred, A. L. N., & Placido Da Silva, H. (2019). A Review, Current

Challenges, and Future Possibilities on Emotion Recognition Using Machine Learning

and Physiological Signals. IEEE Access, 7, 140990–141020. https://doi.org/10.1109/

ACCESS.2019.2944001 Burry, M. (n.d.). Scripting Cultures. 274. Cambria, E. (2016). Affective Computing and Sentiment Analysis. IEEE Intelligent Sys

tems, 31(2), 102–107. https://doi.org/10.1109/MIS.2016.31

Del Castillo, G. (Director). (2019, September 4). 1:18:53 / 1:25:41 [00] Course Introduction

| GSD-6338: Introduction to Computational Design | Harvard GSD [Video file].

Retrieved 2020, from https://www.youtube.com/watch?v=R9Y54UkkArI&list=PLvxxY

ImPCApUXhX3te3IK32ileXHpzKY4&index=1 Ciuk, D., Troy, A. K., & Jones, M. C. (2015). Measuring Emotion: Self-Reports vs. Physio

logical Indicators. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2595359

Crawford, Kate, Roel Dobbe, Theodora Dryer, Genevieve Fried, Ben Green, Elizabeth Ka

ziunas, Amba Kak, Varoon Mathur, Erin McElroy, Andrea Nill Sánchez, Deborah Raji,

Joy Lisi Rankin, Rashida Richardson, Jason Schultz, Sarah Myers West, and Meredith

Whittaker. AI Now 2019 Report. New York: AI Now Institute, 2019, https://ainowin

stitute.org/AI_Now_2019_Report.html. Day, C. (2004). Places of the Soul: Architecture and Environmental Design as a Healing

Art (2. ed). Architectural Press.

116

Corbusier, L. (1927). Towards a new architecture. London: Architectural Press.

Dantone, A. (2012). Our approach estimates facial feature points from 2D images in

real-time. [Digital image]. Retrieved 2020, from https://www.semanticscholar.org/

paper/Real-time-facial-feature-detection-using-regression-Dantone-Gall/5dc056fe

911a3e34a932513abe637076250d96da/figure/0 Delminio, G. (2019). Theatre of Memory [Digital image]. Retrieved 2020, from http://

socks-studio.com/2019/03/03/spatializing-knowledge-giulio-camillos-thea

tre-of-memory-1519-1544/ Fortkamp, S. (2005). Body. Emotion. Architecture: A Phenomenological Reinterpretation.

University of Cincinnati. https://books.google.co.jp/books?id=9KWaDAEACAAJ

Emotiv [Digital image]. (2020). Retrieved 2020, from https://www.emotiv.com/

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

D-tower. (2011, April 20). Retrieved 2020, from https://v2.nl/archive/works/d-towe

Empatica. (2020). Empatica Embrace [Digital image]. Retrieved 2020, from https://www. empatica.com/research/

Eppstein, D. (2016). A topological ordering of a directed acyclic graph: [Digital image]. Re

ADS

trieved 2020, from https://en.wikipedia.org/wiki/Directed_acyclic_graph#/media/

File:Topological_Ordering.svg Freeprintable.com. (2020). Pizza Word Search [Digital image]. Retrieved 2020, from http://allfreeprintable.com/pizza-word-search Glancey, J. (2014, October 21). Sensing spaces: Emotional buildings. Retrieved July 1,

2020, from https://www.bbc.com/culture/article/20140130-can-buildings-be-emo

tional Higgins, T. (2019). [UK companies are at the forefront of machine learning emotional

recognition]. Retrieved 2020, from https://www.telegraph.co.uk/technolo

gy/2019/05/30/british-companies-pioneering-ai-reads-emotions-will-revolutionise/ Hollan, D. (2008). Being There: On the Imaginative Aspects of Understanding Oth

ers and Being Understood. Ethos, 36(4), 475–489. https://doi.org/10.1111/j.1548-

1352.2008.00028.x Hsu, H.-L., Chang, Y.-L., & Lin, H.-H. (2015). Emotional Architecture—A Study of Tadao

Ando’s Genius Loci Design Philosophy and Design Syntax. 3(6), 8.

Huttner, J.-P., & Robbert, K. (n.d.). The Role of Mental Factors for the Design of a Virtual

Memory Palace. 5. 117

References

Huttner, J.-P., & Susanne, R.-B. (n.d.). An Immersive Memory Palace: Supporting the

Method of Loci with Virtual Reality. 10.

Karasawa, K. (2011). Cognitive antecedents of emotions: Findings and future directions.

Japanese Psychological Research 1995, Vol.37, No.1, 40-55

Lawson, B. (2010). Healing architecture. Arts & Health, 2(2), 95–108. https://doi. org/10.1080/17533010903488517 López, D., Charbel, H., Obuchi, Y., Sato, J., Igarashi, T., Takami, Y., & Kiuchi, T. (n.d.). Hu

man Touch in Digital Fabrication. 12.

Malloy, J., & Aarseth, E. J. (1998). Cybertext, Perspectives on Ergodic Literature. Leonardo

Music Journal, 8, 77. https://doi.org/10.2307/1513408

Marcus, G. E., Neuman, W. R., & MacKuen, M. B. (2017). Measuring Emotional Response:

Comparing Alternative Approaches to Measurement. Political Science Research and

Methods, 5(4), 733–754. https://doi.org/10.1017/psrm.2015.65

Mazuch, R., & Stephen, R. (2005). Creating healing environments: Humanistic architec

ture and therapeutic design. Journal of Public Mental Health, 4(4), 48–52. https://doi.

org/10.1108/17465729200500031

Milk, C. (Director). (2016). The birth of virtual reality as an art form [Video file]. Retrieved

April 3, 2020, from https://www.ted.com/talks/chris_milk_the_birth_of_virtual_reali

ty_as_an_art_form?language=en Nabil, S., Plötz, T., & Kirk, D. S. (2017). Interactive Architecture: Exploring and Unwrapping

the Potentials of Organic User Interfaces. Proceedings of the Tenth International

Conference on Tangible, Embedded, and Embodied Interaction - TEI ’17, 89–100.

https:// doi.org/10.1145/3024969.3024981

Neuman, Y. (2010). Empathy: From Mind Reading to the Reading of a Distant Text. Inte

grative Psychological and Behavioral Science, 44(3), 235–244. https://doi.

org/10.1007/s12124-010-9118-7 Novielli, N., & Serebrenik, A. (2019). Sentiment and Emotion in Software Engineering.

IEEE Software, 36(5), 6–23. https://doi.org/10.1109/MS.2019.2924013

Oliveira, I. (2018). What blocks our empathy in the design thinking process? [Digital

image]. Retrieved 2020, from https://blog.prototypr.io/what-blocks-our-empathy-

in-the-design-thinking-process-a09766198308 118

Richters, C. (2014). 'Mashrabiya Facade' at Al Bahr Towers [Digital image]. Re

trieved April 1, 2020, from https://www.archdaily.com/510226/light-mat

ters-mashrabi yas-translating-tradition-into-dynamic-facades/5384afb

2c07a8044af0000bd-light-matters-mashrabiyas-translating-traditon-int o-dynamic-facades-photo Riva, G., Mantovani, F., Capideville, C. S., Preziosa, A., Morganti, F., Villani, D., Gaggioli,

A., Botella, C., & Alcañiz, M. (2007). Affective Interactions Using Virtual Reality: The

Link between Presence and Emotions. CyberPsychology & Behavior, 10(1), 45–56.

https://doi.org/10.1089/cpb.2006.9993 Russell, J. A., & Barrett, L. F. (1999). Core affect, prototypical emotional episodes, and

other things called emotion: Dissecting the elephant. Journal of Personality and

Social Psychology, 76(5), 805–819. https://doi.org/10.1037/0022-3514.76.5.805

Sample, I. (2019, August 16). Maths and tech specialists need Hippocratic oath, says aca

demic. Retrieved June 3, 2020, from https://www.theguardian.com/sci

ence/2019/ aug/16/mathematicians-need-doctor-style-hippocratic-oath-says-ac

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

ademic-hnah-fry

Silva, C. A. (2006). Liquid Architectures: Marcos Novak’s Territory of Information. 6.

ADS

Spuybroek, L. (2011, April 20). D-tower. Retrieved July 26, 2020, from https://v2.nl/archive/works/d-tower Wardrip-Fruin, N. (2004). Book Review: Marie-Laure Ryan, Narrative as Virtual Reality:

Immersion and Interactivity in Literature and Electronic Media, Johns Hopkins

University Press, Baltimore, 2001. 399 p. Computers and the Humanities, 38(3),

335–338. https://doi.org/10.1007/s10579-004-0817-x

Watson, D., & Clark, L. A. (1997). Measurement and Mismeasurement of Mood: Recurrent

and Emergent issues. Journal of Personality Assessment, 68(2), 267–296. https://

doi.org/10.1207/s15327752jpa6802_4 Whitemyer, D. (2010). The Future of Evidence-Based Design. 4.

Yang, T. (2020). Examples of eight facial expressions by older (upper row) and

young (lower row) models. [Digital image]. Retrieved 2020, from https://journals.plos.

org/plosone/article/authors?id=10.1371/journal.pone.0231304 Yoon SY., Wise K. (2014) Reading Emotion of Color Environments: Computer Simulations

with Self-Reports and Physiological Signals. In: Watada J., Shiizuka H., Lee KP., Otani

T., Lim CP. (eds) Industrial Applications of Affective Engineering. Springer, Cham.

https://doi.org/10.1007/978-3-319-04798-0_17

archdaily.com/13358/the-therme-vals

119

Zumthor, P. (2009). The Therme Vals [Digital image]. Retrieved 2020, from https://www.

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

122

Design Studies Master's Thesis Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

Emotion as Input: A Prototype for Creating a Mood Responsive Virtual “Library”

2020 Advanced

ADS

Department of Architecture Graduate School of Engineering

Bunkyō

1 Hongō ku

Tokyo 〒 113

ADS

Eng. Bldg. 1, Room 415

The University of Tokyo

8656

Japan www.t-ads.org

123