Alex Orsholits / Yiyuan Qian

Spatial Audio as an Intuitive Architectural Interface

Tokyo Advanced Design Studies Department of Architecture The University of Tokyo

2020 Advanced Design Studies Master's Thesis

T ADS

Alex Orsholits Yiyuan Qian Spatial Audio as an Intuitive Architectural Interface Emulating the Capacities of Human Sound Localization & Navigation with Spatial Audio in an Urban Context

Š Alex Orsholits and Yiyuan Qian, 2020 Version for digital distribution | 178mm*254mm

Master's Thesis

Spatial Audio as an Intuitive Architectural Interface

Supervising Professor: Yusuke Obuchi Additional Course Support: Shuta Takagi Yusuke Komata

ADS

Navigation with Spatial Audio in an Urban Context

T

Emulating the Capacities of Human Sound Localization &

Spatial Audio as an Intuitive Architectural Interface

Alex Orsholits & Yiyuan Qian

2020 Advanced Design Studies Master's Thesis Spatial Audio as an Intuitive Architectural Interface: Emulating the Capacities of Human Sound Localization & Navigation with Spatial Audio in an Urban Context

Editors: Alex Orsholits Yiyuan Qian Book Design: Alex Orsholits Yiyuan Qian Book Edition Design: Jan Vranovsky Printed in Tokyo, Japan Advanced Design Studies Typeface: F Grotesk by Radim Peško © 2020 Obuchi Lab, T

ADS

Department of Architecture Graduate School of Engineering The University of Tokyo Eng. Bldg. 1, Room 415 7

3

Bunkyō Tokyo

1 Hongō ku 113

8656

Japan www.t-ads.org www.yiyuan.space www.alexorsholits.art

Acknowledgements 1

Ambition and Objectives 1.1 Spatial Interpretation — From 2D to 3D

A. Orsholits

1.2 Outdoor Navigation — About Intuition and Experience

Y. Qian

1.3 Spatial Audio as an Intuitive Interface

A. Orsholits

1.4 Previous Digital Fabrication Research Utilizing Spatial Audio 2

Background and Precedents 2.1 Human Sound Localization 2.2 Precedents in Non-visual Navigation Systems

3

A. Orsholits

3.1 Coordinate System

Y. Qian

3.2 Head Motion

Y. Qian

Y. Qian A. Orsholits & Y. Qian

Navigation System Development 4.1 General Background

A. Orsholits

4.2 Position Tracking — GNSS

A. Orsholits

4.3 Hardware and Network Mapping

A. Orsholits

4.4 Software Development

Y. Qian

4.5 Sound Source Placement Algorithm

Y. Qian

4.6 Wearable Design

Y. Qian

4.7 Feasibility Test

A. Orsholits & Y. Qian

Projection and Outcome 5.1 General Comments, Outcomes and Future Research 5.2 Beyond Unconditional Navigation

6

References

7

Appendix

A. Orsholits & Y. Qian A. Orsholits

ADS

3.5 Summary — Auditory Environment Guideline

A. Orsholits

T

3.4 Computer-generated Spatial Audio

5

Y. Qian

Auditory Environment Guideline

3.3 Audio Source

4

Y. Qian

Spatial Audio as an Intuitive Architectural Interface

Table of Contents

We would like to take this opportunity to thank the University of Tokyo School of Engineering for providing us with this once in a lifetime opportunity graciously offered to us through the T_ADS Digital Fabrication research lab and its accompanying technical facilities. We would like to especially thank Professor Yusuke Obuchi for his extensive contribution and guidance of our thesis, along with the lab's assistant staff members Shuta Takagi, Yusuke Komata, and of course

Spatial Audio as an Intuitive Architectural Interface

Acknowledgements

Kakuko Kato for her invaluable administration aid during the final hours of our thesis submission. Our sincerest gratitude is reserved for our advisors Professor Tomohiro Tachi, professor Masahiko Inami, professor Tomohiro Amemiya, and professor Tomoyuki Gondo, all from the University of Tokyo. Finally, we are infinitely thankful for our friends and families who have admiringly put up with our endless T

endeavours and demands.

ADS

Spatial Audio as an Intuitive Architectural Interface

T

ADS

Alex ORSHOLITS

https://alexorsholits.art/

Spatial Audio as an Intuitive Architectural Interface

T

ADS

Yiyuan QIAN

https://yiyuan.space/

Ambition and Objectives

12

Spatial Audio as an Intuitive Architectural Interface

T

ADS

1 Ambition and Objectives

13

Ambition and Objectives

Architecture design is commonly understood as the design of the space. The designed space, or built environment, is experienced by users through various methods of sensation input. The quality of the design is defined by the perceived spatial characteristics. Instinctively, our prevailing perception of space is of course visual. Due to this, most architects have been focusing on users' visual sensation of space, turning architecture into a subfield of visual design.

We aim to examine another aspect of sensing space — auditory perception. Rather than designing physical space with the help of real materials intended to generate specific sound ambiances and effects, like concert halls, temples, and a plethora of other applications, we wish to address a unique line of research that is contrary to the traditional, generally construction-based or material-based, methods and notions of architecture design by establishing virtual and interactive spatial experiences through the assistance of modern digital technologies.

Explicitly, we intend to conduct thorough research into digital technologies, and establish the limitations and potentialities of implementing interactive acoustic systems as a form of assistance and guidance to better the comprehension of space, through the emulation of human's sound localization capabilities. Based on our research, our goal is to develop an outdoor navigation system based on spatial audio for urban pedestrians, which accomplishes two fundamental intentions: 1. Challenge the traditional norm of visual interpreted navigation, utilizing spatial sound as an intuitive guidance interface to reduce users' cognitive load.

2. Expand the concept of navigation beyond that of a functional tool, towards a new urban interface that is capable of augmenting the implicit relationship each person has with ever-complexifying cities.

14

Spatial Interpretation — From 2D to 3D

Since the very existence of the concept of community, the necessity of representing cultural notions has been a primary preoccupation for the emancipation of a collective consciousness. Beyond the immediate needs of survival, the enrichment of a collective has led to the forage for a common rhetoric based on the pragmatically observable immediate environment. The use of 1D descriptive or 2D illustrative information to convey complex spatial concepts has been

Spatial Audio as an Intuitive Architectural Interface

1.1

the common form of rhetoric within nearly all industries and disciplines involved in the processing of spatial information.

From the scientific and theological mapping of stars, to the triangulation of landmasses, the reality of space has been exclusively mapped to a 2D plane. This "flattening" of spatial data is ingrained in the development of humanity and represents one of our main driving factors behind the

spatial representation has been employed for convenience of

ADS

the use of varying degrees of abstraction as a method of

T

development of intuitive architectural interfaces. Evidently,

distribution of information, as well as out of simple necessity due to distinct technological shortcomings throughout history.

However, it is indisputable that the urge to represent reality with the largest amount of information has been thoroughly examined by countless societies, and although it may not represent the least amount of abstraction possible for the time, the importance of a collective comprehension has mostly been a leading factor in the slow but steady development of methods of representation from two dimensions to three dimensions.

A seemingly well known, yet surprising instance of spatial representation is that of Ancient Egyptian society. Recognized for its application of pictograms as language, the conscious decision to employ this same rhetoric for over three millennia to ensure a long-lasting ideological cohesion through significant socio-economic collapses as well as humanity-scale political struggles deserves a deeper look into the intricacies of this method of representation. Due to the distinct nature of the pictogram, similar standards are applied in theological and architectural disciplines. Within religious and political structures, the employed graphical language for architectural plans and other illustrative information remains coherent with are depicted on a single plane to convey spatial position,

15

the descriptive nature of pictograms. Multiple points of views

Ambition and Objectives

as well as characterization of objects. To the detriment of realistic representation, the expression of space encapsulates both description and illustration. The reason for this is to maximize the capacities of the 2D plane to represent subjects in their entirety, rather than the more modern approach of representing a personal perception of space. To achieve this, all forms of expression of space are defined by a collective of artisans [1], in which the communal values of comprehension [1] Ziegler, Christiane, and Jean-Luc Bovot. 2011. Art Et ArchÊologie: Égypte Ancienne.

Paris: Ecole du Louvre.

outweigh spatial accuracy. Space is depicted as a readable medium in this case, in which multiple points of view, axons, and any other element that assists the comprehension of the symbolic of the illustration are superimposed and "woven" into a continuous plane. Similar approaches to this class of spatial representation can be found throughout history,

Fig. 1.1-1 Garden of Sennefer (reproduction) - 18th Dynasty (circa 1549-1292 BC) WikiMedia Commons public domain

with more recent examples dating back to the 15th century to 16th century in which cartographers included the facades of buildings in their maps to provide a sense of the aspect of villages and their respective landmarks.

16

Spatial Audio as an Intuitive Architectural Interface T ADS

Fig. 1.1-2 S. Ougspurger plan of Aare extract 1751 - Bern City Archives

On the more personal side of spatial representation, the desire to depict a space that is seemingly real to the observer is apparent. Already in the 1st century AD, the inside walls of Pompeii houses were "expanded" into extra rooms or vast outdoor gardens through perspective paintings designed to create such an illusion. From the point of view of the person within this space, they could achieve the perception of a virtual reality. These are of course the first known single point perspectives which were later mathematically perfected several centuries later to depict specific points of view, flattened on a 2D plane, with no possible derivation of the perception. Rather than ensuring the comprehension of the represented object or space, the aim of such a system was to match to the best abilities of the time the inherent capability of humans to perceive space with depth. Indeed, standing perfectly in front of the representation plane results in a mathematical match to what we are capable of observing with our own inherent perception. Upon moving away from the perfect position, the illusion quickly falls apart - it is only capable of a purely static 3D representation. The utility beyond this specific point of observation evidently falls short information of prehistory.

17

in depiction of space compared to the pictographic spatial

Ambition and Objectives

Fig. 1.1-3 Interior decorations of Villa dei Misteri - Pompeii AD 79.

Link: https:// insidemybackpack.com/ portfolio/pompeii-italy/

Fig. 1.1-4 Architectural Veduta Francesco di Giorgio cira 1495 (Wikimedia Commons Public Domain)

To combat this limitation, artists in the early 20th century applied superimposition methods, similar to those of prehistory, to expand the portrayal of their personal perception. Futurists implemented the factor of time to define the represented points of view, while the artists of analytical cubism employed the use of several points of view around an observed object that produce a near 360 degree flattening of the subject.

Fig. 1.1-5 Bottle and Fishes - Georges Braque 1910-1912 (Tate Art Museum)

18

to question the very essence of representation and perception and how the boundaries between the two can be dissolved, "so that art becomes reality", as Naum Gabo articulated in his Essay Sculpture: Carving and Construction in Space [1]. Writing from the point of view of sculptures as a historical method of

[1] Circle: International Survey of Constructivist Art - Praeger 1971 reprint (originally 1937),

3D representation, Gabo senses "a new method of expression"

an almost 300-page art book.

in which the "sensation of space will become for us a more

Nicholson and Naum Gabo and

It was edited by the artists Ben

elementary and everyday emotion the same as the sensation

the architect Leslie Martin with

of light or the sensation of sound". He wishes to reformulate

Barbara Hepworth.

the layout being designed by

Spatial Audio as an Intuitive Architectural Interface

Simultaneously, artists, architects and philosophers started

the act of sculpting, from a predominantly material-based definition of mass towards a mass that in turn depicts a space. Through this he wishes to manifest his personal spatial perception into a universal perception that can be understood by all. Naum Gabo also admits that such a transformation of the discipline of sculpting is impossible in his time and would remain a challenge for future generations. Nevertheless, he

the common history of art definition, Gabo articulates his

ADS

representation. The first is that of abstraction. Rather than

T

brings up two compelling facets of his notion of 3D spatial

opposition to this definition and describes the act of creating artworks, abstract or not, to be, by definition, an abstraction of the reality, "as no material form and no natural event can be re-realized". From this, we can estimate that the act of spatial perception is not derived from the perfect reproduction of reality, rather a simulation is adequate. The second is the "problem of Time" and, by extension, motion. For a "space-idea" to be "Absolute or Constructive", Gabo verbalizes the concept of a new method of execution based on similar principles as that of music, the only form of a long-established art that is time-driven. He believes that Time can play an essential role in a higher level of spatial perception. With music as the master of perception through time, other methods of representation should be examined in a novel way to include such attributes, as an enrichment of the representation, and as a breakdown of boundaries between our inherent spatial perception capabilities and the 3D representation of a space.

"I do not hesitate the affirm that the perception of space is a primary natural sense which belongs to the basic sense of our psychology. [...] I say we experience this sense as a reality, both internal and external."

spatial design? Due to computers being able to mathematically

19

What does all this entail in the current era of computerized

Ambition and Objectives

define 3D space with relative ease, industries requiring precise design of complex geometries at a multitude of scales have largely adopted these new design tools. Yet, the translation of this 3D data into an inherently understood spatial perception, rather than the current flattening of information for human interfacing during the construction process within architecture, has only been marginally researched and can be summarized by the augmentation of mainly sight. Through Virtual Reality headsets, Augmented Reality systems, Mixed reality glasses, we inject, with varying degrees of success, the digital space directly into our inherent notion of space.

This indeed somewhat responds to the constructivist ideals of Naum Gabo. The representation for all intents and purposes becomes reality. We believe it is now time to augment other human senses capable of spatial perception and research Fig. 1.1-6 Two Cubes (Demonstrating the Stereometric Method) - Naum Gabo 1930 - Tate Collection

the digital interface of these senses within the discipline of architecture. Through sound, we aim to breathe new life in the act of navigation with an inherent sense of location and space.

20

Spatial Audio as an Intuitive Architectural Interface

T

ADS

21

Ambition and Objectives

1.2

Outdoor Navigation — About Intuition and Experience

Outdoor navigation refers to the ability to determine the position and direction in an environment. Although humans, as well as most animals, navigate based on a series of integrated inner cues, such as self-motion, as well as outer cues, such as visual landmarks. Outer cues play a much more significant role in human outdoor navigation.

Navigation on the ocean represented one of the major hurdles civilisation had to solve for voyages to the four corners of the globe. Polynesians some of the first to develop a multitude of processes to navigate through the ocean, solely based [1] Sextant: An instrument with a graduated arc of 60° and a sighting mechanism, used for measuring the angular distances between objects and especially for taking altitudes in navigation. (Definition form Oxford Dictionary) [2] Kamal: a celestial navigation device that determines latitude.

on the observation of the natural elements including sun, wind, ocean currents, and the behaviour of creatures. This is acknowledged by the majority to be the earliest form of opensea navigation. However, the special techniques were only endowed to a handful of navigators. Later, to facilitate the act of navigation, various tools were developed to enable higher positioning accuracy, such as the sextant[1], sea astrolabe or a simple kamal[2].

Since the earliest city developments, high-rising landmarks have been consciously used by urban designers as visual signs for the citizens' navigational requirements. Such landmarks consist of, but not limited to, church towers and obelisks.

[3] Lynch K. 1960, The Image of the City, MIT Press: Cambridge, MA. Lynch included his five-year study about several American cities, especially on how observers comprehend the urban space and form mental maps, in the book.

Kevin Lynch, in his text on The Image of the City[3], defined "landmarks" as one of the five essential elements which help to shape the image of a city. These visual navigation cues, unlike the navigation tools mentioned above, serve as not only a tool for indicating orientation, but also as a cultural icon specific to the area which builds up a personal representation of the space in regard to each individual.

In early eastern city planning, landmarks like churches and theaters were rarely constructed. However, the idea of using high-rising nodes which are visible from a distance to shape the city, especially streets, is apparent, and still visible in today's urban fabrics. The current Tokyo street pattern was

[4] Yamanote and Shitamachi are traditional names for area classification of Tokyo.

Although the definition is vague, shitamachi refers to relatively lower lands with lower

initially formed in the Edo period. Although the network of streets seems to be disorganized and organic in form, some of their orientations were admittedly decided by the direction of hills or mountains. The central Shitamachi area[4] of Edo

22

social status.

was built in accordance with the direction of Mount Fuji and

[5] According to Maki F. 1980,

Mount Tsukuba[5]. The natural landmarks were "borrowed" to

city with a hidden past. P89

shape a scenic view as well as to navigate citizens.

Spatial Audio as an Intuitive Architectural Interface

Fig. 1.2-1 An aerial photo of paris. Photo: AFP/Patrick Kovarik, 2014

T ADS

Fig. 1.2-2 The urban fabric formation of a central Shitamachi area with regard to Mt Fuji & Mt Tsukuba.

(Original description: Organizationof the central portion of Edo in the early to mid-17th century. From Shinjiro Kirishiki) Maki F. 1980, city with a hidden past. P90.

Rather than using "nodes" or "landmarks" as visual cues for urban outdoor navigation, some of the more modern architects and designers grew fond of employing an ambiguous but intuitive method of navigation — sequences. This theoretical layout was first brought to architecture through cubism with its multiple perspectives in depicting an object. Architects started to value the "fluidity of space", which was an emphasis of the successive experience of the user inside the space, instead of observing buildings from a fixed point of view. In urban design theories, ideas about serial visions were adopted by several, including Moholy-Nagy's "vision in motion"[1], and Edmund Bacon's advocates about "successive towns spaces"[2]. In lieu of a single focal point as a clear navigation target, the forms a substantial hint for the space coming in the future. A

Vision In Motion. Chicago: Paul Theobald.

[2] Bacon. 1974. Design Of Cities. London: Thames and Hudson.

23

entire experience of the surrounding space over a timeline

[1] Moholy-Nagy, L. 1946.

Ambition and Objectives

simple example would be the Southern Chinese or Japanese gardens, in which people explore by following the hints given by the arrangement of the plants, corridors, stairs and various elements, instead of following any obvious navigation signs.

Rather than the navigation function, these design beliefs emphasize the creation of a pleasing experience for users, as well as providing a certain freedom through vague orientation clues.

Fig. 1.2-3 Wen Zhengming, 1535 1551. Garden of the Inept Administrator (Zhuozheng Yuan).

On the right is part of thirty-one painted views of the garden by Wen, depicting the succesion of scenes within a single garden.

Image from The Met Collection Link: https://www. metmuseum.org/art/collection

With the advent of digital technology, real-time navigation with two-dimensional digital maps became the more popular choice for individuals, especially pedestrians. This method, however, flattens the spatial information onto a twodimensional plane and displays navigation cues on the screens instead of showing them directly in the built environment.

Though not hard, users have to learn to read or decode this two-dimensional information. In digital maps, not only the graphical information is interpreted, but also the names of the places, numbers for distances... Overflowing information appeared on mobile device screens, occupying pedestrians' visual input channels, and significantly interfering with the [1] More discussions about navigation with digital maps on mobile devices can be found in 4.1 (Chapter 4, Section 1)

way that people interact with the physical cities[1]. Besides the popular visual display navigation systems, descriptive audio is often used as a form of navigational assistance, especially for people who have difficulty keeping their eyes on the map,

24

like visually impaired individuals or drivers. Furthermore, navigation devices with tactile cues have recently started

Spatial Audio as an Intuitive Architectural Interface

to appear, lightening the load on human visual input as well as providing more intuitive options for acquiring navigation information.

Navigation in the virtual world, especially video games, has similar evolution paths as the history of navigation in the physical world. Although fundamentally different from reality, most role-playing games (RPG) storylines are position based — the game tasks update as the user explores new areas within the game world. Most RPG games in the early 2000s are based on a linear map system where players can move on the map freely, but only if they follow the preprogrammed track. They have to explore the map following the designed sequence. Clear instructions are provided to the user to follow the route "correctly". On the other hand, the concept of open-world has become popular among RPG game designs. Players, in this case, are able to explore the

T

world freely. Instructions are given in a less obvious and less

ADS

binding manner, such as through colour and lighting changes of certain areas which may attract players who are interested to explore. The aforementioned theories employed in early urban visual navigation design are also used, such as highrising points in the distance.

To summarize, there are two clear trends in the development of human outdoor navigation methods. First, navigation is

Fig. 1.2-4 (left) A screenshot of The Legend of Zelda: the Minish Cap, 2004.

evolving from being regarded as a professional skill towards

(video game published by

being common knowledge, or even a completely intuitive

Fig. 1.2-5 (right)

Nintendo)

apparatus with no need for training. Second, although the

A screenshot of The Legend

functional aspect of navigation tools are no doubt important,

2017. (video game published by

the entertaining side of navigation is being increasingly valued.

of Zelda: Breath of the Wild, Nintendo)

Navigation systems are no longer solely treated as tools, but rather methods to shape pleasing experiences for users. In the application development part of this thesis, these two trends leading the research.

25

of navigation systems represent our fundamental objectives

Ambition and Objectives Fig. 1.2-6 A rough categorization of navigation methods in the dimensions of intuitiveness and the their additional value in user experience

Figure 1.2-6 is a simplified categorization of each navigation method as discussed above. In the Third Quadrant, navigation is seen as a professional skill, in which the functionality is valued the most. We can classify the majority of ancient open-sea navigation methods here. For in-land navigation, the problem has been articulated in too many ways that it either becomes a kind of "hobby", such as hiking shown in the Second Quadrant, or integrated into city design principles.

Especially in the field of architecture and urban design, most visual navigation icons or spaces are byproducts of design, whose original aim was to plan a functional, safe, and pleasing built environment, as shown in the First Quadrant.

The digital navigation developments, as shown in Quadrant Four, represent solutions that are developed to be easy to use for the untrained end-user. However, these methods resolve only the functional aspect of navigation at this current stage.

This is partly due to current technological limitations, such as positional tracking deviation, device size limitations, and insufficient interface methods. Facilitating an interchange between Quadrant Four and Quadrant One, by, for example,

26

enabling a certain entertaining aspect in navigation, is one of the major objectives of this thesis.

Spatial Audio as an Intuitive Architectural Interface

T

ADS

27

Ambition and Objectives

1.3

Spatial Audio as an Intuitive Interface

The current visual systems have great difficulty in communicating the complex spatial relationships contained within contemporary, high density, multilayer cities and therefore revert to simplified representations, solely optimized for navigation. The disconnect between the representation and the user's perception, especially under the function of navigation, undermines the visual capacities of humans.

Indeed, we surmise that the exceptional perceptive efficiency of sight should not be hindered by large cognitive loads such as map-reading to find one's way in an environment.

We therefore believe, in a similar line of thought as haptic navigation systems, for example, that sound is, in the very least, able to offload these essential cognitive loads from our main architectural interfacing sensibilities as it already has been doing for several centuries. That being said, unlike the aforementioned haptic-based navigation systems, sound [1] Serafin, Stefania, Michele Geronazzo, Cumhur Erkut, Niels C. Nilsson, and Rolf Nordahl. 2018. "Sonic Interactions In Virtual Reality: State Of The Art, Current Challenges, And Future Directions". IEEE Computer Graphics And Applications 38 (2): 31-43.

has a distinct advantage: the capability of describing threedimensional space with similar precision as that of sight, if executed properly. Rather than it being a mere navigational tool like most implementations of navigational guidance, sound represents a completely parallel spatial referencing apparatus to our visual perception. As Serafin et al. (2018) states[1]: The auditory modality possesses unique features. Unlike its visual counterpart, auditory perception is always on, since we cannot close our ears. Thus, this sensory channel always flows with information about the surrounding environment, regardless of whether we are pay[ing] attention to it or not.

This enables us to simplify, and even, augment the act of navigation. By employing portable audio sources that have already been adopted as normality in urban environments, we are able to achieve direct emulation of sound cues without any flattening of information, enabling what can be considered a grafted-on apparatus that the human body simply accepts as an inherent capacity. Our initial goal was to develop a new and improved method of navigation, yet soon realized that spatial audio can add an entirely new layer of interfacing capable of enriching our relations with architecture, all while completing the given navigational tasks.

28

Furthermore, as a result of the 360° omnidirectional nature of hearing, we are confident that spatial audio responds to

which require an increasingly three-dimensional approach to design and perception. Sound, or rather, the act of hearing, is the only other sense that extends far beyond our bodily, or proprioceptive, range. It can even extend beyond sight.

Through astute urban digital mapping, we are able to integrate virtual, three-dimensional, interface cues within the urban environment that can be heard before they can be seen, if comparing to visual AR or MR applications. A final citation from the very same paper, thoughtfully summarizes another

Spatial Audio as an Intuitive Architectural Interface

the increasing complexity of multilayer contemporary cities

fundamental aspect of auditory perception, as well as one of the reasons for employing spatial audio as a means of advanced three-dimensional interpretation, as well as a cutting-edge navigational tool. Sound as an apparatus is an intuitive architectural interface.

Visual perception is superior in terms of spatial resolution,

perceive the surrounding environment. Auditory perception,

ADS

entails that we have to turn our heads or bodies in order to

T

but it is inherently directional. Our limited field of view

on the other hand, is omnidirectional.

29

Ambition and Objectives

Visualization of a theoretical virtual soundscape overlaid onto the city

30

Spatial Audio as an Intuitive Architectural Interface

T

ADS

31

Ambition and Objectives

32

Previous Digital Fabrication Research

Spatial Audio as an Intuitive Architectural Interface

1.4

Utilizing Spatial Audio The authors of this thesis have been directly or indirectly involved in the following Digital Fabrication Research projects previously conducted within Obuchi Lab. These research projects represent the basis of our motivation to actively pursue the development of the theoretical and technical aspects of spatial audio systems within the context of digital fabrication and architecture. Brief introductions and discussions about these previous projects will be elaborated in the following section.

1.4.1

PAFF

PAFF[1] (Projectile Acoustic Fiber Forest, 2017) is one of

[1] Project Supervising

the University of Tokyo Digital Fabrication Lab (DFL)'s

OBUCHI, Prof Jun SATO. It is

sound guidance system was developed as the method of construction for PAFF, revealing personal differences based

20 people. Please refer to the link for further information: http://t-ads.org/dfl/pavilion-

ADS

machine interaction. Rather than usual visual guidance, a

developed by a team of over T

pavilion fabrication projects which is based on human-

Professors: Prof Yusuke

paff-projectile-acoustic-fibreforest.html

on individual perception of space.

Specifically, each constructor of PAFF went through an acoustic test to record their ability to aim at virtual sound sources placed in various positions on site.

Fig. 1.4-1 Sound localization test of PAFF, using a tripod for aiming, Unity for spatial sound generation and grasshopper for data recording and analysis.

Image credit: DFL

Fig. 1.4-2 Part of the result of sound localization test of PAFF (two subjects) Image credit: DFL

33

Ambition and Objectives Fig. 1.4-3 (up) A photo of the aiming & shooting tool PAFFZOOKA Image credit: DFL Fig. 1.4-4 (down)

During the construction phase, each individual shot coconut fibre projectiles with the help of PAFFZOOKA, a blower fan driven shooting tool as shown in figure 1.4-3. The goal was to

An aerial photo during the

in the direction where the virtual sound source was positioned.

construction of PAFF pavilion

Based on the hypothesis that each individual's deviation

Image credit: DFL

distribution is consistent or predictable, the system helps by offsetting the position of the virtual sound source, so that the user aims at the real position based on their tested deviation pattern.

[1] See more information about the structural design in: Furuichi, Shohei, Shuntaro

Throughout the shooting phase accomplished by a multitude of users, the coconut fibre accumulates in a series of hammock

Nozawa, Machiko Asahara,

structures and forms the "roof" of the pavilion. Structurally,

Priya Murugeswaran, Yuqing

PAFF adopted a self-stabilizing iterative algorithm[1] to balance

Shi, Ziyi Wu, Yusuke Obuchi and Jun Sato. “Using non-linear logic for construction of self34

stabilizing hammock structure.�

AAG (2018).

the coconut fibre distribution. This algorithm is used to generate the next "instruction", or the position of the virtual sound source for the users.

Spatial Audio as an Intuitive Architectural Interface

Discussions PAFF is a digital fabrication project in which spatial sound guidance is used as a method to guide non-professional participants to "construct" with their natural intuition. Although various aspects are included in the project, in consideration of the thesis theme, the discussions below are limited to those related to spatial audio guidance system in PAFF. Tracking is crucial to setting up a virtual spatial audio environment in which users can enjoy free head movements.

PAFF uses HTC VIVE trackers[1] to track the rotation of users' heads and HTC VIVE controllers[2] for the rotation of the PAFFZOOKA tool. As the VIVE tracking system depends on infrared lasers emitted from the fixed base stations, the tracking is severely limited in outdoor use, especially under strong sunlight[3]. This specific tracking system also limits the size of the construction area to around 10 by 10 meters.

controllers are both accessories for HTC VIVE virtual reality system. They have functions of absolute position and rotation tracking, basic haptic feedbacks etc.

[3] See O. Kreylos,Lighthouse tracking examined (2016).

Link: http://doc-ok. org/?p=1478

ADS

consistent sound localization performance. PAFF streams

T

The setup of virtual spatial sound environment is key for

[1] [2] HTC VIVE trackers and

music from Spotify to users equipped with Bluetooth in-ear headsets. The spatial sound is generated through Oculus Spatializer[4] in Unity, based on the position and head rotation provided by Grasshopper for Rhinoceros via UDP data transmission. A virtual room with unique reverberation characteristics was set up within the Spatializer. The inherent

[4] A Univty plugin for rendering spatial audio.

Link: https://developer.oculus. com/downloads/package/ oculus-spatializer-unity/

processing latency of Grasshopper, the transmission latency of the Bluetooth headset, and the virtual environment not corresponding to the construction site had a significant negative bias on the spatial audio aspect of the project.

Despite the technical issues discussed above, the major motives of PAFF were appreciably achieved, which motivated a succession of further research. [...]we believe that the role played by digital tech- nologies in

[5] Part of the beliefs of DFL

design and construction does not necessarily have to make

(Digital Fabrication Lab), the

them more efficient and streamlined. Their potential is

The paragraph is from PAFF,

instead focused on personalising the fabrication process.

[5]

University of Tokyo 2017. Chapter 1 Statement

Some notable motives achieved in the project are the employment of human intuition, in this case, the inherent ability to localize a given sound source, as well as the digital augmentation of said ability. The project managed to create rather than a mundane construction task.

35

an entertaining construction process, a "game" if you like,

Ambition and Objectives

1.4.2

Shooting Game

From the PAFF project, we took the guidance and shooting aspects and repackaged them into a new project that we [1] This project is developed

by Yiyuan Qian, Alex Orsholits and Joe Li, instructed by Prof.

Obuchi Yusuke in DFL, the University of Tokyo 2019-2020

shall title "Shooting Game"[1]. Rather than developing a second version of PAFF, the main goal of the Shooting Game was to revise the spatial audio issues present in PAFF and develop a demonstrative interactive featurette of how digital technologies can augment human capabilities of sound localization.

For research purposes, we ran, similarly to PAFF, a series two-hour tests with a selection of users prior to a public demonstration of the system. These tests were designed to extensively map out each individual's aiming ability and generate a correction profile. Rather than a tripod-mounted cannon, we decided to mount all hardware to the user directly. This somewhat reduced the offset between the sound localization and targeting abilities. The correction profile functioned in the same manner as PAFF: it defined an offset vector that, when applied to the virtual sound sources (targets), the user's aim would be corrected to aim at the correct spot, within reason.

Fig. 1.4-5 A group of photos during sound localization tests in Shooting Game. Photo: Alex Orsholits & Yiyuan Qian

36

Spatial Audio as an Intuitive Architectural Interface T ADS

The interactive shooting game was showcased in front of a crowd in an auditorium. Random seats were selected as the "target". Those sitting in the selected seats were given a box to catch incoming projectiles. The previously tested users shot sponge balls based on where they thought the target was in space, based on our spatial audio guidance system. help of the previously generated correction profiles.

An example of the result of sound locailzation test of user2.

Image: Yiyuan Qian Fig. 1.4-7 (down) Shooting Game showcased in an auditorium.

Photo: Alex Orsholits

37

The Users' targeting capabilities were augmented with the

Fig. 1.4-6 (up)

Ambition and Objectives

Discussions For positional tracking, the shooting game employed the same system as PAFF: a VIVE tracker for head rotation and position, and a VIVE controller mounted to the handheld autoloading cannon. Positional and rotational data is processed without any added latency directly in Unity with the native SteamVR plugin. In consideration of the prerequisites required for a shooting game scenario, room-scale indoor tracking proved to be sufficient.

Our selection of audio source was driven by the enticement of better audio localization capacities. Instead of IEMs, we chose the Grado GW100, and open-back bluetooth headphone.

This selection allowed for a significantly more accurate reproduction of binaural cues (Adamas and Boland, 2010), as well as free, wireless, movement on-stage to verify the position of the sound source.

The largest difference between PAFF and this shooting game is the cannon. as mentioned before, we designed a wearable autoloading cannon that employs a manufactured foam ball projectile. PAFF on the other hand had an air-powered cannon mounted to a hardpoint that employed DIY coconut fibre-based projectiles. The change from air to a motorpowered launching mechanism, as well as the adoption of manufactured projectiles led to a drastic improvement in shooting consistency.

Fig. 1.4-8 The design of head-mount and arm-mount in Shooting Game.

Design & Photo: Joe Li

38

Comparison and Discussions

Spatial Audio as an Intuitive Architectural Interface

1.4.3

Below is a comparison of the virtual spatial audio environment setup between PAFF and Shooting Game.

PAFF

Shooting Game

tracking system

HTC VIVE

HTC VIVE

aiming tool

tripod-mount

arm-mount

coordinate system

interaural polar

single-pole

coordinate system

coordinate system

headphones

Bluetooth IEM

open-back headphones

spatializer

Oculus Spatializer

Oculus Spatializer

data processing

Grasshopper

Unity

T

project name

ADS

Despite various upgrades in hardware and software, both PAFF and Shooting Game still share common limitations.

The most conspicuous one is that both projects largely rely on human's sound localization in primarily elevation, which represents a substantial limitation in localization precision due to most spatializers not being able to accurately reproduce elevation information[1]. This led to a distinct lack of consistency

[1] Further discussion about the

in our altitude-based user measurements. Correction profiles

can be found in 3.4.1

limitation in current spatializers

on this axis simply did not provide an adequate aiming ability.

Secondly, the specific type of spherical coordinate system adopted in both projects is not ideal for user error analysis.

Double-pole spherical coordinate systems have been proved to be the most efficient and unbiased methods for sound localization performance analysis[2]. That being said, the amount of error caused by an improper coordinate system

[2] Leong, Philip, and Simon Carlile. 1998. "Methods For Spherical Data Analysis And

was relatively negligible due to both projects' limited range

Visualization". Journal Of

of direction. Indeed, the azimuth axis totaled 90°, while the

191-200.

elevation axis totaled 40°.

Neuroscience Methods 80 (2): and Knudsen, EI. 1982.

"Auditory And Visual Maps Of Space In The Optic Tectum

Thirdly, the choice of tool used for aiming significantly influences the result, especially for the hearing map testing. mounted pointer in PAFF, resulting in the intuitive recentering

Neuroscience 2 (9).

39

Although an arm-mounted pointer replaced the tripod-

Of The Owl". The Journal Of

Ambition and Objectives

of the aiming system on the user, rather than a separate coordinate system, the gap between perception and action still causes significant variability in the results, and causes great difficulty in predicting and correcting the user. Studies on the choice of method for reporting a perceived sound source direction included an evaluation on verbal reports, head [1] Populin, Luis C. 2008.

"Human Sound Localization: Measurements In Untrained, Head-Unrestrained Subjects Using Gaze As A Pointer".

Experimental Brain Research 190 (1): 11-30. and Bahu, HÊlène, Thibaut

pointing, pointing with tools like a gun, a laser beam, etc[1].

These studies suggested that egocentric methods ensure more accurate results than allocentric methods, and methods which involve smaller body movement, such as pointing with only head rotation, ensures more accurate results, especially for higher elevation directions.

Carpentier, Markus Noisternig, and Olivier Warusfel. 2016.

"Comparison Of Different Egocentric Pointing Methods For 3D Sound Localization Experiments". Acta Acustica United With Acustica 102 (1): 107-118.

Fourthly, the employed tracking system leaves much to be desired in terms of ubiquitous implementation. The limited tracking area of just 10 by 10 metres, along with the requirement of setting up base stations in every chosen environment illustrate the fundamental limitations in using this system as an intuitive urban guidance system.

That being said, the shared achievements in both projects motivate the ever-growing research on spatial audio as an intuitive architectural interface. The usage of spatial sound as a directional guidance system was proven to be functional with many healthy users without prior training. Not only that, this method of spatial sound guidance represents a novel development compared to current visual guidance systems, and provides little to no challenge for most users, thus raising their interest in using such a system. In other words, most users found the process to be functional, interesting, and entertaining.

40

Spatial Audio as an Intuitive Architectural Interface

T

ADS

41

Background and Precedents

42

Spatial Audio as an Intuitive Architectural Interface

T

ADS

2 Background and Precedents

43

Background and Precedents

2.1

Human Sound localization

In order to utilize spatial sound as an intuitive human-computer interface, especially for directional information, it is necessary to have a profound understanding about sound localization.

Sound localization refers to the listener's ability to localize the source of sound, or to tell the locus of the source of sound.

The perception of both direction and distance are included in sound localization.

2.1.1

Cues for sound localization

Different from localization through visual systems which maps the optic acquisition directly onto retina, this ability of sound localization depends on the cues from the interaction between the sound waves and the listener's physical parameters.

Azimuthal Cues [1] Middlebrooks, John C., and David M. Green. 1991.

"Sound Localization By Human Listeners". Annual Review Of Psychology 42 (1): 135-159

Middlebrooks and Green (1991)[1] summarized the specific cues on which human azimuthal sound localization ability is based: Interaural level difference (ILD), Interaural phase difference (IPD) and Interaural phase difference (IPD).

ILD appears to be the most obvious cue, which is caused by the blockage of head resulting in a sound pressure level difference. The specific pattern of such blockage depends on the wavelength of the sound source as well as the dimension of the listener's head. In other words, the amount of the “sound shadow" behind the subject's head depends on the frequency of sound source due to diffraction. The threshold of human detectable ILD is generally 0.5 to 1.0 dB (Grantham, 1984; Yost and Gourevitch, 1987) while ITD is also an obvious cue for azimuth sound localization. According to the experiment conducted by A.W. Mills (1958), there is a maximum of 0.6 milliseconds ITD for normal human head dimensions.

Although normal human threshold of detectable ITD depends on the sound frequency and varies greatly between subjects, according to Zwislocki and Feldman (1956), subjects are able to detect around 10 to 60 microseconds ITD, as 10Âľs corresponds to around 1 degree in azimuthal change.

IPD can be another cue while encountering some lowfrequency sound which may pass the listener's head with diffraction, causing negligible ILD. In such circumstances, the listener can distinguish the offset of the sound wave

[2] Zwislocki and Feldman (1956), Trahiotis and Robinson 44

(1979).

phase. However, as many researchers[2] point out, human IPD sensitivity is very limited for high-frequency sound source above 1.5kHz, unlike animals.

Spatial Audio as an Intuitive Architectural Interface

With all cues mentioned above, humans are able to locate sound source azimuthal direction with a certain precision.

According to A.W. Mills' research, as shown in figure 2.1-1, the precision is highly related to sound source frequency for all subjects. For frequency below 1000Hz, the distinguishable angle remains small; as the frequency grows up to 1500 - 2500Hz, the sound source direction becomes relatively ambiguous to most subjects, and the precision increases as the frequency become even higher and drops again reaching extremely high frequency.

Fig. 2.1-1

Elevational Cues In the case that the sound source is placed on the median plane, however, none of the cues above (ILD, ITD, or IPD)

Minimum audible angle between successive pulses of tone as a function of the frequency and the direction of the source measured for angles

can help for human sound localization because the sound

(bottom to top at left hand side)

arriving at both ears appear to have no difference in level,

Mills, "Auditory Localization",

0°, 30°, 60° and 75°

phase or time. This issue leaves vertical localization of sound

in Tobias, ed. Foundations of

a "blind spot".

Press, 1972, p. 310

Auditory Theory, Academic T ADS

Monoaural pinna cues are less obvious cues for sound localization. D. W. Batteau (1967) provided the first research about the influence of pinna shapes on sound localization.

Based on experiments and mathematical calculations, he provided the evidence that a specific external ear shape transforms the sound signal in a specific way, which is the reason that monoaural pinna cues are also called spectral shape cues. The convolution serves as an essential cue for human sound localization, especially for vertical spatial information. On the other hand, distorting the pinnae of a listener can effectively disable the listener's ability to detect sound source vertical direction.

The acuity of vertical sound localization based on pinna cues, however, is far less than that of horizontal sound localization although minimal detectable angular difference varies depending on the individual. According to Perrott and Saberi (1990), minimal audible angle in elevation is much larger than

Fig. 2.1-2

that of azimuth to the same listener.

difference required to perform

Minimum azimuth / elevation the MAA task.

Perrott, David R. and K Saberi.

Distance Cues All cues mentioned above, however, are to assist to localize the direction of sound source but not distance. As researches have shown, human auditory distance perception accuracy

“Minimum audible angle thresholds for sources varying in both elevation and azimuth.”

The Journal of the Acoustical Society of America 87 4 (1990): 1728-31 .

is far more limited than that of direction, leaving this subject with minimal research. perception include both perceptual factors like sound level[1]

45

According to Kolarik et al. (2016) the cues for distance [1] DJ Coleman, 1963

Background and Precedents

[1] Gardner (1969), Shinn-

and non-perceptual factors like the subject's familiarity with

Cunningham, KopÄ?o and Martin

the audio as well as the environment[1]. The article also points

(2005).

out that human perception of audio distance is much more effective in peripersonal space than that of extrapersonal space (also proved by Serino, Canzoneri and Avenanti, 2011).

Van Opstal (2016) suggested that the cues for distance remain ambiguous without knowledge of the absolute traits of the sound source, especially in free far-field sound localization.

Also, Relative distance judging is more effective than an absolute one because the level difference can be a distance cue, according to Zahorik et al. (2005). Furthermore, Simpson and Stanton (1973) showed that even head motion does not facilitate better distance judgments.

To summarize, the cues for directional sound localization includes both binaural cues and monoaural cues. Binaural cues, which majorly help to detect azimuth, includes ITD, ILD, and IPD. The efficiency of all three cues depends on the listener's head dimension, as well as the sound frequency band. Monoaural cues, which majorly helps to detect elevation, refers to pinna cues. Pinna cue is unique to each subject because it is a convolution of the audio source based on each listener's physical pinna shape.

2.1.2

HRTF

Head related transfer functions (HRTF) is a transformation to summarize the impact on the sound received of an individual's head shape, which includes the size and density of the head, the distance between ears, the external and inner ear shape, etc. HRTF is unique to each individual, and provide essential "customized" cues for sound localization.

Studies have shown that each individual's HRTF is crucial for their ability of sound localization (Wenzel et al., 1993), especially for elevation and front-back confusion (Matsumoto et al., 2003). As discussed above, azimuth is implied by ITD, ILD, and IPD, which are partly related to HRTF (distance Fig. 2.1-3 An example of front (continuous line) and back (dotted line) HRTF. Image source: https://lmkprod. com/what-is-hrtf-briefexplanation/ [2] Dummy heads are designed to replicate average-sized human heads, usually used for 46

binaural recordings.

between ears and head size), but cues for elevation (pinna cues) are entirely decided by HRTF. Since changing the external or inner ear shapes can distort the sound reaching a listener's eardrum, it surely affects the individual's ability in sound localization. In most studies, dummy heads[2] are used to represent individual differences in HRTF (Gardner & Martin, 1994 and Algzi et al., 2001). They consist of a head of a certain size and interchangeable ears to simulate different ear shapes. At the place of eardrums,

In computer-generated spatial sound, HRTFs are necessary to render individualized spatial audio. However, measuring each individual's HRTF is a tedious and high-precision required task. It may not be feasible to measure HRTFs of each potential subject of the spatial audio system. Some researchers (Gardner & Martin, 1994 and Algzi et al., 2001) measured a large amount of subjects' HRTF behavior in a uniform setting and published the database for further study

Spatial Audio as an Intuitive Architectural Interface

microphones are installed to analyze received sound waves.

of person-to-person differences. Although there is a possibility to replace the measurements with simplified options like taking photos (Torres-Gallegos et al., 2014), the result remains low precision.

Wenzel et al. (1993) showed that in audio played with headphones, while interaural difference (ITD, ILD and IPD) is readily rendered in computer synthesized audio, the spectral

front-back. However, several recent studies (Mendonรงa et al.,

ADS

resulting in the subjects' confusion in elevation as well as

T

cues are significantly distorted by non-individualized HRTFs,

2012, Stitt et al., 2019 and Cho et al., 2006) suggested that the errors caused by using non-individualized HRTF can be reduced through training, and the effect of training can last for a decent amount of time, but the increase in performance varies between individuals.

To summarize, HRTFs are essential to rendering realistic spatial sound for individuals. However, due to the diversity of each individual's HRTF, measurement for all subjects is not likely to be feasible in practice. Using non-individualized HRTF may be acceptable with enough training to have the user adapt to a certain HRTF. In general, the performance in elevation and front-back detection may be limited even in such a situation.

47

Background and Precedents

2.2

Precedents in Non-visual Navigation Systems

Among currently available devices and developed researches, voice command in audio display and vibration feedback in tactile display is the most popular.

2.2.1

Auditory Guidance

Descriptive Audio Auditory navigation applications majorly depend on descriptive language, which refers to giving the users commands by speech. Among the evaluation of ten available VI navigation devices in the literature review of Roentgen et al. (2008), nine of them use voice commands as the major user interface.

Most commercially available applications depend on [1] An all-in-one handheld talking GPS navigation device developed by HumanWare.

voice commands. The product Trekker Breeze +[1] provides visually impaired people with human voice descriptive audio guidance. The voice tells the users their whereabouts as well as commands for orientation and distance, for example, "Next intersection is in 196 feet". On top of the navigation function, it provides a simple audio-based searching function for nearby facilities as well. Micro-navigation handheld device

[2] Research on technology designed to substitute for missing visual cues for those with limited or no sight. See more information at: http://www.geog. ucsb.edu/~marstonj/ TALKINGSIGNS.HTM

is also widely available. Talking Signs[2] Remote Infrared Audible Signage (RIAS) scans the immediate surrounding environment around the user and provides human-voice information. Marston et al. (2006) studied the performance of a system named PGS to provide similar voice command for VI users based on GPS position tracking and IR for obstacle detection. However, all devices above are highly limited by GNSS accuracy at that time.

[3] Wakana Sugiyama, 2019 See full article: https://www. blog.google/products/maps/ better-maps-for-people-withvision-impairments/ [4] A descriptive-audio based navigation software for indoor and outdoor use. See more information at: https://www.blindsquare.com/

Other than all-in-one navigation device, numerous audiobased guidance software are available on smart phones.

For example, the detailed voice guidance in Google Maps[3] designed for visually impaired users. Another application Blindsquare[4] serves similar features. However, these smartphone-based applications are severely limited by the precision of the smart devices' position tracking ability, which is far beyond the acceptable threshold.

Spatial Audio Although not popular, spatial sound is used as part of the user interface in some of the products or proposals. Gallay et al. (2013) mentioned spatial sound as a novel and highly

48

potential user interface for guidance of visually impaired individuals with lower cognitive load. However, the limitation

remains to be researched. NAVIG (Katz et al., 2012) is an M&O guidance system for the visually impaired developed in a highly professional environment, for both indoor and outdoor use including complex tasks like staircases and narrow winding sidewalks. It contains voice commands as usual but rendered with spatial sonification Max[1] with corresponding sound

[1]Max, a visual programming

trajectory in each indoor or outdoor environment. It consists

multimedia developed by

of a GNSS module for tracking, an IMU for head rotation, bonephones instead of traditional headphones as the output

language for music and Cycling '74

Spatial Audio as an Intuitive Architectural Interface

of distal information that can be provided by spatial sound

device, cameras, and other necessary parts. However, as the need for processing power required a whole laptop to be mounted. Although spatial audio here is not used for direction guidance but a general image of a more realistic environment experience, it is an inspiring precedent. Zhu et al. (2019) studied subjective user feedback from visually impaired individuals and proposed an idea of visually impaired runner

audio for general direction. However, the article failed to

ADS

orientation respectively: vibration for obstacle and spatial

T

guidance consisting of two types of feedback for mobility and

accomplish the wearable device due to several technical issues (lack of tracking precision, missing IMU module, the limit of earphones in spatial audio, etc.).

Apart from portable navigation devices, spatial audio is used as guidance display in other situations. Rutherford (2001) depicted an indoor emergency situation in which spatial audio is used for building evacuation. Studies about navigation through spatial sound for wayfinding in a virtual environment (Lokki and Grohn, 2005) also suggested that spatial audio cues alone are possible to navigate human, and in the case where both visual and auditory cues are used, the efficiency in navigation would be higher than any single kind of cue alone.

Comparison between Descriptive and Spatial Audio The popular method of using voice guidance should be questioned. As Gallay et al. (2013) suggested, it is a common mistake for the design of guidance for visually impaired individuals to be the same way as sighted people, which is based on allocentric satellite maps and generate commands like "turn left at next corner". In the study of Klatzky et al.

(2006), the performance of spatial sound instruction versus spatial language, which is to tell the user "left", "right" or "straight", was compared. The subjects, although not visually traveled under spatial sound to complete a virtual path. It is

49

impaired, showed less time consumed and shorter distance

Background and Precedents

one of the convincing evidences that spatial audio can be a potential way of guidance in way-finding, providing sufficient information about direction in an intuitive way. Another article

by Ordonez et al. (2002) suggests that the navigation cues through spatial sounds are more helpful in avoiding collisions than a simple warning tone for blindfolded users.

2.2.2

Non-auditory guidance

Although audio guidance is the most popular for navigation systems for visually impaired individuals, there are various other feedback display methods, including tactile, haptic, and even neural.

Pieralisi et al. (2017) introduced a readily developed and tested infrastructure for VI marathon runners, or any fixed-track guidance for VI individuals. It consists of a light RF generator mounted on the user as a belt, and two wires constructed along the marathon track. While reaching the vicinity of the track's edge, the user will receive a vibro-tactile warning from both arms. Although the system is fully constructed, it revealed issues in a test with a VI marathon athlete, indicating the importance of choosing constant-lasting feedback to the user. Velรกzquez et al. (2018) provided an outdoor navigation device receiving GPS data from a smartphone and gives feedback to the user with a tactile display inserted into shoes. "BuzzClip" (Bin Liu, 2017) is a button-size attachable device designed for VI individuals. It detects obstacles in the immediate surrounding environment as the user moves and reflects it as vibration. Amemiya et al. (2005) provided a hand-held haptic device, Buru-Navi, for direction navigation.

Instead of common vibration, the device generates a pseudoattraction force which intuitively guides visually impaired users for obstacle avoidance as well as wayfinding. Kon et al. (2017) proposed a haptic walking navigation system using hanger reflex to have users "naturally" turn to the desired direction.

On top of the haptic displays, Orion Visual Cortical Prosthesis [1] See more information about Orion at: https://secondsight. com/

System (by Second Sight[1]) uses a camera to capture normal human vision, and send the signal through a pre-implanted wire to the user's visual cortex, "constructing" a mapping of the real world directly onto the visual cortex. However, the project needs surgery and a long learning curve because the constructed vision is not intuitive to understand. The

50

user interface is novel in technology but still requires further research and tests.

Summary

Although the main method of available navigation application display is visual-based, there is a large amount of research effort into non-visualized navigation systems, for both mobility and orientation guidance. They can be roughly categorized as: audio interface (majorly through descriptive audio), tactile interface (majorly through vibration) and neural interface.

A number of applications mentioned above, especially spatial

Spatial Audio as an Intuitive Architectural Interface

2.2.3

audio and haptic based ones, are in research stage and have great issues in precision if used in actual outdoor situations.

On top of technical problems, it is not hard to see that in almost all precedents of navigation systems, especially for visually impaired people, the experience aspect of the system is being ignored. Most applications appear to be mere interpretations of rigid functions. They utilize modern technologies which are

not hard to find that the amount of research into navigation

ADS

users to understand their systems. More importantly, it is also

T

heavy to implement and require a certain amount of training of

systems for healthy people is negligible. Most navigation systems for healthy people are developed for the purpose of emergency rather than enhancing the experience of daily navigation.

51

Auditory Environment Guideline

52

Spatial Audio as an Intuitive Architectural Interface

T

ADS

3 Auditory Environment Guideline

53

Auditory Environment Guideline Fig. 3-1 Summary of cues for human sound localization

Auditory environment setup vary substantially in different acoustic research and applications with different purposes.

For example, human heads are strictly fixed in some of the research about human performance in sound localization to ensure accurate outcomes. As for digital fabrication and spatial perception assistance, the goal is to let users localize the sound source to their best capabilities to ensure a better experience. Consistency and control variates, however, are secondary concerns. In such circumstances, users should be given enough freedom, such as no limitation in movement and customized sound source. However, it is also necessary to scientifically translate the sound localization cues, as summarized in figure 3-1, to a virtual auditory environment setup guideline, to ensure the effectiveness of sound localization tasks as a human-computer interface.

In this chapter, the guideline of auditory environment setup for spatial sound as an intuitive human-computer interface for spatial guidance purposes will be elaborated based on both theory and former research experience.

3.1

Coordinate System

In this thesis as well as former research in applications of spatial audio as a guidance interface, a single-pole spherical coordinate system indexed by azimuth ( ) and elevation ( ) is adopted. Taking the middle point of the user's two ears as the imaginary sphere centre, azimuth refers to the azimuthal direction with the user's front as 0 and user's direct left as 90째 and direct right as -90째. Taking an imaginary plane across the sphere centre and parallel to the ground plane as a base plane, elevation refers to the angle between the line across the centre and the measured point and the base plane, with 54

direct above as 90째 and direct below as -90째. Further details about the coordinate system choice are discussed below.

Spatial Audio as an Intuitive Architectural Interface

A healthy human utilizes both egocentric and allocentric perceptions during navigation. Specifically, allocentric perception is mostly used for capturing the surrounding visual environment, including landmarks, so as to build an enduring representation of the immediate surroundings. Egocentric perception, however, plays a fundamental rule for localizing once the visual cues are absent or obscure (Wang and Spelke, 2000). Similar to what Gallay et al. (2013) suggests, visually impaired individuals naturally adopts egocentric perspectives rather than allocentric perspective. With the needs of placing the virtual sound source on a fixed distance to the user, it should be placed on the surface of a virtual sphere. Due to the reasons above, a spherical coordinate system rather than a cartesian coordinate system is adopted.

Spherical coordinate systems are suitable for quantitative description as well as visualization for egocentric perception.

T

Spherical coordinate systems contain a wide range of specific

ADS

definitions suitable for different situation of use. The most commonly used spherical coordinate system in physics (ISO convention) is defined with

,

and r, shown in figure 3.1-1.

The convention in geography is the geographic coordinate system, which defines position relative to the earth by latitude, longitude and elevation. Although both systems use similar

Fig. 3.1-1 spherical coordinate system for

ways of defining a point in three-dimensional space, and

physics (ISO convention)

actually

https://en.wikipedia.org/wiki/

in ISO spherical coordinate system is nearly the

same as what is defined as longitude in the geographic

Image source: Andeggs Spherical_coordinate_system

coordinate system, the other indexes have different standards.

Therefore, given spherical coordinates (a, b, c) can technically cause confusion, if the specific kind of spherical coordinate system is not stated.

In spatial audio research, different spherical coordinate systems are adopted, or constructed by researchers to fit their specific needs. For example, for the specific study of interaural differences, Reijiniers et al. (2014) and Mattes et al.

(2012) used an interaural polar coordinate system, which is similar to a "flipped" version of the

and

in ISO spherical

coordinate system. Although relatively hard to comprehend intuitively, this "modified" coordinate system is more suitable

Fig. 3.1-2 geographical coordinate system. Image source: http:// wildernessnavigation.blogspot. com/2011/03/geographiccoordinate-symbol.html

for interaural-only studies since it shows minimal numerical difference in errors caused by the inability to localize the elevation of the sound.

As suggested in one of the earliest article discussing the issue Methods for spherical data analysis and visualization

55

"coordinate system choice for sound localization analysis"

Auditory Environment Guideline

by Leong and Carlile (1997), the most popular spherical coordinate systems used in spatial auditory research are single-pole coordinate system and double-pole coordinate system (Knudsen, 1982). However, in all of these spherical coordinate systems, the direction of the point is marked as azimuth and elevation. In past auditory research, the absence of defining the specific spherical coordinate system has also led to confusion. As shown in figure 3.1-3, a point in the same three-dimensional space can be marked with different numbers in different coordinate systems. Similarly, a point with the same azimuth and elevation numbers may refer to different points in the actual three-dimensional space.

double-pole coordinate system

Fig. 3.1-3 Same point represented in different spherical coordinate systems for sound localization: double-pole coordinate system

single-pole coordinate system

interaural polar coordinate system

Since our research, specifically for this thesis, doesn't include the data analysis of users' errors (the difference between the perceived sound location and the actual sound source

(left), single-pole coordinate

location), the choice of coordinate system is relatively less

system (middle) and Interaural

important. Still, we adopted single-pole coordinate system

polar coordinate system (right).

for two reasons. First, outdoor navigation doesn't include much elevation change compared to positional change on the ground plane, or, azimuthal information is relatively more important. The projection of single-pole coordinate system onto the ground plane is similar to a two-dimensional polar coordinate system and most suitable for navigation use.

Second, single-pole coordinate system is the easiest for human to understand (Carlile, 1996). It is also the most similar to the popular geographic coordinate system and ISO convention spherical coordinate system. In future research which may include user error data analysis, however, the choice of coordinate system should be decided based on the aspect of the data that is being analyzed.

56

Head Motion

Spatial Audio as an Intuitive Architectural Interface

3.2

Various experiments sound localization have shown that a subject's accuracy increases as the sound is located in front of the subject, which is 0 degree in both azimuth and elevation. In figure 3.1-4, the average error of the subject, as known as minimum audible angle (MAA), increases as the angle in azimuth from the median plane increases for given frequency sound sources.

Thurlow & Runge's research (1967) also shows that head movement is a spontaneous part of sound seeking during the localization process. Moreover, in Wallach's experiment

Fig. 3.2-1 Minimum Audible Angle (as depicted by the inner values) changes depending on both the

(1939, 1940) about sound localization with head movements,

different frequencies used (left

it is proved that moving the sound source together with the

and the orientation around

and right halves of the diagram)

subject's head lead to certain confusions.

the user.

In the literature review article Sound Localization by human

Roberts, Jonathan. (2003). Pie

listeners (1991), Middlebrooks did not include head movement

However, it still serves as a positive factor in localization

of the International Conference on Information Visualisation.

ADS

provides only a moderate increase in localization accuracy".

chart sonification. Proceedings

T

as a critical sound localization cue because "head motion

Image source: Franklin, K.M. &

efficiency. The research by McAnally and Martin (2014) suggested based on experiments that spatial information for 3D audio display is likely to be improved if users are able to move their heads during stimuli presentation. Further, it increasingly helps to detect elevation and solve front-back confusion as the freedom of head movements in azimuth increases. Perrett and Noble (1997) also suggested the positive influence of head rotations in localizing elevation of the sound source.

Free head movement helps for sound localization especially in our case as non-individualized HRTFs are applied in the sound render. As McAnally and Martin's study suggests, free head movements above certain range (i.e., over 32 degrees in azimuth) can compensate part of the paucity of using nonindividualized HRTFs. On top of this, Steadman et al. (2019) compared the efficiency of training for non-individualized HRTFs computer-generated audio between merely providing a positional confirmation and the utilization of head-tracking (free head movement). The result suggested that interactive head movement significantly improves sound localization accuracy.

Based on the reasons given above, it is clear that freedom in accuracy. Therefore for the auditory environment setup,

57

head movement certainly helps in human sound localization

Auditory Environment Guideline

subjects have the full freedom to move their heads instead of facing a certain direction constantly as most sound localization tests did. On top of the theoretic reasons, we believe that freedom to move head around to localize sound source increases user's interaction with the system with realtime response, which is an more entertaining process rather than a serious test. Therefore, we tend to allow users' free head movements for our researches and application developments.

58

Audio Source

Spatial Audio as an Intuitive Architectural Interface

3.3

Currently, one of the few viable and accessible methods of 'true' virtual spatial audio consists of the use of over-the-ear headphones as they are capable of accurately reproducing azimuth, elevation, and distance, and offer the greatest amount of control over the auditory experience of the listener.

However, the application of over-the-ear headphones can be considered intrusive by the user, especially those who are visually impaired. The muffling of external sound cues, and lack of mixing between real and virtual audio represents a significant hurdle in the perceptual localization of the surrounding space (Shilling, 2001).

Additionally, the pairing of headphones and user eardrums varies heavily between listeners due to an abundance of distinct HRTF-unique ear shapes, in conjunction with the simple positional displacement of the headphones on the

optimizations and/or equalization algorithms that minimize frequency artifacts. Indeed, an integral aspect of headphone listening is whether stimuli are heard as being 'inside' the head or coming from an exterior source 'outside' of the head. "For this reason, dynamic binaural spatialization of the acoustics of the space is a key element for externalization, localization,

Fig. 3.3-1 Image source: Armstrong,

ADS

headphone-induced coloration needs to be reduced by design

T

user's head. For a robust virtual spatial audio experience,

Cal; Thresh, Lewis; Murphy, Damian; Kearney, Gavin. 2018.

"A Perceptual Evaluation of Individual and Non-Individual HRTFs: A Case Study of the SADIE II Database." Appl. Sci. 8, no. 11: 2029.

realism, and, immersion (Serafin et al., 2018)."

Evidently, the ideal in virtualized spatial audio is a free-field 360° speaker array that englobes the listener. According to the study of comparison of free-field and headphone based field setup provides significantly more accurate ITD and

Speaker Array Spatial Sound System. Image source: The Sound of the Mountain

59

sound localization tasks by Brechetto et al. (2018), a free-

Fig. 3.3-2

Auditory Environment Guideline

ILD cues for subjects than headphones. That being said, unlike IMAX Cinemas that employ Dolby Atmos spatialized sound, it is categorically impossible to implement a speaker array in an urban context. Albeit less precise, headphones represent a convenient middle ground for our testing due to their adequate spatialization capabilities. The question arises which headphones perform the best in spatial audio workflows, and if any lighter weight alternatives exist that meet the stringent needs of the visually impaired.

The least recommended audio source for spatial audio applications are in-ear monitors, also known as IEMs. They are potentially the most common audio source within an urban context, in the very least, significantly more popular than a full-size headset. The reasons for their subpar spatialization are evident. They block the entirety of the user's ear canal, all while bypassing any kind of HRTF filtering. The result is a muffled situational awareness, audio with weak stereo separation and localization, and a "direct transmission to the brain" effect. IEMs represent a worst-case scenario for spatialized audio. An additional generalized criticism that affects most audio source types, specifically Bluetoothenabled versions, is the connection latency. Anything above 100 milliseconds is considered too intelligible in normal usage scenarios. Along with head tracking, this should be reduced as much as possible. Currently, the only Bluetooth-enabled headsets capable of low-latency audio playback are those with Qualcomm aptX(LL) functionality. These headsets generally achieve a latency of 30 to 40 milliseconds.

Fig. 3.3-3 Wikimedia Commons (CC BYSA 3.0) - Etymotic ER-4S IEM Photo by user Julo

60

Wikimedia Commons (CC BY-SA 4.0) - Commercial Bone Conduction Headset Photo by user Shirouri

Spatial Audio as an Intuitive Architectural Interface

Fig. 3.3-4

There are other types of audio sources that are more commonly implemented into wearable applications of virtual spatial audio. One of them is bone conduction headsets. Both lightweight and unobstructive, the visually impaired tend to favor such audio sources as the ear is not at all affected by the virtualized sound source, thus enabling a very consistent comprehension of the real environment. These headsets ADS

transmit the vibrations to the inner ear lymphatic fluid.

T

function by vibrating a specific point on the skull that then Due to the ability to remain situationally aware, most bone conduction systems are rated for industrial use applications like construction, general high noise environments, or even deep diving. They are therefore the usual choice for audio feedback in AR glasses. Yet, these headsets are significantly worse performing in spatial audio applications than a normal pair of headphones. There are in fact two main issues. The first is the lack of HRTF filtering. Just like IEMs (in-ear monitors), as the ear and even the ear canal are bypassed, the audio feels like it is being played straight to the brain, with no influential equalization by external factors. The second is the lack of stereoscopic recognition in certain edge case scenarios, specifically when a virtual sound source is placed directly to the left or to the right of the head, facing the ears. In this specific scenario, the nearly homogenous vibration of the skull leads to a mostly uniform interpretation of the virtual audio source, reducing navigational capacities.

We therefore made the conscience decision to closely examine the utilization of headphones, starting from the question on headphone types. We consider three main types of headphones (assuming that all compared headphones are over-the-ear): Closed-back, Open-back and the more contemporary floating speaker type headphones. The first two

61

have existed for decades and are very well documented, while

Auditory Environment Guideline

the third has only been commercially implemented in Valve's Index VR headset to our knowledge. The latter is specifically designed for spatial audio in VR scenarios. Due to the floating over the ear design, these 'headphones' take advantage of the unique ear shapes of the user to deliver improved HRTF filtered audio. Unfortunately, the accessibility of such a system leaves much to be desired. We therefore made the decision to settle on more common audio sources. As for the choice between closed-back and open-back headphones, we [1] See full text: oculus Audio SDK Version 1.18 https://

referred to Oculus' audio SDK[1] recommendation for open-

securecdn.oculus.com/sr/

back headphones as the audio leakage due to the inherent

audiosdk-1.18/

design of the driver enclosure allows for improved mixing of real and virtual audio cues with significantly less muffling.

Fig. 3.3-5 Grado Labs - Grado GW100 Wireless Bluetooth Open-back Headphones (our selected audio source)

A more recent implementation making its way into spatial audio applications is the adaptation of active noise-cancelling headphones to pass through outside sounds instead of blocking them. Employing such a method can eliminate the inherent issues of a common headphone in terms of the combination of real-world and virtual audio cues, but can be

[2] J. Rämo and V. Välimäki, "Digital augmented reality audio headset", Journal of Electrical and Computer Engineering, vol. 2012, Article ID 457374, 13 pages, 2012

difficult to effectively achieve, requiring a latency of a mere 1.4 milliseconds, including processing and digital equalization[2].

As of writing, Apple at WWDC 2020 has announced their own IMU-enhanced spatial audio that is comparable to our research. That being said, Apple's implementation lacks the

62

capability of absolute position head tracking, a fundamental prerequisite for true spatial audio based navigation. On top of

are the Airpods Pro, a wireless in-ear monitor that lacks HRTF filtering and has noticeable Bluetooth latency at a measured 162 milliseconds[1], though the earphone itself may be locally

[1] https://www.rtings.com/

processing the rotational transformation to the audio; this is

airpods-pro-truly-wireless

headphones/reviews/apple/

unknown in the current stage. Regrettably, we have not been able to test any sort of active audio passthrough headset for this thesis, but it would be a compelling implementation in future applications of our system.

Fig. 3.3-6 A rough categorization of audio source types based on their portability and spatial audio capabilities. Please note that

Spatial Audio as an Intuitive Architectural Interface

this, their chosen hardware for their spatial audio application

these points represent the bestcase scenario of each depicted technology.

T ADS 63

Auditory Environment Guideline

3.4 3.4.1

Computer-generated Spatial Audio Spatializer

In sound localization performance research, it is common to use speaker arrays to provide sound from various directions.

Theoretically, if one understands all potential cues for sound localization, one should be able to simulate the sound stimulus digitally from a computer instead of using physical speakers. Such simulation algorithms, also known as a sound spatialization, gives an output of spatialized sound to headphones depending on the sound source, the location of the sound source, user's HRTF, environment reverberation, etc, and provides an interactive immersive environment of real-time sound rendering. Savioja et al. (1999) concluded the input into digital simulation of spatial sound as three major parts: source modeling, room modeling, and listener Fig. 3.4-1

modeling.

A diagram of the process and componnts of implementing virtual acoustic environments Image source: L. Savioja et al., “Creating interactive virtual acoustic environments,�

Journal of the Audio Engineering Society, vol. 47, no.

9, 1999, pp. 675–705.

However, computer simulated spatializer today can not provide fully accurate and authentic spatial sound like physical speaker arrays due to some issues, like the limitations of headphones as well as difficulties in getting each individual's unique HRTF data. Therefore, it is hardly used in sound localization research requiring precision, but is widely used in applications like the virtual reality industry since it helps to create an immersive auditory environment as well as visual. It is also popular among other applications related to digital audio. It is also considered less important to ensure accuracy and authenticity 64

when using spatial audio simulation for interactive multimodal systems, compare to having a perceptually plausible

Current public available spatializer plugins include free choices

[1] Steam Audio: https://

like Steam Audio by Valve[1], Oculus Spatializer by Facebook[2],

audio/

Resonance Audio by Google and paid service like DearVR

valvesoftware.github.io/steam-

[3]

by Dear Reality[4] and Spatializer by Blue Lab[5].

A sound spatializer can be evaluated in two major aspects: sound propagation rendering and binaural rendering. Similar to ray tracing in computer graphics, sound propagation rendering includes environment settings such as reflection, reverberation and occlusion before reaching the user's ears. Binaural rendering, however, differs according to the

[2]Oculus Spatializer: https:// developer.oculus.com/ downloads/package/oculusspatializer-unity/ [3] Resonance Audio: https:// resonance-audio.github.io/ resonance-audio/ [4] DearVR: https://www. dearvr.com/products/dearvr-

user's head dimensions and ear shapes, or HRTFs. Since

unity

none of the spatializers mentioned above includes custom

[5] Spatializer by Blue Lab:

HRTF input, the main difference between these plugins is

https://bluelab-plugs.com/ plugins/spatializer-binaural-

about sound propagation performance. Chris Lane (2018)

spatialization-plugin/

compared the performance of Steam Audio, Oculus Spatializer and Resonance Audio in detail, pointing out each plugin's

https://designingsound. org/2018/03/29/lets-test-3daudio-spatialization-plugins/

ADS

that "the core sound quality is very good across all three".

[6] See full article at:

T

limitations in sound propagation and reached a conclusion

Spatial Audio as an Intuitive Architectural Interface

simulation algorithm (Serafin et al., 2018).

In this thesis as well as former research of PAFF and the interactive shooting game, spatial sound is used as a guidance information rather than to create an immersive experience. In such situations, sound propagation rendering is less important that binaural rendering. In other words, environment modeling including reverberation, attenuation, occlusion, etc., plays a minimal role. On the contrary, interaural cues are crucial for users to effectively comprehend the guidance information. As stated above, since all plugins adopted a non-individualized HRTF which doesn't correspond to any individual's exact HRTF data, the difference in quality between each spatializer is negligible.

Interaural cues rather than monoaural cues are more important in the application proposed in this thesis. However in previous research (PAFF and the interactive shooting game), and potential future research about spatial sound as a guidance interface, monoaural cues can be equally important.

For sound spatializer, it is most challenging to present elevational differences. In non-spatialized audio, although the left and right of headphones can present azimuthal change of the sound, there is no change in audio with elevational change.

As discussed in 2.1.1, elevational difference in human sound localization is recognized based on monoaural cues, or pinna Therefore, there is technically no algorithm that can simulate

65

cues, which is decided by the pinna shape of each individual.

Auditory Environment Guideline

the actual elevational change in spatial sound without the users' HRTF data. Spatializers mentioned above use other ways to simulate elevational changes, such as the level change of certain frequency band, or using the reverberation and reflection of the environment.

3.4.2

Frequency band

As stated in 2.1, sound sources with different frequencies and durations substantially differ the cue of interaural differences. A large number of researchers have suggested that subjects perform better with broadband stimuli in sound localization tests. As early researchers (Oldfield and Parker, 1984, Wightman and Kistler, 1989, and Makous and Middlebrooks, 1990) suggested, sound localization user tests using broadband spectra stimuli provides users with means to resolve the spatial ambiguities compared to using narrow frequency bands. According to Harris and Sergeant (1971), the minimum audible movement angle (MAMA) is also smaller for broadband stimuli than tonal stimuli. King and Oldfield (1997) suggested broadband signals covering frequencies from 0 to over 13 kHz are required in order for listeners to accurately localize signals actually presented from a range of spatial locations.

The frequency of sound source, for example, affects the ILD (interaural level difference) due to sound wave diffraction. With high pitch sound source (especially from the user's sideways), the blockage of head would be significantly heavier than that of a lower frequency sound, whose wavelength is close to the dimension of the physical head. As a result, high pitch sounds of above 1800Hz from a listener's side can easily cause more [1] Sound shadow, an area through which sound waves fail to propagate, due to topographical obstructions or disruption of the waves.

than 10dB level difference between the two ears, leaving a sound shadow[1] on the far side, while low-frequency sound results in negligible level difference. For example, a sound source of 200Hz performs nearly 0dB difference ILD even from 90 degrees on the side.

Using broadband stimuli is especially crucial to sound localization tasks involving elevation judgement. Human sound localization relies on spectral cues more in elevation than azimuth (Middlebrooks, 1990). Although human azimuthal sound localization can be dependent on ITD without the influence of frequency band, elevational judgement relies

66

on spectral cues.

It seems that using broadband noise as the sound source is

Spatial Audio as an Intuitive Architectural Interface

the ideal choice for sound localization tasks. It is also true that most research adopted broadband noise, although a few subjects questioned it. In Rutherford's experiment (2001), users gave negative comments on broadband noise in the aspect of user experience. Similarly, in our former research project, the interactive shooting game, users gave negative feedbacks about using broadband noise and performed fewer errors with musical stimuli. These cases suggested that broadband noise, although theoretically provides relatively complete spectral cues which should improve human sound localization ability, may possibly have negative psychological impacts on users that prevents them from an optimum result, especially in actual applications rather than experiments.

In the navigation application in this thesis, we are using music pieces chosen by users as the sound source. Unlike broadband noise which presents all frequency bands at the same time,

T

a large number of music pieces include different frequency

ADS

bands with the flow of time. Theoretically, music pieces which cover enough frequency bands with enough duration also provide enough spectral cues for sound localization. On top of this, the navigation application relies majorly on users' azimuthal sound localization, which is less influenced by the coverage of frequency. Other aspects, like users' familiarity with the sound signal (Middlebrooks and Green, 1991), play more important roles.

Fig. 3.4-2 (up) Waveform of a musical sound (notes C, E, and G played simultaneously) Fig. 3.4-3 (down) Waveform of noise (a recording of scratching sounds) Image source: Burg et al., DIGITAL SOUND & MUSIC, 2014

67

Auditory Environment Guideline

3.4.3

Audio Duration

As mentioned in 3.2, free head movement should be allowed in using spatial sound as an intuitive guidance human-computer interface. Obviously, this is under the premise that the sound stimuli are constant playing.

However, in most sound localization tasks, short sound stimuli within one second are chosen over that are constantly playing for test purposes. Even in some of the applications using spatial sound as the display, a short beeping sound is adopted.

In guidance systems, using short sound stimuli largely limit the accuracy of human sound localization capability. In the experiments of Pollack and Rose (1967), it was proved that human errors in sound localization tasks significantly increases as the duration of the sound source is shortened.

On the contrary, increasing the length of the sound stimuli has a positive effect in better sound localization performances.

Based on the reasons given above, we decided to use continuous playing sound in both the auditory test setup and application setup, as it certainly assures subjects' better performance in sound localization. On top of the scientific accuracy-related reason, continuous sound (for example, a piece of specific music) has an entertaining and aspect over a short signal (for example, a beeping sound) providing users with a pleasant experience, which is also an important reason for our sound duration choice.

3.4.4

Motion of Sound Source

In using spatial sound as a guidance human-computer interface, the shift of sound source location must be included in order to guide users continuously. Although head movements in turn result in sound source movement, it is also important to keep the absolute movement of sound source (or, the movement of sound source relative to the ground) gradual and smooth instead of jumping.

Based on the interactive shooting game project, we learned that moving the sound smoothly at a certain speed to the next location guides the user better than shifting the sound source immediately. As the sound source moves, users intuitively follow the same motion direction as the sound source which helps them to localize continuously. Using jumping stimuli is 68

a more accurate solution in sound localization tests since it does not provide extra influence on the result by giving hint

Spatial Audio as an Intuitive Architectural Interface

to users, it takes significantly more time for users to follow up to the next sound source direction.

As suggested by Kudo et al. (2006), using dynamic localization cues, or swinging sound source, compensate for the use of non-individualized HRTFs in sound localization tasks.

However, as Grรถhn et al. (2002) suggest, moving sound source does not improve users' sound localization abilities.

Users in dynamic and static sound source tests are showing a similar amount of errors in sound localization.

To summarize, although dynamic sound sources do not help to improve the accuracy of sound localization tasks, it provides a smoother experience and guides users to follow the direction with less time consumed. In using spatial sound as a humancomputer interface for guidance information, it is helpful to adopt smooth sound source motion.

Virtual Environment Setup

T

3.4.5

ADS

In most sound localization tasks for research purposes, anechoic rooms are used to eliminate the influence of reflection, reverberation, etc. As for spatial sound used for other purposes which do not require absolute accuracy, a room setup, or an environment setup, is popular mainly for

Fig. 3.4-4 A typical sound localization

two reasons: to create immersive experiences and to enhance

experiment setup

the elevational change in spatial sound.

Knowing (photography)

Photo: Daniel Stier, Ways of

69

Auditory Environment Guideline

In the sound localization test for the interactive shooting project, a room with the same dimension as the actual test field was modelled for sound occlusion, reflection and reverberation calculations. It creates an auditory environment that corresponds to the visual input of the users since they are able to look around during the sound localization tasks. It is similar to virtual reality applications involving spatial sound where the spatial sound is mapped to match the visual input of the user, so as to ensure better experience.

As discussed in 3.4.1, spatializers and headphones today have limitations in creating differences in monoaural cues to display vertical changes of the sound source. Such changes are rendered through other methods including environment modeling. For example, in an environment designed with a reflective smooth stone floor and a open ceiling, sound on different vertical levels in such an environment will sound different due to the reflection difference. In short, setting up an environment for spatial sound rendering is helpful in providing more elevational cues in most situations.

In this thesis, anechoic room settings are adopted. It is due to that no matching with visual input is needed for users' experience, and elevational cues play minimal roles compared to azimuthal cues in an urban navigation case.

70

Summary — Auditory Environment Guideline

The following is a summarized guideline to develop digital auditory environments for spatial sound as an intuitive humancomputer interface. As discussed before, this guideline is mainly for the purpose of augmenting human sound localization ability instead of testing it. Therefore, some of the setup choices are determined based on users' experience rather than the theoretical optimal choice. This guideline is the foundation for further development into spatial audio

Spatial Audio as an Intuitive Architectural Interface

3.5

navigation systems elaborated in the following chapters.

Coordinate System: Double-pole coordinate systems are the best choice if data analysis for both azimuth and elevation are required. If data analysis is only for azimuth, single-pole coordinate systems may be used. If data analysis is only for elevation, interaural

easiest for most people to intuitively comprehend. Therefore,

ADS

coordinate systems, single-pole coordinate systems are the

T

polar coordinate systems are the best choice. Among the three

single-pole coordinate systems are the ideal choice if no data analysis is needed for sound localization tasks.

(In the navigation application of this thesis, a single-pole coordinate system is adopted.)

Head Motions: For the purpose of enhancing human sound localization ability, free head rotation should always be included except for specific testing scenarios requiring the measurement of sound localization capacity.

(In the navigation application of this thesis, free head rotation and movements are included.)

Audio Source: For the utmost accuracy in sound localization, a 360° speaker array represents the highest achievable accuracy, but is not accessible for wearable systems. If working with visually impaired users, bone conduction headsets that do not block the ear canal should be selected if a lightweight solution is needed, if not then active passthrough headsets can be used.

For normal users, a pair of high quality, preferably openback headphones or floating speaker type headsets can be used if available. Active passthrough headsets can also be bone-conduction headsets may be used. Wireless latency

71

included, and if a lightweight solution is absolutely required,

Auditory Environment Guideline

of Bluetooth-enabled sound sources should be thoroughly inspected. Support for Qualcomm aptX(LL) is ideal for low latencies. (in the navigation application of this thesis, we employ a wired open-back headphone for sound localization) Spatializer: Ideally, spatilizers with individualized HRTFs should be adopted, especially for comprehending elevational change of sound source. However, custom HRTF data is hard to acquire and current available spatializers seldom support custom HRTF input. For non-individualized HRTF spatializers, Oculus Spatializer and Resonance Audio are both good choices.

Steam Audio is inferior relatively in some aspects but all three are acceptable.

(In the navigation application of this thesis, Resonance Audio is chosen as the spatializer.)

Frequency band: Audio with broadband coverage should be chosen over that with narrow band coverage. Theoretically, broadband noise should enhance human sound localization ability the most and is adopted in most experiments. Music pieces with broad frequency band coverage is also a decent choice considering other aspects of user experience. High-pitch or low frequency only audio should not be adopted.

(In the navigation application of this thesis, music pieces chosen by users are set as the sound source.)

Audio Duration: Longer duration audio has been proved to ensure better sound localization performance than short audio signals.

It is suggested to use continuous audio such as music or looping noise rather than a beeping signal.

(In the navigation application of this thesis, music pieces are played continuously.)

Motion of Sound Source: Place shifting of the sound source should be done is a smooth and continuous manner for the purpose of guiding the user to follow the guidance. "Jumping" of the sound source placement should be avoided unless for specific test purposes. Sound sources should keep at a fixed distance from the user to

72

prevent distance confusion.

(In the navigation application of this thesis, music pieces are

Virtual Environment Setup: The anechoic room setting is the least confusing setup for users in sound localization tasks but lacks in creating immersive experiences. In situations with visual input, it is ideal to set the spatial audio environment with the same dimensions as the visual input. In situations where elevational judgement is especially important, it is beneficial to set the environment to be asymmetrical above and under the user,

Spatial Audio as an Intuitive Architectural Interface

played continuously.)

so as to compensate for the absence of individualized HRTF. (In the navigation application of this thesis, the environment is set as an anechoic room.)

T ADS 73

Navigation System Development

74

Spatial Audio as an Intuitive Architectural Interface

T

ADS

4 Navigation System Development

75

Navigation System Development

4.1

General Background

Mobile navigation provides a two-dimensional interpretation of the surrounding environment, as well as a visually optimized guidance interface. Descriptive, non-spatial audio guidance, represents an additional feature for users whose visual input is occupied or impaired. Two major issues arise in consideration of these current visual guidance systems.

Our proposed spatial audio urban navigation system aims to challenge the very foundation of previous navigation systems, as well as solve the evident problems related to these visual navigation systems.

Interfacing with the City First, the graphical navigation interface largely occupies the user's visual input, depriving the user of the opportunity to experience the physical urban environment. "Trapped" by mobile device screens, this visual "captivity" is a direct source for the unconditional disregard of the surrounding conditions, on top of the already dubious safety concerns of walking with our eyes glued to a device in a dense environment. The act of navigating has inevitably become a visually taxing task requiring active attention to be allocated to digital map-based navigation. The incentive to break away from this simplified 2D representation of the surrounding context is met by significant hurdles. From the comfort of knowing where you are at any Fig. 4.1-1 A photo of pedestrians checking smartphones, photo taken within the city of Tokyo.

Image source: Nikkei Economics News

point in time, to the complete disconnect from the potentially tedious urban environment. With current technologies, it simply isn't a possibility to interface with the city as it was when visual cue navigation was the norm. This shouldn't be considered an escape from the social diaspora through the singular of function of navigation from one point of interest to the other. Rather, due to the drastically increased complexity of contemporary cities, with multiple layers, and obstructed visual navigation points, it has become a requirement to restrict interaction with the city to be able to reach your goal.

Comprehension of Guidance Information Secondly, the comprehension of a two-dimensional interpretation of a three-dimensional space in regards to being in said depicted environment leaves a lot to be desired.

76

Despite the simplified top-down, or rather, plan view of the

Spatial Audio as an Intuitive Architectural Interface

user's environment, inconsistencies between what is depicted and what is actually present are a cause for difficulties of comprehending one's position in space. Coupled with the fact that mobile devices suffer from significant positional drift. Several users choose to utilize a first-person view mode, providing a pseudo three-dimensional view of the virtual environment that geometrically aligns with the user's view of the physical environment. It can therefore be said that these users crave an augmented, intuitive, experience over the simplified one.

Furthermore, the graphical information in these common guidance systems is 'coded'. Streets are marked with names, distances labeled with intuitively abstract units of measurement, directions reduced to "north", "west", and so on.

The user always needs to decode this representation. Although those that are accustomed to these coded messages show

T

no real difficulty in navigation, the graphical map interfaces

ADS

occupy a significant cognitive load of users and interferes with the process of interacting with the physical city.

The idea of visual landmarks in urban design may be an adequate solution to both issues discussed above. As demonstrated in 1.2, urban designers consciously use visual landmarks to guide citizens since the beginnings of city planning. Such guidance is evaluated to be very different from navigation tools like using a compass because it shapes a pleasing experience for users and does not cause comprehension problems if culturally known. Similar ideas, like the arrangement of a succession of scenes, are also adopted in urban design practices. It is obvious that these navigation systems significantly lack in terms of precision and customization, and therefore cannot replace digital maps in

Fig. 4.1-2 Screenshots of navigation through top-down view and first person view.

Image source: sample images of "Map" application in App Store by Apple Inc.

navigation capacities. That being said, these urban design navigation ideas motivated us to further develop a precise navigation system using human intuition and to provide users with more opportunities to interface with the city.

On top of the basic technical issues like precision and latency, we majorly aim to resolve the two issues elaborated above, to free users from the screens as a means of enabling the interfacing with the urban environment, and to utilize human intuition to ease the problems in decoding guidance 77

information.

Navigation System Development

Target Users The main target users for our navigation system are healthy pedestrians in an urban environment.

Outdoor navigation for any moving individual or device along a certain path includes two aspects — mobility and orientation (M&O) (Psathas, 1976). Mobility, also known as micronavigation, refers to sensing the immediate environment, for example, avoiding obstacles. Orientation, or macronavigation, includes choosing a route, adjusting orientations, piloting along the correct path, etc.

For healthy users, macro-navigation is more important than micro-navigation since they have the ability to detect local obstacles (e.g. walls, vehicles...), but lack knowledge of the general orientation due to the limits of human self-tracking ability, as well as the ever-enveloping, large scale urban space.

Fig. 4.1-3 (up) Main target user: pedestrians

Therefore, the term M&O is more often used in navigation

Photo: Ryoji Iwata, taken at

for visually impaired individuals. For these visually impaired

Osaka Station

individuals who have a disadvantage in way-finding and

Fig. 4.1-4 (down) Secondary target user for

obstacle-avoidance, both macro and micro navigation are

challenge of precision: visually

a requirement.

impaired marathon runners.

A photo of athlete MICHISHITA Misato, image source: https:// www.parasapo.tokyo/en/ athletes/27/

[1] T11 / F11 athletes have a very low visual acuity and/or no light perception.Athletes with a T12/ F12 sport class have a higher visual acuity than athletes competing in the T11/F11 sport class and/or a visual field of less than 10 degrees diameter.

See more information in: https://www.paralympic.org/ athletics/classification

Our outdoor navigation system, based on spatial audio, is designed for healthy pedestrians in urban environments. Thus, the micro-navigation aspect is not included. However, we aim to challenge the system to be precise enough for guiding visually impaired individuals under specific circumstances.

For example, visually impaired marathon athletes. Currently in Paralympics marathon class T11 & T12[1], athletes require one or two guide-runners guiding them with an elbow lead or tether. The speed difference and the necessity to keep the runner in the centre of the track requires high precision of the tracking, as well as a very low latency within the entire navigation system.

By aiming beyond the scope of our system's intentions, for example, the above mentioned extreme case of visually impaired marathon athlete guidance, we aim to open up further routes of development by ensuring that both the hardware and software can exceed the expectations associated with its initial conception. To accomplish such a feat, significant technological hurdles need to be taken into consideration.

To this effect, the specifics of the hardware and software development are discussed in the following sections.

78

Spatial Audio as an Intuitive Architectural Interface

T

ADS

79

Navigation System Development

4.2

Position tracking — GNSS

A majority of our research required some sort of precise realtime position tracking within a certain space as a method of spatial data acquisition for structural safety computation systems, and various other secondary functions. Having gone through the application of static AR trackers, and basic computer vision, we found that, rather than tracking the structure itself, the use of human-biased positional tracking led to significantly more freedom of construction processes.

Due to this, the main positional tracking system we have adopted in recent years is the SteamVR mocap-like active IR tracking. The decent pricing of the system compared to cinema-grade mocap systems, as well as the excellent tracking performance and user-friendly programming, have enabled us to design, for example, high precision spatial audio guidance systems within a room-scale context. As previously shown, these previous audio guidance systems were severely limited in range as the SteamVR tracking can Fig. 4.2-1 (left) AR trackers used in digital fabrication process.

Image source: the University of Tokyo DFL See more project info: http://tads.org/tag/toca Fig. 4.2-2 (right) Absolute indoor tracking through HTC VIVE hardware, during user tests in Shooting Game project.

Photo: Alex Orsholits

only cover a fixed area of 10 metres by 10 metres on average, leading to projects that aim to extend the reach of the user through cannons designed to shoot construction material.

To achieve urban-scale positional tracking, two issues arise: the portability of such a system, and the achievable precision. Valve's SteamVR Tracking is evidently not very portable due to its requirement of at least one base station (lightbox), and more problematically, due to its significant processing overhead, necessitating a VR-ready laptop or desktop computer.

80

Spatial Audio as an Intuitive Architectural Interface

One of the most common large scale positioning technologies is GNSS, more generally known as GPS even though the latter only describes the USA designed system, and does not encapsulate the entirety of the Global Navigation Satellite Systems available. GLONASS, BeiDou, Galileo, and of course GPS represent for the most part the positioning capabilities of a receiver on the surface of the planet earth. From this point on we will refer to GNSS positioning as an absolute position system, even if theoretically, all coordinates are relative to the planet and are therefore not absolute at the solar system scale. GNSS positional tracking is a very accessible technology in the 21st century. Nearly all smartphones and vehicles now have some sort of implementation of the service. It has effectively become the standard for outdoor navigation. Unfortunately, these navigation systems are not known for their accuracy, especially at the human scale.

GLONASS is rated for 2.8 to 7.38 metres (Miller, 2000),

[1] See The State Council

BeiDou 2.6 to 3.6 metres[1], GPS 0.3 to 5m[2], and Galileo

the People’s Republic

Information Office of of China, 2016: http://

T

1m . These are of course best-case scenarios, meaning

en.beidou.gov.cn/SYSTEMS/

under open sky in an empty area. Real-world tracking can

WhitePaper/201806/

ADS

[3]

be significantly lower precision based on the urban context.

Tall skyscrapers for example can place you several kilometres away for your actual position in certain situations. That being

P020180608507822432019. pdf [2] See U.S. government information about the Global

said, these accuracy ratings are for the consumer use of

Positioning System:

these systems. Various systems have higher performance

gps/performance/accuracy/

capabilities reserved for government and military use. For example BeiDou can achieve 10cm accuracy on its encrypted band, while Galileo can achieve 1cm accuracy. The military precision of GPS is not made public and therefore cannot

https://www.gps.gov/systems/

[3] See European Space Agency (ESA), 2011: https://gssc.esa.int/navipedia/ index.php/Galileo_Public_ Regulated_Service_(PRS)

be quoted.

As a majority of our digital fabrication research requires high precision position tracking to ensure coherent spatial data acquisition, the civilian accuracy of GNSS is simply not viable.

Fortunately, there are accessible methods to correct a GNSS position in real-time to achieve under 10cm position accuracy.

Although not necessarily a requirement for simple navigation applications, we decided nonetheless to implement these correction systems for the highest possible precision. Having access to both the QZSS (Quasi-Zenith Satellite System) from the Japan Aerospace Exploration Agency JAXA capable of under 10cm accuracy in CLAS mode, as well as RTK GNSS (Real Time Kinetic), we decided to employ both systems results but function on fundamentally different principles.

81

depending on situation. These two systems achieve similar

Navigation System Development

QZSS is a group of four satellites in, as the name states, a quasi-zenith orbit which ensures at least one satellite is almost directly overhead to counteract the influence of tall buildings in high-density urban contexts. It is a time transfer system that originally served as an augmentation service of USdeveloped GPS satellites. Through more precise timing and orbit estimation, the correction computation called MADOCA[1] [1] Multi-GNSS Advanced Demonstration tool for Orbit

can provide a corrected position of 6cm to 9cm accuracy

and Clock Analysis. See more

depending on satellite system. The advantage of this system

information at:

is the elimination of base stations commonly used in RTK

https://ssl.tksc.jaxa.jp/ madoca/public/public_index_ en.html

GNSS correction systems, but requires a more expensive dual-band GNSS receiver to achieve the corrections. RTK GNSS on the other hand functions by installing a series of base station GNSS receivers at known fixed coordinates and computing the deviation between the known coordinates and the measured coordinates at the site. The corrections stemming from this system do not require any specialized services and are therefore very easy to implement as an individual. The main downside is that for the correction data to be useful, the GNSS receiver of our system, also defined as the "rover" of the RTK system needs to be within a 10km

[2] More information about CQ Publishing: https://www.

radius of the base station on average to ensure that both GNSS receivers are influenced by the same atmospheric

cqpub.co.jp

factors that cause positional deviation. For convenience, a

[3] A list of available open-

majority of our tests were executed using a local open-source

source RTK basestations

base station hosted by CQ Publishing[2] in Bunkyo-ku, Tokyo

can be found here: http://rtk. silentsystem.jp/

among multiple available open-source RTK base stations[3].

Our initial tests compared the precision of our dual-band GNSS receiver development board from the Swiss firm U-Blox

[4] The specific setup script for the tests can be found in Appendix 8.

(C099-F9P) with that of a regular single-band smartphone (in this case, an iPhone 7)[4]. It should be noted that the iPhone 7 was running in Assisted-GPS mode which enables more precise positioning through the fusion of WiFi triangulation, cell tower triangulation, and of course GPS. Even with all this, the precision of the assisted single-band GNSS demonstrates significant deviations in low building height urban contexts compared to the tested routes, with highrise urban contexts causing a total loss of tracking. The dual-band RTK-corrected GNSS receiver in contrast achieves near-perfect positional tracking in low height situations, and adequate tracking in high building height contexts. In our tests, we managed to find a blind zone near Tokyo Station that caused significant

82

deviation of the dual-band receiver, but was quickly recovered once out of the zone.

Spatial Audio as an Intuitive Architectural Interface

Fig. 4.2-3 T

U-Blox C099-F9P Application Board (left) and U-Blox ANNPhoto: Yiyuan Qian

ADS

MB GNSS Antenna (right)

Fig. 4.2-4 Outdoor tracking precision tests with dual-band RTKcorrected GNSS reveiver (left) and a regular single-band receiver smartphone (right) in central Tokyo.

Yiyuan Qian

83

Model: Alex Orsholits, photo:

Navigation System Development

Fig. 4.2-5 Outdoor tracking precision tests with a regular single-band receiver (iPhone 7 in-built) Tested at 1Hz (1 dot/sec) in a low-rise residential zone, walking in ther center of the smaller streets and the right sidewalk on the main street.

Background image: Google Earth Satellite Image. Imagery Date: 3/14/2019 Maximum deviation (compared to dual-band receiver with RTK correctioin): 7.8m

Fig. 4.2-6 Outdoor tracking precision tests with a dual-band receiver.

Other configurations are the same as Fig. 4.2-5 Maximum deviation (compared to dual-band receiver with RTK correctioin): 2.6m

Fig. 4.2-7 Outdoor tracking precision tests with a dual-band receiver with RTK correctioin.

Other configurations are the same as Fig. 4.2-5

84

Outdoor tracking precision tests with a regular single-band receiver (iPhone 7 in-built) Tested at 1Hz (1 dot/sec) in a high-rise commercial zone, walking in ther center of the street. Background image: Google Earth Satellite Image.

Imagery Date: 3/14/2019 Maximum deviation (compared to dual-band receiver with RTK correctioin): 9.7m

Spatial Audio as an Intuitive Architectural Interface

Fig. 4.2-8

Fig. 4.2-9 Outdoor tracking precision tests with a dual-band receiver.

Other configurations are the

T

same as Fig. 4.2-8

ADS

Maximum deviation (compared to dual-band receiver with RTK correctioin): 1.9m

Fig. 4.2-10 Outdoor tracking precision tests with a dual-band receiver with RTK correctioin.

Other configurations are the same as Fig. 4.2-8

85

Navigation System Development

86

Hardware and Network Mapping

Our initial setup intentions consisted of two methods as shown in Figure 4.3-1. The first method employed roughly the same components as our tested system in terms of functionality. The two main differences are the use of a Pycom GPy [1] ESP32-based microcontroller mounted directly to the C099-F9P GNSS development board for

[1] Pycom GPy, a micropythonprogrammable triple bearer.

LTE-CAT M1 connectivity to receive correction data for the RTK capabilities of the receiver, and the use of a Raspberry Pi 4 (4GB RAM)[2] for spatial audio processing. This setup had significant limitations, for instance the lack of internet

Spatial Audio as an Intuitive Architectural Interface

4.3

[2] Raspberry Pi, a series of education-oriented arm-based single-board computer

connection on the Raspberry Pi for receiving active mapping data, and little to no support for spatial audio systems within an arm-based Linux environment. The second method aimed to outright remove the need for local spatial audio processing by developing a lightweight system solely controlled by the

downlink and 375kbps uplink) incorporated on the Pycom GPy would have been theoretically acceptable for receiving

Fig. 4,3-1 Initial setup intentions (two

ADS

that the bandwidth of the LTE-CAT M1 modem (300kbps

T

Pycom GPy ESP32-based microcontroller. We estimated

options) for the hardware and network mapping

87

Navigation System Development

NTRIP correction data and a compressed audio stream, and sending position and rotation data. When tested however, the total available bandwidth, as well as the latency (theoretical 10-15 milliseconds), were not always achieving the theoretical limit, which was the bare minimum required to run such a system. On top of this, The processing power of the on-board ESP32 would have been maxed out with the first core doing all networking, position and rotation acquisition, and the second core decompressing the audio stream. Furthermore, our skills in programming are not up to the task for developing a highly optimized, low CPU cycle solution. We therefore combined these two methods in a sense, by enabling LTE networking, and by introducing easy to program spatial audio processing via the adoption of an Android-based Smartphone.

Our proposed network setup is summarized as shown in figure 4.3-2. It consists of an ESP32 Xtensa dual-core 32bit microcontroller operating at a configured 160MHz (out of 240MHz). The ESP32 is both very fast (around 50 times faster per core than a classic Arduino according to Andreas Spiess 2020) and has the added benefit of supporting the IEEE 802.11 b/g/n 2.4GHz WiFi standard for data transmission, as well as low power Bluetooth (BLE). The board is powered by an LiHV (high amperage LiPO) 300mah battery, with an estimated battery life of around 3 hours. The ESP32 transmits to the server the data provided by the Bosch BNO055 9 axis managed absolute orientation IMU which calculates an absolute orientation at a rate of 100Hz through its own integrated microcontroller. This is connected via the ESP32's I2C bus. For the GNSS, wifi transmission is completed by the onboard 5GHz (IEEE 802.11n) Wifi controller ODIN-W260.

Two low profile WiFi antennas were added to replace the single large antenna included with the dev board for portability reasons. The ODIN-W260 acts as a virtual serial port and transmits to the server pre-configured NMEA messages received over UART at 460'800 baud rate from the ZED-F9P dual-band GNSS module. We initially reserved Bluetooth for the Grado GW-100 open-back Bluetooth headphones [1] Grado GW100 Wireless

to reduce the total amount of cables to just one, but after

Headphones Review by

extensive testing, we found that the Bluetooth audio latency

RTINGS, 2018. See full information at: https://www.rtings.com/ headphones/reviews/grado/ 88

gw100-wireless

to be far too high for coherent spatial audio. Indeed, the measured latency is 224ms[1]. We therefore settled for cabled use of the headphones.

A diagram of hardware and network mapping of the navigation system based on spatial audio

Spatial Audio as an Intuitive Architectural Interface

Fig. 4.3-2

T ADS 89

Navigation System Development

As architects, the preferred solution for network routing between various software is Grasshopper. Included with Rhino3D, Grasshopper is a node-based geometric editor that, with 3rd party plugins, is able to send and receive various kinds of data over all the common network protocols. Due to its integration with one of the principal CAD design softwares, our experience with other platforms is considerably lacking.

Unfortunately, the restrictive nature of Grasshopper induces several issues for wearable projects and data processing in general. We found, through several tests, the performance of Grasshopper to be atrociously bad. Indeed, networking is only able to function on a single processing thread and it is not even able to saturate this thread. In fact, the UDP performance tested on a low-performing Intel Core i5 10210y and on a workstation-grade AMD Ryzen 3950x yielded similar performance. Furthermore, we found that simple sensor data similar to our IMU, with updates at 100Hz, was too much for Grasshopper's network stack, and therefore several sensor updates were dropped. On top fo this, the requirement to employ a powerful x86 laptop or desktop capabale of CAD work simply isn't a viable solution in our case. Of course, another option would be to entirely eliminate the use of Grasshopper by finding alternatives that are able to emulate the (theoretical) capabilities of the aforementioned. This is indeed the case, all map-based routing and audio processing was moved to Unity, more on that later. For networking, we tended to hardcode all network routes. This translates programming the active ports on each device and sensor manually. This proved to be too difficult to manage during the evolution of the thesis.

Therefore, rather than hardcode all networking between IMU, [1] Node-Red is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways.

GNSS, and phone, we decided to implement Node-Red[1].

Developed by IBM's Emerging Technology department, this open-source development tool allows for rapid network prototyping through a visual node-based programming system. The use of this software enabled us to test out a variety of system configurations, with several devices acting as servers. Initially, we intended to run all spatial audio code on a Raspberry Pi 4 (4GB RAM model) but the weight and lack of integrated battery led to portability issues, as well as a general incompatibility with many spatial audio engines that are destined to run as a Unity plugin or on x86 devices. It was

90

decided to move to running all data processing on an android smartphone instead. Thankfully, due to Node-Red's approach

JSON[1] file, all our previous network routing was ported to the android device with a simple copy and paste. Another significant improvement over the hardcoding of network routes is the ability to wire in new devices in a matter of seconds. This

[1] JSON (JavaScript Object Notation) is a lightweight datainterchange format. It is easy for humans to read & write, and also easy for machines to parse and generate.

allowed us to develop the spatial audio guidance application on any computer, with real data being routed from the IMU and GNSS for testing purposes. There are still some limitations compared to the potential workflow of Grasshopper, the main one being the lackluster support for combined asynchronous

Spatial Audio as an Intuitive Architectural Interface

of saving flows and all programming as a single, immense,

data. Data flows running at different speeds simply cannot be combined together like Grasshopper's 'Merge' node, they need to be processed separately and sent to different ports or websockets. This is apparent in figures 4.3-3 and 4.3-4.

Fig. 4.3-3 (up)

On a side note, installation of Node-Red can be considered

Dataflow for IMU sensor

quite delicate as it is essentially installed through the Linux

Fig. 4.3-4 (down)

terminal emulator app Termux as a sandboxed package with

Dataflow for GNSS receiver processing and UI mapping & NTRIP correction data routing

T

restricted access to the Android filesystem.

processing and UI mapping

ADS 91

Navigation System Development

The UI mapping of incoming and outgoing data enabled us to verify the functionality of the system while in use by test subjects by loading the webpage on any secondary smartphone connected to the server's hotspot. We were also able to visually calibrate the IMU when needed, as well as test the overall latency of the system which never exceeded 4 milliseconds.

Fig. 4.3-3 The webpage for verifying the functionality of the system

92

smartphone is, as of June 2018, the WiFi hotspot tethering assigns a randomized router IP within the 192.168.43.0/24 subnet rather than the usual functionality of a fixed base IP of e.g 192.168.43.1 due to certain edge case scenarios, and will furthermore be obfuscated in future releases by further randomizing the subnet itself. To somewhat counteract this issue for connecting Node-Red to the IMU, the ESP32 board was programmed with a fixed IP and to wait in the setup() single-run loop for a packet generated by the server.

Spatial Audio as an Intuitive Architectural Interface

One consequential setback to using an Android-based

This equates to the "Connect to IMU" button within the UI which sends a simple timestamp packet on port 3333 to the ESP32. A similar workaround could sadly not be applied to the GNSS receiver which runs exclusively on a preconfigured AT configuration command list rather than low-level programming.

T ADS 93

Navigation System Development

Finally, a basic NTRIP client for receiving correction data from web servers via simple HTTP was programmed in python and set to run on the Android smartphone alongside Node-Red.

94

Spatial Audio as an Intuitive Architectural Interface

T

ADS

95

Navigation System Development

4.4

Fig. 4.4-1 A rough flowchart diagram of the software system for the urban outdoor navigation system based on spatial audio

Software Development

As the whole navigation system has to run real-time on a portable device, the main challenge is to keep it simple and light, both in hardware and software. As mentioned in 4.3, we chose to use an Android smartphone to do all data processing. For the data processing software, we chose to build an Android application with Unity running on any Android-based smartphone. We chose Unity because it provides an efficient integration of maps, geometric algorithms, spatial sound simulation API and a visual interface for development purposes.

[1] [2] UDP refers to the Internet protocol suite User Datagram

As shown in figure 4.4-1, the input into the Unity-built appli-

Protocol. Both GNSS RTK

cation includes GNSS RTK data through UDP [1], IMU data

and IMU data are transmitted

(absolute rotation data) through UDP [2], a navigation route

wirelessly through UDP. See complete scripts (in C#) in Appendix 3 & 5.

as a curve from third-party software (for example, Google Maps) and the music pieces selected by the users saved locally on the smartphone. All four inputs together generate the real-time spatial audio output to the headphones.

Although the navigation system uses spatial sound as the only

[3] Maps SDK is a collection of tools for building Unity applications from real map

interface for guidance, it is ideal to include a visual interface showing the status of each input source for checking purpos-

data. V2.1.1 is used for our

es, especially during development. We use Unity with Maps

navigation system.

SDK for Unity from Mapbox [3] so that visual maps can be

Link: https://www.mapbox. com/unity 96

[4] See complete scripts (in C#) in Appendix 4.

integrated into the system, and GNSS data can be translated from raw latitude and longitude to local Unity coordinates, with simple commands shown below [4]:

Spatial Audio as an Intuitive Architectural Interface T ADS

A spatializer, as mentioned in 3.4.1, helps to render a normal stereo sound into spatial audio. We chose Resonance Audio

[1]Resonance Audio is a spatial

SDK for Unity from Google to simulate the spatial sound in a

audio software development

virtual anechoic room. Technically, the output of the rendered

from Google’s VR Audio

[1]

kit (SDK) based on technology

spatial audio is capable of presenting audio source from all

SDK, and it works in multiple

possible directions around the user in a three-dimensional

platform, including Unity, which

space so that the user has full freedom to rotate, including turning head or bending over, to confirm the direction of the

development environment or is used in our case.

Link: https://resonance-audio. github.io/resonance-audio/

sound source.

As the sound source is selected by the user from a local album, it is most possible that the user is familiar with the audio which helps for sound localization both in speed and accuracy. The audio source is kept at a fixed distance from the user to reduce unnecessary confusion. In other words, the level of the audio source, which is related to the distance from the receiver to processing the navigation route curve and the placement

97

the audio source, remains stable. The specific algorithm of

Navigation System Development

[1] See complete scripts (in C#) in Appendix 6.

of the audio source is discussed in the following section [1].

As shown in figure 4.4-2, a simple visual display of the user's current locational and rotational information together with the map and the sound source is also included in this system.

This graphical interface is only developed for checking purposes. Similar to most digital map applications, it provides a top-down map view centered with the user's current position. Simple functions are included, like the completed route length display and controllable view zoom with a slider bar.

The user is displayed as a three-dimensional triangle with a certain thickness instead of a two-dimensional circle (or, in most map applications, a one-dimensional dot) to present the three-dimensional rotation data in real-time. This provides developers, or users if necessary, an intuitive and easy way to verify the working status of the GNSS and IMU data

Fig. 4.4-2 Screenshots of the graphical interface (for checking purposes in development only) for the urban outdoor navigation system based on spatial audio.

receiving. Apart from the triangle, the navigation route and the audio source are displayed as well. On the right bottom corner, a small window showing the first-person view of the user also provides developers or users an easy approach to check on the system.

98

Spatial Audio as an Intuitive Architectural Interface

Fig. 4.4-3

T

Photos of developers checking

ADS

the graphical interface during initial tests of the navigation system.

99

Navigation System Development

4.5

Sound Source Placement Algorithm

Unlike visual information, spatial audio can only provide one direction information at a time, as shown in figure 4.5-1. In other words, unlike visual guidance which can demonstrate an arrow of first straight then right-turn guiding the future move, spatial audio is relatively limited in the amount of information it can present, merely providing the guidance for the current moment. Therefore, the placement of the virtual sound source according to time is essential for users' experience in a navigation system based on spatial audio, especially in an urban environment. Since most urban streets are linear and intersect each other in the form of crosses or T shapes, the guidance signal given by the spatial audio has to sharply change its direction at the turning points, unlike a continuous curving street, where guidance should change direction gradFig. 4.5-1 A rough diagram for comparison between graphical

ually or an open space where guidance is towards a certain direction constantly.

visual guidnace and spatial audio guidance - a continuous guidance for the near future vs. a directional information for the current moment

The simplest way of spatial sound guidance placement is to change the sound source immediately towards the direction where the user should turn to the right when the user approaches the intersection or turning point. However, this kind of guidance provides no turning information for users until they reach the very turning point. Another solution may be gradually changing the direction of the sound guidance as the user approaches the turning point so as to provide certain expectation before the actual guidance, which may in turn cause confusion about where exactly to turn if the amount is not well-controlled or the urban environment is complex. In our case, we adopted a simple but tolerant algorithm for spatial sound source placement to establish users' expectations before turning without confusing users about the correct place to turn.

As the rotation of the user changes, which can be caused by the user turning head or bending over, the position of the virtual sound source guidance stays in the same absolute place. In other words, the sound source moves relative to the user. However, as the position of the user updates, the position of the virtual sound source should update accordingly.

The placement of the sound source according to the position change of the user is essential to the project since it has to: first, apply to various situations with different turning angles

100

and different street width; second, apply to both visually impaired marathon runners and also healthy walkers in cities.

Resarch Title

In the case of visually impaired marathon runners as target users, the sound source should be able to keep the users in the center of the path by giving correction guidance depending on the amount that the user is off the track center; however in healthy pedestrians' case, the system should be tolerant enough to let the users interface with the city freely, for example, to walk on the edge of the street, to temporarily explore places off the track, or cross to the other side of the street at any time.

For example, in a situation of a straight street where the target is ahead of the users, if a visually impaired marathon runner reaches close to the sidewalk on the left side of a street, the sound guidance should tell the user to go to the right to be back to the center of the track. But for a healthy user walking on the left sidewalk for daily walking, the sound source should not give the user the confusing signal of going to the right.

T ADS

In order to solve the issues stated above, we applied a simple geometrically defined placement algorithm. The function

Fig. 4.5-2 A rough diagram for the tolerance difference between

input, as described at the beginning of 4.4, includes a

visually impaired marathon

preloaded single-directional curve representing the centre

pedestrians as target users

runners and healthy

of the street (curve t); a preloaded parameter of the average path width (w), a real-time updating position data of the user p(x, y) and a pre-defined parameter of margin m for especially of the sound source s1(x, y).

101

of marathon case. The output is a real-time updating position

Navigation System Development

In the case of visually impaired marathon running navigation, if a circle centered at s1 with a radius of r (r = 0.5w - m) intersects with curve t at two different points, the runner is "on track",

and the sound source will be placed at the intersection point with higher interpolation value among the two (or, "in front of" the user). If the circle intersects s at only one point, or the circle is tangent to curve t (depending on real situation of urban street shapes and given that r is restricted as r = 0.5w - m, it is not likely that the circle can be tangent to the curve at more than one point), the runner is at the edge of going "off track". The sound source will be placed at the tangent point in this situation, which is on either side of the runner to efficiently correct the running route. In the situation of no intersection point, the runner is "off track" and sound guidance will halt, replace with a warning signal sound and wait for manual help on site.

Fig. 4.5-3 (right) A diagram showing the geometric definition of the guidance sound source placement

Fig. 4.5-4 (down) A rough diagram of the navigation system in the case of marathon running guidance for visually impaired individuals in an urban context

102

Resarch Title

In the situation of navigation for healthy pedestrians, the sound guidance is similarly placed but a lot more tolerance should be included. There are two major differences between the algorithm of sound source placement for healthy individuals' walking navigation and visually impaired marathon runners' navigation.

First, the value of r is a lot bigger in navigation for healthy walkers. In our case, we chose the value r = 3w, which is three times the width of the street. The most direct influence of increasing the r value is that the system has much larger tolerance for the amount that the user is off the center of the path. First, the user won't be guided back to the center of the street while walking on one side of the street. Second, the user will not get lost stopping by the street side attractions.

For example, the user can wander into a store on one side of the street and on exiting the door, the sound source guidance continues to direct the user to continue the previous path.

T

However, in visually impaired marathon runners' version, if

ADS

the user wanders to one side of the street and gets lost in direction, the immediate drastic change in spatial sound direction will lead the user back to the center of the track (the sound source placed on the tangent point as shown in figure 4.5-5) and then show the running direction. Third, with larger r value, each turning guidance "signal" is not abrupt, but starts to gradually show up in advance. For example, a user on a 10m width street will sense the sound source to be 15° to the

Fig. 4.5-5 Diagrams showing extreme cases in which the user is accurately on track and on the edge of going "off track"

right 15m away from the actual turning point where the user should turn to the right. The angle of the sound source off the median plane will become bigger gradually as the user approach the turning point and reach nearly 90° in the end.

This gradual change of the sound source placement gives a mental expectation for users, which we believe is crucial to smooth user experiences.

Another difference between the algorithm of sound source placement in healthy and visually impaired users' cases is that the system only provides macro navigation, or orientation guidance, for healthy users. Micro navigation, or obstacle avoidance, will be efficiently accomplished by each individual naturally. Although it is possible to load sidewalk paths in the third-party map data and place the virtual sound source based on the sidewalk information so as to better suit the navigation for walking, it may be challenging, though, to get the data accurate enough to decide which side of the street the user for the route should also adapt to any change. For example,

103

is currently on, and provide further navigation. The algorithm

Navigation System Development

when the user crosses the street to continue walking along the other side. The status of streets of famous pedestrian Fig. 4.5-6 Diagrams showing the navigation system for healthy users with tolerance to provide expectations for turning and freedom to explore the city off track

areas, especially in Tokyo, also changes according to time to switch between pedestrian and driveways. Due to the issues above, we currently adopted the formerly discussed geometric algorithm for the project instead of taking sidewalk position tracking into account.

104

Spatial Audio as an Intuitive Architectural Interface

T

ADS

105

Navigation System Development

4.6

Wearable Design

In initial tests with the hardware and network system (as shown in figure 4.6-1, figure 4.6-2 and figure 4.6-3) the system functioned well but exposed the problem of difficulties in organizing the devices. In the initial tests, devices are held together simply with rubber bands and tapes. During the test, issues in mounting the devices together also appeared.

For example, the magnet attached to the GNSS antenna interferes the magnetometer in the IMU module, but both devices have to be mounted upward together with the user's head. Reasonable weight distribution should also be taken into consideration, as a number of devices weighing more than 100g have to be included: Fig. 4.6-1 An outdoor test of the GNSS and IMU data transmission without any wearable design and arrangement. Model: Alex Orsholits, Photo: Yiyuan Qian

Fig. 4.6-2 (left) The headphones with IMU module together with the microcontroller simply taped on for temporary use. Photo: Yiyuan Qian Fig. 4.6-3 (right) An outdoor test of the whole navigation system without any wearable design. Model: Alex 106

Orsholits, Photo: Yiyuan Qian

Smartphone (sony xperia xz premium) 195g, arm-mount; GNSS Antenna (ublox ANN-MB series) 173g, head-mount; Headphones (Grado GW100) 170g, head-mount; Portable battery (LDTEK WT-H330) 116g, arm-mount.

a system to neatly and easily mount all devices onto users, and at the same time having all connection ports and modular parts easily accessible for developers to check, modify and upgrade the system. In the final design of the wearable system, the user should wear an arm pouch and a pair of

Fig. 4.6-4 (below)

headphones, each containing necessary parts for navigation

A demonstration photo of a user

in order to accomplish local processing for the spatial sound output.

Fig. 4.6-5 (next page) equipped with the wearable design. Model: Alex Orsholits, Photo: Yiyuan Qian

Spatial Audio as an Intuitive Architectural Interface

In order to solve the problems described above, we designed

T ADS 107

Navigation System Development

108

Spatial Audio as an Intuitive Architectural Interface T ADS

As shown in figure 4.6-6, the arm pouch holds the GNSS board which is powered by a portable battery, together with

Fig. 4.6-6 A close-up photo of the armmount part and head-mount

a smartphone. The headphones contain a custom unit on

part of the wearable design.

its top with GNSS Antenna, IMU module and WiFi-enabled

Qian

Photo: Alex Orsholits & Yiyuan

ESP32 microcontroller. The weight of the entire wearable system, including the arm-mount and the head-mount and cables, is within 1kg.

Although power supply and the main processor (in our case, an Android smartphone) can be held in a pouch mounted on the user's arm, there are three parts that have to be mounted on users' heads: the headphones for obvious reasons, the IMU module for head absolute rotation tracking, and the GNSS-RTK antenna for position tracking . As for the IMU [1]

module and the GNSS-RTK antenna, both devices have to

[1] According to GNSS Antennas Application Notes by u-blox, the position of the antenna mounting is crucial for an optimal performance.

The antenna plane should be parallel to the geographic horizon, and the antenna

be mounted together with the user's head and have to be

should have full view of the sky

parallel to the geographic horizon. In order to achieve these

with as many visible satellites

requirements, we decided to utilize the headphones as the

ensuring a direct line-of-sight as possible.

head-mount frame and attach all other devices on the frame.

As discussed in 3.2, free head rotation plays an important role in enhancing human sound localization capability. In previous research about navigation using spatial audio like "AudioGPS" (Holland et al. 2002) and "Running Guide" (Zhu et imposes strict limits on users' experience in actual wayfinding.

109

al. 2019), the absence of rotation tracking for head movement

Navigation System Development

In applications which included rotation tracking module, like "gpsTunes" (Strachan et al. 2005), users have to stop moving and switch mode to allow head rotation to check the audio direction. In all three cases, better integration with real-time absolute rotation tracking was mentioned as an important future improvement direction. In our case, the involvement of IMU module enables free head motion for users while following the track, which achieves the objective.

In order to mount these devices onto the headphones steadily, and at the same time keeping each module's potential to be exchanged or re-configured, a small case is designed and 3d printed with resin to hold the IMU module, the microcontroller and the battery in place. Since the case design includes tiny details to fit the connection parts between each device, resin 3d printing is chosen over filament 3d printing because that the precision of resin printers is much higher than that of the same grade filament 3d printers.

The case provides mounts for the GNSS antenna on the top. The antenna is the device with the most weight (173g) but has to be mounted on the very top. One side of the case Fig. 4.6-7 The design of resin 3d printed case as part of the head-mount to hold, protect and provide easy access to the electronics

remained open to enable exchange of battery and connection to the micro-USB interface. The case is mounted onto the headphones with rubber bands on the two "knobs" with enough tolerance for headphones with various shapes.

110

Spatial Audio as an Intuitive Architectural Interface T ADS

Fig. 4.6-8 (up) Fig. 4.6-9 (down) An exploded view and an assembled view of the head-mount design (cables are omitted in the drawing)

111

Navigation System Development

Before proceeding into tests with unexperienced users, the system was tested briefly outdoor by the developers.

Fig. 4.6-11 (up) Fig. 4.6-12 (down) An outdoor test of the navigation system with the wearable design by the developers. Model: Alex Orsholits, Photo: Yiyuan Qian

112

Spatial Audio as an Intuitive Architectural Interface

T

ADS

113

Navigation System Development

4.7

Feasibility Test

Our spatial audio navigation system was tested with five healthy users in three different urban environment settings.

In each of the five cases, the subject was given the device without training or any explanation of the mechanisms behind the system. Only a simple order to "follow the sound to find your way" was given. In each case, the subject had to complete a certain route, around 500m in length, on foot without the knowledge of this very route.

4.7.1

Test Setup & Results

In all five tested cases, the users were able to complete the pre-determined route using only the spatial audio guidance system.

In one case out of the five, the user continued forwards in the wrong direction at an intersection for about 5 meters, but soon realized their error through the spatial audio and went back to the correct route. We believe this was due to the availability of a side road parallel to the tested route. One user also experienced the IMU failing, suddenly stopping them in their tracks until the battery was replaced.

Although various obstacles hindered the positional precision of the system, in no case was the GNSS signal lost. The trees on one of the routes caused the RTK fix to revert to an RTK float situation, while the highrise buildings on one of the other routes caused loss of all RTK fix, with the system reverting to DGPS (differential GPS). We estimated a worst-case-scenario positional offset of under 0.5 meters in our user tests. All other systems, apart from single the sudden battery loss on the IMU, functioned without any issues.

The information on each test route is as shown in the satellite maps. The following satellite views are set to the exact same scale to comparatively portray the varying urban context.

All three sites a located within 5km from the CQ Publishing RTK basestation to guarantee the precision of RTK correction. The purpose of choosing the three specific areas for testing is to challenge the system's capabilities under [1] Floor area ratio, the ratio of building's gross floor area to the area of the plot.

According to City Planning Law in Japan, Land Use Zones restrictions controls volume and height of buildings by FAR 114

and BCR (Building Coverage Ratio) in each zone.

various environments. The three sites include typical urban pedestrian environment elements, as shown in figure 4.7-3, figure 4.7-5 and figure 4.7-7: narrow (4m) and wide (10m) streets, pedestrian and mixed type of roads, various kinds of intersection turnings, different vegetation coverage and different FAR [1] regulations.

A diagram of Test Route A based on Google Earth Satellite Image. Imagery Date: 3/14/2019

Spatial Audio as an Intuitive Architectural Interface

Fig. 4.7-2

T ADS

Route A: Ushigome (Residential Area) Number of tested user: 2 Total route length: 597m Average street width: 4.5m Type of streets: residential alleys, and distributor roads.

Area building height: below 10m Number of turning points: 8

Fig. 4.7-3 A rough street section drawing of Route A

115

Navigation System Development

Fig. 4.7-4 A diagram of Test Route B based on Google Earth Satellite Image. Imagery Date: 3/14/2019

Route B: Hongo (University Campus) Number of tested user: 1 Total route length: 583m Average street width: 8m Type of streets: avenues, and distributor roads Area building height: 15 - 50m Number of turning points: 7 Fig. 4.7-5 A rough street section drawing of Route B

[1] According to Bastos et al.

(2013), canopy sturctures like tree coverage have significant influence on GNSS accuracy.

[1]

116

A diagram of Test Route C based on Google Earth Satellite Image. Imagery Date: 3/14/2019

Spatial Audio as an Intuitive Architectural Interface

Fig. 4.7-6

T ADS

Route C: Hanzomon (Business Area)

Fig. 4.7-7 A rough street section

Number of tested user: 2

drawing of Route C

Total route length: 514m Average street width: 10m Type of streets: sidewalks + national roads, distributor roads Area building height: 20 - 50m Number of turning points: 7

117

Navigation System Development

4.7.2

User's Feedback

All users were able to complete the tested routes without any additional instruction. The users were able to effectively utilize our spatial audio system to intuitively navigate within an urban context. However, some issues were revealed by the users following testing. We asked each person from their own point of view to provide their opinion on the system, both positive and negative.

Positive comments Generally, the users were satisfied with the system and provided positive comments after completing the entire route. Some comments include "it works very well beyond my expectation", and "it is very easy to figure out the direction."

User 1 commented on the gradual change of the position of the sound source at turning points, also known as the follow-the-leader system, to be a "smooth and easy to follow" aspect. User 4 was especially pleased that there was no need to read a map on a phone anymore, and that they could be guided simply by the music they normally listen to. User 4 also appreciated the inherent feature of spatial audio guidance in which if the user happens to stray off-track towards the wrong direction, it is very easy to realize and correct it.

After a short introduction concerning the system, user 3 attempted to go off route on purpose, to a vending machine, and found it easy to return to the main route and continue the guidance. Both user 3 and user 4 appreciated the freedom of this guidance system as it does not impose a strong feeling of "being guided" onto the users. User 4 especially mentioned their fondness of forgetting the fact that they were being guided while walking straight ahead, only to have the music start to change direction significantly at one point, reminding them to turn.

Problems Despite the positive comments, the negative feedback from the users helped us realize the shortcomings of the system and make further adjustments. The issues include stereo sound confusion, background noise and safety concerns in high noise environments.

During the tests, we allowed users to choose their own

118

music as the guidance audio source. Some of the selections contained specific strong stereo effects, specifically two

Spatial Audio as an Intuitive Architectural Interface

singers, each on a single sound channel, as user 5 commented.

This significantly affected the spatial sound effect. It caused confusion for user 5 at several intersections to hesitate and to wait a certain moment to confirm the next direction. This problem can be easily solved by simply converting all stereo sound to mono, as if it were downmixed to a single speaker, before rendering the spatialized output.

As user 4 and 5 were tested in a business area on the sidewalks of multilane city arteries, the background noise level was significantly higher than that of user 1,2 and 3 who were tested in a residential area and a completely empty campus. User 4 and 5 both pointed out that they had difficulty judging the sound direction when cars were passing by. User 4 adjusted the volume during the test process but failed to notice an approaching vehicle while preparing to cross the street. This phenomenon is due to the masking effect [1]. According to A.

90dB in 30km/h zone. Based on figure 4.7-8, it is necessary to keep the headphones around 65dB or above to have the

by making it hard to discern or not at all detectable. Masking will also occur for a few milliseconds before and after

ADS

broadband noise distributed majorly below 4000 Hz, at around

of one sound has on another T

Can et al. (2010), urban street noise due to traffic is roughly a

[1] Masking effect: the effect

the desired sound.

audio discernable. As for the masking effect specifically for sound localization tasks, it remains unclear of the masking effect's influence since a large majority of sound localization experiments are conducted in quiet indoor rooms. For outdoor usage of spatial sound, especially along urban streets, it is necessary to conduct further research on the masking effect's influence on human sound localization capabilities.

User 1, 2, 4, and 5 all mentioned that the guidance is not very noticeable when they are heading towards the correct direction

Fig. 4.7-8 The change in the threshold of detectability for masking by (a)

straight ahead. This problem was within our expectation as

a 800 Hz pure tone; (b) a 3500

explained in 2.1.2. Front-back confusion is a common issue

uniform spectrum (from Miller,

Hz pure tone; (c) white noise of

in human sound localization, even when using free-field

Language and Communication,

speaker arrays. For digitally rendered spatial sound using

54-55)

McGraw-Hill, N.Y., 1963, pp.

non-individualized HRTFs, the front-back confusion becomes more or less frequently, turned their heads left and right while

119

a more prominent issue. We also noticed that nearly all users,

Navigation System Development Fig. 4.7-9 / Fig. 4.7-10 / Fig. 4.7-11 120

Photos during user tests (photos were taken with the consent of users) Photo: Alex Orsholits, Yiyuan Qian

towards the correct direction. This is a possible solution to the problem at the current stage with the limits in technology.

As for the wearable design, user 1 reported it as "quite heavy" while the other four users found the device acceptable to carry around daily. User 4 complained about the headset being top-heavy, causing difficulties in bending over or running.

Admittedly, our selection of headphones already suffers from a distinct lack of clamping force, and therefore was accentuated

Spatial Audio as an Intuitive Architectural Interface

walking straight, so as to confirm that they were heading

by the added weight.

Further possibilities User 4 explained further about the front-back confusion problem, and how they forgot the fact that they were being guided while walking straight ahead. To this user, it sounded like normal everyday listening experience that they are used

the cognitive load of users by implementing a simple drop-in

ADS

is negative or positive, as our original purpose was to lighten

T

to. It is a questionable judgement whether this phenomenon

augmentation to their auditory perception capacities. User 4 further suggested to add a short audio clip, a light beep or chirp, when the user is approaching a turning point to remind the user that they are being guided.

User 3, who completed the route without even hesitation at all intersections, suggested the possibility for future development to mount the device onto bicycles. The main concern is that the current device, although below 1kg, is an extra weight to the user. Since there is no significant latency for the system, it can be mounted to bicycles to release the burden on the users. In such situations, the open-back headphones can be replaced with bone conduction headsets for safety reasons, as discussed in 3.1.3.

121

122

Spatial Audio as an Intuitive Architectural Interface

T

ADS

123

Projection and Outcome

124

Spatial Audio as an Intuitive Architectural Interface

T

ADS

5 Projection and Outcome

125

Projection and Outcome

5.1

General Comments, Outcomes & Future Research

The effectiveness of our spatial audio guidance system has been established through rigorous technological development, and of course user testing. Albeit a small number of users due to current era human interaction limitations (COVID19), all showed great enthusiasm towards our navigation system due to its immediate intuitive comprehension and general ease of use. Furthermore, the tracking precision we strived to achieve indicates a decisive possible practical and efficacious implementation in the guidance of the visually impaired, in the provided example case of Paralympic marathon runners.

We believe that such a situation could accurately illustrate the raw navigational capabilities of our system, through the demonstration of both the precision, as well as the planetary scale tracking potential. This is an issue that is heavily criticized in other research endeavours as nearly all other developments do not implement high precision positional tracking, nor any form of rotational tracking of the head.

As per user suggestions, the inclusion of spatially agnostic warning cues, such as bell sounds, sirens, bird chirps, etc, along with our developed spatial cues could constitute a part of a further research route. In addition to this, it might be prudent to look into grafting in obstacle detection systems, via LIDAR or computer vision for example, for general visually impaired guidance, and, specifically, how such obstacle detection or depth data can be intuitively interfaced with our spatial auditory capabilities. One of our shortcomings in our research is the absence of the testing of several types of audio sources. Apart from common headphones (closed back, open back), as well as IEMs, we sadly did not have access to active-passthrough headphones, floating speaker headsets, or bone-conduction headsets. The testing of these represents a certain interest of ours. As a central future development, we hope to look into the combining of precise indoor positional tracking methods with our current exterior tracking capabilities to combine the two fundamental scales of architecture.

We wish to emphasize that our goal is not merely an unconditional navigation apparatus. One fundamental aspect of our spatial audio guidance system is that you are visually

126

free while you navigate yourself. We believe that we have differentiated our system from that of a typical navigation

mapping services like Google Maps, Bing Maps, Mapbox, or other, provide. Rather than providing the utmost perfect follow-the-leader type guidance that processes even the exact crosswalk to cross, or the side of sidewalk to walk on. We determined that the user should make their own judgment of specific route, and have therefore adapted our guidance process to reflect this decision. It can be argued that absolute routes lead to potential pedestrian traffic flow control, or worse, active socio-economic manipulation through publicity-based

Spatial Audio as an Intuitive Architectural Interface

tool renouncing the usage of absolute navigation routes that

traffic shaping. Pedestrian traffic is, after all, a fundamental economic motor. We have therefore derived, from the visually freeing facet of our system, a subtly ambiguous route planning algorithm that remains mathematically precise, yet has the definite potential to be expanded beyond simple navigation towards a true urban interface and digital augmentation of our underutilized auditory sense.

T ADS 127

Projection and Outcome

5.2

Beyond Unconditional Navigation

As previously mentioned in the introduction to navigation, urban designers in the Edo period exploited Mt Fuji's palpable towering disposition as an urban landmark to align the central streets of Tokyo. Mt Fuji is to this day a significant cultural icon that still enables, on a clear day, citizens to look beyond the intertwining cityscape towards an unmoveable navigational beacon. It has even become a celebrated event to await the appearance of this icon through the shroud of industrialism.

From ever-rising man-made vantage points, we seek the visual confirmation of a 100'000-year-old volcano. It would simply be absurd to think that Mt Fuji did not profoundly impact the way people interact with the city - Mt Fuji shaped the urban fabric and continues to be celebrated as a metaphysical presence exceeding reality. Likewise, as architects, we aspire to pragmatically achieve, in the very least, a comparable outcome through methods accessible by a majority of society. From traditional architecture design, to the digital augmentation of our very senses, the key to a lively urban environment lies in the elementary spatial perception of the city's citizens, and the mechanisms that complete these perceptions. Throughout our thesis, we roughly described the concept of virtual landmarks as navigational cues. Through our dynamic routing via spatial audio guidance, we expressed our interest in challenging the accepted notions of navigation, seen exclusively as an apparatus for guiding one's self from point A to point B. As the subsection title suggests, we aim to provide a glimpse of our system beyond this elementary act of navigation by simultaneously emphasizing two conditions: 1. The concretisation of prospective urban planning systems 2. The imperative justification and rationalization of said systems

The Architecture of Virtual Reality On a more generalized note, virtual reality has only very recently established itself as a commonly accessible medium. Still in its infancy, many architects regard this visual augmentation device to be a tool for simulating realities, or in the very least the notions common to the physical 128

dimension. Yet, extreme use-cases of this technology treat it more as an outright replacement to our perceptions of the

Spatial Audio as an Intuitive Architectural Interface

world. Albeit quite unhealthy, there is some merit to such an extreme procedure. Virtual environments may no longer be considered mere extensions of our perceived realities. These environments are equipped with their own infrastructural constraints and unique architectural expression. Just as certain architectural practices address the interfacing of the spiritual reality with that of human reality through a long history of religious structures, we believe that architects should consider the design of virtual environments with equal importance to that of physical reality as the two are astutely woven together, yet essentially independent from each other.

We can no longer disregard this entirely new fabric, simply due to its very existence. Additionally, the freedom acquired by operating within an entirely clean slate can be approached to the ideals of Modernism. Not only can an infinite amount of

[1] Hart, Vi & Hawksley, Andrea

virtual environments be overlaid on the same physical space,

Segerman, Henry. (2017).

Euclidian space itself no longer needs to be respected .

[1]

both at the same time if one so desires.

explorations of H3

ADS

designing new spaces, as well as augmenting existing spaces,

Non-euclidean virtual reality II:

T

Fundamentally, Virtually Reality provides the possibility of

& Matsumoto, Elisabetta &

Prospective Developments - Soundscapes as Urban Cues A particular feature of implementing absolute position tracking in our system is the flexibility to design universal cues which natively mesh with the urban fabric in a dynamic, alwaysshifting state. This represents a virtually infinite and invariable empty environment, ripe for the introduction augmented auditory cues, in addition to our active route following guidance. Virtual reality, or rather, virtual realities can be specifically adjusted for intuitive architectural interfacing, by decomposing the intricacies of the complex urban fabric into unique waypoints that emphasize explicit features of the surrounding environment. By streaming, for example, the sound of trains, or the specific train announcements to a virtual speaker just at the entrance of the station could provide an easy to find navigational cue, as well as temporal information equating to the general availability of the public transport at the time of passing by. Additional collective waypoints could be applied essentially anywhere. Our follow-the-leader guidance could be further revised to convey the final destination in 3D space, whether it be simply a street we wish to reach the case for our user tests), or a distinct floor of a high-rise.

129

which happens to be at the same level as ourselves (this was

Projection and Outcome

Due to the omnidirectional properties of spatial audio, the final destination doesn't need to be seen to be discovered.

Overhead, or below our feet, our augmented acoustic sense is able to accurately detect the provided cues. These virtual cues could be considered similar to visual, physical signage, but as previously stated, auditory cues implemented within a virtual infrastructure can be re-arranged, remixed, combined, stacked, or deleted. Another, entertaining, development possibility is that of giving the users the ability to apply custom mappings to their own surroundings by adding personal cues at positions significant to them. This may be the most sincere and accessible architectural interfacing achievable. It allows users to adopt the urban fabric, a foreign entity, as their own uniquely appreciated environment. A personal space beyond the threshold of one's home, and infinitely expandable. The provided examples are meant to reflect real-world scenarios rather than be developments of novel concepts, as such concepts would require significant research before they can be thoughtfully articulated.

Virtual Infrastructures as a Platform for Intuitive Interfacing Through rapid mobile networking, a seemingly unlimited amount virtual landmarks can be collectively experienced, entangling the real urban environment with a new class of intuitively understood data. It is our conviction that the consolidation of both mutual and personal urban cues, possible with digital technologies exclusively, will in time redefine what it means to interface with our immediate and interpersonal surroundings. The inexhaustible complexification of the urban fabric, the incessant obstruction of navigational icons, the boundless layering of infrastructures — a new degree of alertness needs to be realized. As some users mentioned, they would feel safer with the implementation of short, non-spatial, auditory cues like beeping, or bird chirps in the very least.

Perhaps the always-shifting nature of virtual environments could enable the implementation of positionally anchored, yet personalized navigational cues that empower the user to both experience and remember the city. Such cues would always be available, at the exact points in space where our self-motion capacities remember them to be. Rudimentary implementations of the sort, like PokĂŠmon Go, have already 130

taken the world by storm. For the sake of entertainment, such videogames already implement several prerequisites

collective landmarks of communal PokĂŠgyms, that mostly coincide with real landmarks, is a clear demonstration of the fundamentals of virtual layering as an augmentation of the existing urban fabric. In essence, our spatial audio system enables us, as architects, to design complex soundscapes that can be reconfigured on the fly. It is a virtual infrastructure that no longer suffers from the unchangeable, permanent nature of physical infrastructures, thus enabling a certain resilience to the inevitable decay of these physically-bound

Spatial Audio as an Intuitive Architectural Interface

to our vision of a universal and enhanced environment. The

urban fabrics.

Through our development of a technologically advanced spatial audio guidance system, we are confident that we have defined a coherent guideline that will enable us to commit to our proposed developments. We wish, in the near future, to continue our research into the explored topics presented in our

in our endeavours, or in the very least feel persuaded by the

ADS

and hope that you, the reader, have been enticed to help us

T

thesis on Spatial Audio as an Intuitive Architectural Interface,

presented theories. We look forward to your comments and suggestions and eagerly await the ensuing debates.

131

Projection and Outcome

Close-up visualization of a theoretical soundscape overlaid onto the city

132

Spatial Audio as an Intuitive Architectural Interface

T

ADS

133

References

134

Spatial Audio as an Intuitive Architectural Interface

T

ADS

6 References

135

References

Adams, S., and Boland, F. 2010. "On The Distortion Of Binaural Localization Cues Using Headphones". IET Irish Signals And Systems Conference (ISSC 2010).

Algazi, V. R., Duda, R. O., Thompson, D. M., and Avendano, C. "The CIPIC HRTF database," Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat. No.01TH8575), New Platz, NY, USA, 2001, pp. 99-102.

Amemiya, T., and Sugiyama H. 2010. "Orienting Kinesthetically: A Haptic Handheld Wayfinder For People With Visual Impairments". ACM Transactions On Accessible Computing 3 (2): 1-23.

Bacon, E. 1974. Design Of Cities. London: Thames and Hudson.

Bahu, Hélène, Thibaut Carpentier, Markus Noisternig, and Olivier Warusfel. 2016. "Comparison Of Different Egocentric Pointing Methods For 3D Sound Localization Experiments". Acta Acustica United With Acustica 102 (1): 107-118.

Bastos, Alex Souza, and Hisashi Hasegawa. 2013.

"Behavior Of GPS Signal Interruption Probability Under Tree Canopies In Different Forest Conditions". European Journal Of Remote Sensing 46 (1): 613-622.

Batteau, D. W.. “The role of the pinna in human localization.” Proceedings of the Royal Society of London.

Series B. Biological Sciences 168 (1967): 158 - 180.

Can, A., L. Leclercq, J. Lelong, and D. Botteldooren. 2010.

"Traffic Noise Spectrum Analysis: Dynamic Modeling Vs. Experimental Observations". Applied Acoustics 71 (8): 764770.

Carlile, Simon. Virtual auditory space: Generation and applications. New York: Chapman and Hall, 1996.

Cho, S. J., Ovcharenko A., and Chong, U. "Front-Back Confusion Resolution in 3D Sound Localization with

136

HRTF Databases," 2006 International Forum on Strategic Technology, Ulsan, 2006, pp. 239-243.

Priya Murugeswaran, Yuqing Shi, Ziyi Wu, Yusuke Obuchi and Jun Sato. “Using non-linear logic for construction of self-stabilizing hammock structure.” AAG (2018).

Gallay, Mathieu, Denis, Michel, and Auvray, Malika. 2013.

"Navigation assistance for blind pedestrians: guidelines for the design of devices and implications for spatial cognition." Representing Space in Cognition, pp.244-267.

Spatial Audio as an Intuitive Architectural Interface

Furuichi, Shohei, Shuntaro Nozawa, Machiko Asahara,

Gardner, William G., and Keith D. Martin. 1995. "HRTF Measurements Of A KEMAR". The Journal Of The Acoustical Society Of America 97 (6): 3907-3908.

Grantham, D. Wesley. 1984. "Discrimination Of Dynamic Interaural Intensity Differences". The Journal Of The Acoustical Society Of America 76 (1): 71-76.

T

dynamic sound source localization in a virtual room.” 337-

ADS

Gröhn, M & Lokki, Tapio & Takala, Tapio. 2002. “Static and 344.

Hart, Vi, Hawksley, Andrea, Matsumoto, Elisabetta, and Segerman, Henry. 2017. “Non-euclidean virtual reality II: explorations of H3” Holland, S., Morse, D. & Gedenryd, H. 2002. "AudioGPS: Spatial Audio Navigation with a Minimal Attention Interface." Personal Ub Comp 6, 253–259 (2002).

Katz, Brian & Kammoun, Slim & Parseihian, Gaëtan & Gutierrez, Olivier & Brilhault, Adrien & Auvray, Malika & Truillet, Philippe & Denis, Michel & Thorpe, Simon & Jouffrais, Christophe. 2012. "NAVIG: Augmented reality guidance system for the visually impaired: Combining object localization, GNSS, and spatial audio." Virtual Reality. 16. 17.

Klatzky, Roberta L., James R. Marston, Nicholas A. Giudice, Reginald G. Golledge, and Jack M. Loomis.

2006. "Cognitive Load Of Navigating Without Vision When Guided By Virtual Sound Versus Spatial Language." Journal Of Experimental Psychology:

137

Applied 12 (4): 223-232.

References

Knudsen, EI. 1982. "Auditory And Visual Maps Of Space In The Optic Tectum Of The Owl". The Journal Of Neuroscience 2 (9): 1177-1194.

Kolarik, Andrew J et al. “Auditory distance perception in humans: a review of cues, development, neuronal bases, and effects of sensory loss.” Attention, perception & psychophysics vol. 78,2 (2016): 373-95.

Kon, Y., Nakamura, T., and Kajimoto, H. "Interpretation of navigation information modulates the effect of the waisttype Hanger Reflex on walking," 2017 IEEE Symposium on 3D User Interfaces (3DUI), Los Angeles, CA, 2017, pp.

107-115.

Lane, Chris, and Gould, Richard. 2018. "Let’s Test: 3D Audio Spatialization Plugins". Link: https:// designingsound.org/2018/03/29/lets-test-3d-audiospatialization-plugins/ Leong, Philip, and Simon Carlile. 1998. "Methods For Spherical Data Analysis And Visualization". Journal Of Neuroscience Methods 80 (2): 191-200.

Lokki, T., and M. Grohn. 2005. "Navigation With Auditory Cues In A Virtual Environment". IEEE Multimedia 12 (2): 80-86.

Lynch, Kevin. 1960. The Image Of The City. Cambridge, Mass.: MIT Press.

Maki, Fumihiko, Yukitoshi Wakatsuki, Hidetoshi Ōno, Tokihiko Takatani, Naomi R Pollock, and Hiroshi Watanabe. n.d. 2018. City With A Hidden Past. Kajima Institute.

Marston, James R., Jack M. Loomis, Roberta L. Klatzky, Reginald G. Golledge, and Ethan L. Smith. 2006.

"Evaluation Of Spatial Displays For Navigation Without Sight". ACM Transactions On Applied Perception 3 (2): 110-124.

Matsumoto, Mitsuo, Mikio Tohyama, and Hirofumi Yanagawa. 2003. "A Method Of Interpolating Binaural

138

Impulse Responses For Moving Sound Images". Acoustical Science And Technology 24 (5): 284-292.

Localization With Head Movement: Implications For 3-D Audio Displays". Frontiers In Neuroscience 8.

Mendonça, Catarina & JA, Santos & Campos, Guilherme & Dias, Paulo & Vieira, Jose. 2012. "On the adaptation to non-individualised HRTF auralisations: A longitudinal study". Proceedings of the AES International Conference.

Mattes, Symeon & Nelson, Philip & Fazi, Filippo & Capp,

Spatial Audio as an Intuitive Architectural Interface

McAnally, Ken I., and Russell L. Martin. 2014. "Sound

Michael. 2012. "Towards a human perceptual model for 3D sound localization." Proceedings of the Institute of Acoustics. 34.

Middlebrooks, John C., and David M. Green. 1991. "Sound Localization By Human Listeners". Annual Review Of Psychology 42 (1): 135-159.

T

Journal Of The Acoustical Society Of America 30 (4): 237-

ADS

Mills, A. W. 1958. "On The Minimum Audible Angle". The 246.

Moholy-Nagy, L. 1946. Vision In Motion. Chicago: Paul Theobald.

Okreylos. 2016. "Lighthouse tracking examined". Link: http://doc-ok.org/?p=1478 Opstal, John van. 2016. The Auditory System And Human Sound-Localization Behavior. Academic Press.

Perrott, David R., and Kourosh Saberi. 1990. "Minimum Audible Angle Thresholds For Sources Varying In Both Elevation And Azimuth". The Journal Of The Acoustical Society Of America 87 (4): 1728-1731.

Ordonez, C. & Barreto, Armando & Gupta, Navarun. 2002.

"2-D auditory mapping of virtual environment using realtime head related transfer functions". Proceedings IEEE SoutheastCon 2002, Columbia, SC, 2002, pp. 364-369.

Perrett, Stephen, and William Noble. 1997. "The Contribution Of Head Motion Cues To Localization Of 1026.

139

Low-Pass Noise". Perception & Psychophysics 59 (7): 1018-

References

Pieralisi, Marco, Valentina Di Mattia, Valerio Petrini, Alfredo De Leo, Giovanni Manfredi, Paola Russo, Lorenzo Scalise, and Graziano Cerri. 2017. "An Electromagnetic Sensor For The Autonomous Running Of Visually Impaired And Blind Athletes (Part I: The Fixed Infrastructure)". Sensors 17 (2): 364.

Pollack, I., & Rose, M. 1967. "Effect of head movement on the localization of sounds in the equatorial plane".

Perception & Psychophysics, 2(12-A), 591–596.

Populin, Luis C. 2008. "Human Sound Localization: Measurements In Untrained, Head-Unrestrained Subjects Using Gaze As A Pointer". Experimental Brain Research 190 (1): 11-30.

Psathas, George. “Mobility, Orientation, and Navigation: Conceptual and Theoretical Considerations.” Journal of Visual Impairment & Blindness 70, no. 9 (November 1976): 385–91.

Rämö, Jussi, and Vesa Välimäki. 2012. "Digital Augmented Reality Audio Headset". Journal Of Electrical And Computer Engineering 2012: 1-13.

Reijniers, Jonas & Vanderelst, Dieter & Jin, Chim & Carlile, Simon & Peremans, Herbert. 2014. "An ideal-observer model of human sound localization". Biological cybernetics.

Roentgen, U. R., Gelderblom, G., J., Soede, M., & de Witte, L. P. 2008. "Inventory of electronic mobility aids for persons with visual impairments: A literature review".

Journal of Visual Impairment and Blindness, 102, 702-724.

Rutherford, Peter. 2001. "Virtual Acoustic Technology: Its Role in the Development of an Auditory Navigation Beacon for Building Evacuation". Brunel University Serafin, Stefania, Michele Geronazzo, Cumhur Erkut, Niels C. Nilsson, and Rolf Nordahl. 2018. "Sonic Interactions In Virtual Reality: State Of The Art, Current Challenges, And Future Directions". IEEE Computer Graphics And Applications 38 (2): 31-43.

140

Serino, Andrea, Elisa Canzoneri, and Alessio Avenanti.

Representation Of Peripersonal Space In Humans: An Rtms Study". Journal Of Cognitive Neuroscience 23 (10): 2956-2967.

Shilling, R. and B. G. Shinn-Cunningham. 2000. "Virtual Auditory Displays. To appear in Handbook of Virtual Environment Technology." K. Stanney (ed), Lawrence Erlbaum, Associates, Inc., in press.

Spatial Audio as an Intuitive Architectural Interface

2011. "Fronto-Parietal Areas Necessary For A Multisensory

Simpson WE, Stanton LD. "Head movement does not facilitate perception of the distance of a source of sound."

Am J Psychol. 1973;86(1):151-159.

Steadman, M., Kim, C., Lestang, J., Goodman, D. and Picinali, L., 2019. "Short-term effects of sound localization training in virtual reality". Scientific Reports, 9(1).

T

"Auditory Accommodation To Poorly Matched Non-

ADS

Stitt, Peter, Lorenzo Picinali, and Brian F. G. Katz. 2019.

Individual Spectral Localization Cues Through Active Learning". Scientific Reports 9 (1).

Strachan, Steven & Eslambolchilar, Parisa & MurraySmith, Roderick & Hughes, Stephen & O'Modhrain, Sile. 2005. "GpsTunes: controlling navigation via audio feedback." Proceedings of the 7th Conference on HumanComputer Interaction with Mobile Devices and Services, Mobile HCI 2005, 2005, Salzburg, Austria, September 19-22, 2005 Sugiyama, Wakana. 2019. "Voice guidance in Maps, built for people with impaired vision". Link: https://www.blog. google/products/maps/better-maps-for-people-withvision-impairments/ Thurlow, Willard R., and Philip S. Runge. 1967. "Effect Of Induced Head Movements On Localization Of Direction Of Sounds". The Journal Of The Acoustical Society Of America 42 (2): 480-488.

Torres-Gallegos, Edgar A., Felipe Orduña-Bustamante, and Fernando Arámbula-Cosío. 2015. "Personalization Automatic Photo-Anthropometry And Inference From A

141

Of Head-Related Transfer Functions (HRTF) Based On

References

Database". Applied Acoustics 97: 84-95.

Trahiotis, C, and D E Robinson. 1979. "Auditory Psychophysics". Annual Review Of Psychology 30 (1): 3161.

Velázquez, Ramiro, Edwige Pissaloux, Pedro Rodrigo, Miguel Carrasco, Nicola Giannoccaro, and Aimé Lay-Ekuakille. 2018. "An Outdoor Navigation System For Blind Pedestrians Using GPS And Tactile-Foot Feedback". Applied Sciences 8 (4): 578.

Wallach, H. 1940. "The Role Of Head Movements And Vestibular And Visual Cues In Sound Localization." Journal Of Experimental Psychology 27 (4): 339-368.

Wang, Ranxiao & Spelke, Elizabeth. 2001. "Updating egocentric representations in human navigation.

Cognition". Cognition. 2000;77(3):215-250.

Wenzel, E. M., Arruda, M., Kistler, D. J., and Wightman, F. L. "Localization using nonindividualized head-related transfer functions". The J. Acoust. Soc. Am. 94, 111–123 (1993).

Yost, William A, and George Gourevitch. 1987. Directional Hearing. New York: Springer US. Zahorik, Pavel, Brungart, Douglas, and Bronkhorst, Adelbert. 2005. "Auditory distance perception in humans: A summary of past and present research." Acta Acustica united with Acustica. 91. 409-420.

Zhu, Yancong, Cong Wang, Wei Liu, and Yi Lv. 2019.

"Running Guide". Proceedings Of The Seventh International Symposium Of Chinese CHI On - Chinese CHI '19.

Ziegler, Christiane, and Jean-Luc Bovot. 2011. Art Et Archéologie: Égypte Ancienne. Paris: Ecole du Louvre.

Zwislocki, J., and R. S. Feldman. 1956. "Just Noticeable Differences In Dichotic Phase". The Journal Of The Acoustical Society Of America 28 (5): 860-864.

142

Spatial Audio as an Intuitive Architectural Interface

T

ADS

143

Appendix

144

Spatial Audio as an Intuitive Architectural Interface

T

ADS

7 Appendix

145

Appendix

Appendix 1. ESP32 IMU UDP Transmission (Arduino)

146

Spatial Audio as an Intuitive Architectural Interface

T

ADS

Appendix 2. NTRIP Correction Data Client (Python 3.x)

147

Appendix

Appendix 3. Unity GNSS UDP Data Client (C#)

Appendix 4. Unity Player Translation (C#)

148

Spatial Audio as an Intuitive Architectural Interface

Appendix 5. Unity IMU UDP Data Client

T

ADS

149

Appendix

Appendix 6. Unity Sound Source Translation (C#)

150

Spatial Audio as an Intuitive Architectural Interface

Appendix 7. ODIN-W2xx WiFi Configuration (AT Commands)

T

ADS

151

Appendix

152

Spatial Audio as an Intuitive Architectural Interface

T

ADS

153

Š Alex Orsholits and Yiyuan Qian, 2020 Version for digital distribution | 178mm*254mm