Virtual Art Spaces: InterFREQUENCIES

Chair of Architectural Informatics

Technical University of Munich

Hlib Novosolov, Yoran Erami, Clemens Lindner, Noah Lokocz

InterFREQUENCIES

InterFREQUENCIES

Chair of Architectural Informatics

Prof. Dr.-Ing. Frank Petzold

Interactive visualization: Virtual Art Spaces

Frank Petzold, Lars Wüstemann, Gerhard Schubert, Nick Förster

Hlib Novosolov, Clemens Lindner, Noah Lokocz, Yoran Erami

Table of Contents

Introduction

Vicinity and distance have been presented as increasingly important factors since the pandemic invaded our lifes. The safe and personal space and its inherent vulnarability and fragility in bigger groups has been an inspiration for the spatial interaction between the artwork and the participants.

interFREQUENCIES aims to create an interactive art installation that enables a visual conversation between an artwork and a person. In collaboration with the Munich-based artist, Tatjana Busch, a resonance space with a responsive sensor system was developed. This system expands the emotional perception of her artworks by creating a perception loop, where sensors and the human body constantly react to each other in relation to distance, allowing visitors to influence the flow of the exhibition.

The original installation „Frequencies“ by the artist, Tatjana Busch, plays and communicates. It moves between the objective, the physically determined and the personal perception of the viewer, the world of the emotional, the irrational, the mystical. In this process, objects are created that change their role from 3-dimensional sculpture to light - reflector, generating reflective meandering light spaces, attended with immersive sound, composed by Christian Losert. The color frequencies are translated into sounds : One could say, how does red sound? What does 480 tetra hertz sound like? What does it do to me?

We attach to her work by creating a communicative space around her artwork by using sensors and automised variability in lightintensity, -frequency and sound.

Distance and approach have become the language of the participant while the artwork expresses itself through in- and decreasing audio-visual spatial immersion. Vulnarability and fragility is in this communication the topic of conversation.

Research introduction

To create the conversation between artwork and observer, it is critical to understand the bodily communication systems relating to distance and spatial perception. Furthermore a sensor must be found that can be applied to the artwork so that the illusion is created that the artwork has awareness of its surroundings.

Therefore the project started with a thorough research along the topic “Corporeal interaction and spatial perception in hybrid spaces”, presented in this chapter. The chapter has been divided in three sections with separate research questions:

Section 1

What is corporeal communication and how is it translated?

Section 2

How is space perceived and translated?

Section 3

What is a hybrid space and what kind of examples are present in current time?

Each section starts with a definition of the separate topics creating a first direction.

Corporeal Interaction

“corporeal” - adjective; physical and not spiritual, relating to the body

“interaction” - noun; an occasion when two or more people or things communicate with or react to each other

“corporeal interaction“ - an occasion when two or more things relating to the body communicate with or react to each other.

“For the body to function properly, its various parts and organs must communicate with each other to ensure that a constant internal environment (i.e., homeostasis) is maintained.” - Hiller-Stürmhofel

Corporeal communication

The human body consists of two systems to help ensure communication: the nervous system and the hormonal (i.e., neuroendocrine) system. The nervous system generally allows rapid transmission (i.e., within fractions of seconds) of information between different body regions. Conversely, hormonal communication, which relies on the production and release of hormones from various glands and on the transport of those hormones via the bloodstream, is better suited for situations that require more widespread and longer lasting regulatory actions. Thus, the two communication systems complement each other. In addition, both systems interact: Stimuli from the nervous system can influence the release of certain hormones and vice versa. (Hiller-Stürmhofel, 1998)

nervous and endocrene system

Fig.

INFORMATION INPUT

FEEDBACK INFORMATION (hormones)

DETECTION

PERCEPTION

Information (hormones)

REACTION X

Information (impulses)

REACTION Y

FEEDBACK INFORMATION (impulses)

Fig. 2: schematic representation of the nervous system (left) and Fig. 3: the endocrin system (right)

Fig. 4: summary of the interaction between the endocrin and nervous system

Both systems have receptors that translate the external influences to the language of the accompanying system; in the case of the nervous system the language is electrical impulses and in the endocrine system the language is hormones. These systems detect changes through receptors and can elicit external responses only by acting on effectors. The effectors of the nervous system are muscles and glands, and the endocrine system only uses glands. A related point to consider is that these muscle contractions can have various expressions, varying from speech to facial expression and bodily posture. (Brodal, 2016)

The bridge between both systems is the hypothalamus. This gland, in the brain, is affected by the nervous system and reacts by giving off regulatory hormones to the other glands in your body. In a sense the hypothalamus functions as the translator between the language of the nervous system and the endocrine system (Brodal, 2016).

Corporeal

translation

After the clarification of the communication systems of our body, it is important to understand how our body translates the perception of space to the language of these systems.

External information is perceived through receptors of the nervous system. The human body has 5 systems to collect information: the somatosensory, vestibular, visual, auditory and the olfactory system, respectively the systems for touch/pain/position/temperature, balance/orientation, vision, sound and smell. To perceive space, the human body mainly uses the vestibular, visual and auditory systems. (source)

The information that our body senses is put into context by our brain. According to Chatterjee and Vartanian aesthetic experience emerges from the interaction between three systems: the sensory-motor, emotional valuation and meaning knowledge systems, all together called „the aesthetic triad“. The sensory motor system is comprised of all the sensors (the 5 systems) we have in our body, the knowledge-meaning system focuses on context, culture and the valuation of expertise and lastly the emotion valuation focuses on everything related to the emotional reward system in the human brain. Although this triad often collaborates to create our aesthetic experience it is not stated that the observed object always equally influences all the systems, it is very possible to have an object only activating one of the three systems in the brain.

The emotion-value system and meaning knowledge system is where the perception of fragility and vulnarability is evoked, these terms are related to previous experiences and emotional references. As these experiences and references might show differences between people it is imperative to generalise`and understand perception theory.

Fig.5

corporal translation diagram

Spatial perception

“Spatial perception” - the ability to be aware of your relationships with the environment around you (exteroceptive processes) and with yourself (interoceptive processes) (Cognifit, 2022).

“Space perception” - process through which humans and other organisms become aware of the relative positions of their own bodies and objects around them. Space perception provides cues, such as depth and distance, that are important for movement and orientation to the environment. (Encyclopedia Britannica)

Introduction

This chapter will review 4 key aspects related to visual perception. This research will provide a basic understanding of different ways to manipulate the mind and assists in making decisions related to the proposed distortion.

Perception of contrast

Visual information processing starts at the retaina by the detection of light. The retina processes the input of light focussing on contrast instead of light intensity. Results of a study performed in 1973 show that perceived contrast is not determined solely by luminance nor lightness (Guth, 1973). Instead they suggest that luminance and lightness, and contrast share common underlying mechanisms but can be assessed independently at least to some extent, „It is possible that in complex scenes perceived contrast is determined at a later stage and in a higher cortical area in the hierarchy of the visual system, after lightness is computed based on global information“ (Pamir, Z. Boyaci, H. 2016).

Fig.6 relativity of brightness perception diagram (Pamir, Z. Boyaci, H. 2016)

Fig. 7

Fig. 8

symmetry perception diagram

(J. Martinovic, B. J. Jennings, A. D. J. Makin, M. Bertamini, I. Angelescu., 2012)

pattern perception diagram

(S.O. Murray, D. Kersten, B.A. Olshausen, P. Schrater, D.L. Woods. 2002)

Pattern perception

Secondly, according to a group of researchers from the National Academy of Sciences, U.S.A., visual perception involves the grouping of individual elements into coherent patterns that reduce the descriptive complexity of a visual scene.

A measurement session of certain parts of the brain relating to vision indicates that activity in early visual areas is reduced as a result of grouping processes performed in higher areas. This indicates that, first of all, our brain is susceptive to pattern recognition (Murray et al., 2002).

Perception of symmetry

Thirdly, symmetry (from Ancient Greek: symmetria “agreement in dimensions, due proportion, arrangement”) in everyday language refers to a sense of harmonious and beautiful proportion and balance. In mathematics, “symmetry” has a more precise definition, and is usually used to refer to an object that is invariant under some transformations.

“Perception of visual symmetry is fast and efficient and relies on both early low-level and late mid- and highlevel neural mechanisms.“, “symmetry perception engages a relatively broad extrastriate network that is not fully constrained by low-level inputs once contrasts are set to high, suprathreshold levels, contextualizing the lack of V1/V2 contributions reported by previous neuroimaging work.” (J. Martinovic, B. J. Jennings, A. D. J. Makin, M. Bertamini, I. Angelescu., 2012)

Perception of similarity

Lastly, similarity as perceptual resemblance is a psychological construct in somewhat the same way that such sensory attributes such as hue, brightness, or pitch are constructs or intervening variables. Like such attributes, similarity depends heavily on the physical characteristics of stimulus objects, but this dependence is complex, and in the case of similarity the details of the physicalpsychological relationship are usually unknown. However, similarities may be specified independently of any physical measures of the stimuli involved; that is, one may determine the similarities among a group of objects from behavioural responses to the objects without specifying anything about them except which behavioural measures go with which object (Dr. R. G. Cook, 2001)

Fig. 9

similarity perception diagram (Dr. R. G. Cook, 2001)

Hybrid spaces

Rethinking the accessibility of art

In addition to the actual space and virtual space, there is a third type that is be called hybrid space. Hybrid space utilizes a stream of information to empower the physical space around us using technologies such as augmented reality, virtual reality, and augmented virtuality. Hybrid space has been explored and conceptualized in the literature, but it has yet to reach its potential as an effective medium in museums. An opportunity to interact with virtual objects while maintaining the awareness of actual physical space is nevertheless promising. Such space has a potential to make art more accessible and allow visitors to engage with each other (Baradaran Rahimi, F., Levy, R. M., & Boyd, J.E., 2018).

Reality integration

When integrating the “new reality” into an art space, it is important to understand the relationship between virtual and real components of the experience. The most common combination is augmented reality (AR), which overlays a virtual world on top of the real one. The opposite principle of overlaying real world on top of a virtual one is also possible and is called augmented virtuality (AV). The less common technique is so-called diminished reality (DR), where some aspects of real world are purposely removed, creating altered experience for a user. Mediated reality (MR) is a more complex way to alter one’s experience, because it influences and modifies information about the world instead of adding or removing it. In addition to that if the AR, MR, or AV overwhelm user with too much information for comprehensive response, the user deals with hyper reality (HR)(C. Maxfield, 2022).

Fig.

Sensor catalog

A wide range of sensors have been researched to create a certain awareness of the tools that can be used to assist in creating a hybrid space that works with spatial perception and corporeal interaction. We compared the different inputs and outputs of existing sensors with similar use cases, to achieve the wanted outcome in real time.

For more detailed information about the output signals, see Appendix 1.

Spatial Bodily Hybrid

Ultrasonic distance sensor

Heartbeat sensor

Thermal camera

Infrared distance sensor

ECG/EKG

Laser distance sensor

Sweat sensor

Accelaration sensor

Sound sensor

Temperature sensor

Fig. 11 sensor catalog

Scenario

As previously stated, interFREQUENCIES will be a conversation between art piece and participant with the artwork´s fragility and vulnarability as main topics with distance as language. A constant loop between the sensors attached to the body of the participants and the sensor of the artwork is created to develop this communication: the sensors measure the approach to the artwork by the participant, an audio-visual response is generated and this is sensed by the participant evoking a new kind of motor response that restarts the entire process as is shown in figure 12.

Fragility and vulnerability appear at borders of comfort, both indicating an apparent but sudden danger of destruction or hurting. As humans we create a set of boundaries and borders of which not to cross both unspoken, as norms and values and written down in the law. The artwork does not possess the tools to create the same set of rules that we created over years of dialogue. InterFREQUENCIES provides the artwork with tools to indicate its comfortable borders to the participants of the exhibition.

To develop a first understanding of this border among the participants, the project aims to use space in such way that a forced interaction is created at the start of the experience, as is shown in Fig. 12. Participants are, by merely the layout of the room, required to closely pass one of the artworks, thereby initiating a response and realising their effect.

Fig. 12 scenario diagram

Fig. 13

working principle diagram

These tools should create a sudden discomfort along the preset borders of the different artworks to state that the participant is getting too close. This discomfort is created by the disruption of the artwork. The calming sounds fade and the play of light gets disturbed by a stroboscope creating fragmentations of the visible, thus disorienting the participants and invading the calmness of the exhibition.

As shown in figure 13, the approach is connected to the intensity of disturbances. The closer a participant gets to the artwork the higher the frequency of disturbance.

This same intensity gradient is also applied to the amount of people approaching the artwork, therefore the more the artwork is ´threatened´ the higher the intensity of the disturbance, adding to the concept of borders and boundaries.

Prototyping

Introduction

To develop a better understanding about the programming behind our proposed sensor loop, a prototyping approach has been chosen. Two test setups have been created which gave new insights in the coding of the final setup, this chapter will discuss these different prototypes. For more detailed information about the different scripts see Appendices 3 & 4.

At the same time a wearable that attaches to the experience of the artwork has been designed, iterations and developments will be discussed in this chapter secondly.

Technical prototype 1

Our first setup consisted of an Arduino Uno connected to a breadboard with 6 different LED‘s and an ultrasonic distance sensor. The goal of this setup was to develop an understanding of the preciseness of the ultrasonic sensor. We quickly discovered that the ultrasonic sensor was not applicable. The sensor is unidirectional and singular in its measurement, meaning that it only reacts to the nearest object. The code of this prototype was written with if ‚statements‘ meaning if that if the ultrasonic sensor measured a certain distance X, a certain amount of Y lights will turn on. With this setup we have to determine the distances ourselves and the approach will not influence the lights gradually but rather stepwise as shown in figure 15.

Prototype 1

From left to right: the shorter the distance, the more LED‘s turn on

Current data interpretation: if distance < than X, perform Y

Fig. 14: prototype 1 demo

Fig.15: data interpretation graph

From left to right: the closer our testsubject, with the ESP tag, comes to the ESP anchor, the brighter the LED gets.

New data interpretation: a gradual, parabolic reaction to distance; the closer you get, the faster the reaction increases.

Prototype 2

Fig. 16: prototype 2 demo

Fig.17: data interpretation graph

Technical prototype 2

The second prototype consisted of two ESP32 microcontroller, the Arduino UNO microcontroller and one LED. The ESP32 microcontrollers use the Bluetooth® signal strength of the other ESP32 to approximate their distance.. Initially they were programmed to use Bluetooth® Low Energy or BLE, in order to use less power when powering the devices with a portable battery. These ESPs approximated the distance and sent this data to a laptop over a wireless network. On the laptop, we received the data using a program written in Python, the program remaps to values between 0 & 1 which were then squared to create a more parabolic response arch and thus a more ‚sudden‘ reaction to the approach. Although the test setup worked we quickly noticed that the values were not as precise as we wanted them to be; the data had a lot of major anomalies that resulted in the LED suddenly increasing in brightness but also sometimes falling back to a previous measurement and therefore dimming while the person was still approaching. After reprogramming the ESP32‘s to use regular Bluetooth® the data quality slightly improved. Unfortunately, this data also wasn‘t precise enough, which is why we conducted futher research and ended up with DWM1001 development modules. These development boards with on-board Ultra-wideband modules are also part of our final implementation.

Final implementation

Technical

InterFREQUENCIES makes use of an ultra wide band (UWB) transceiver module mounted on a DWM1001 development board. The DWM1001‘s function as both anchor and tag. The DWM1001 anchor measures time of flight (TOF) of signals travelling between the anchor and the tags. The distance between the objects can be calculated by using the following formula:

Distance [m] = Speed of radio waves [m/s] x TOF [s]

The DWM1001 communicates with an ESP32 via a universal asynchronous receivertransmitter (UART), the ESP32, in its turn, uses a wireless network to transfer all the data to the laptop. A python program remaps the received data and directs this to a QLC+ controller that operates the light and sound installation as seen in figure 18.

For more detailed information about the python code, see Appendix 2.

ESP32

Wi-Fi

Access point DWM1001

DMX interface

Laptop (server) Lights

Fig. 18

technical concept diagram

Approach

In figure 19 you can see the impact of the height of the sensor on the artwork in relation to the height of the wearable. As the sensor is mounted on the ceiling the distance to the wearable increases almost parabolic when you approach the artwork. This enhances the effect of interaction with the artwork as it feels like a pulse rather than a gradually fading answer of the light and sound installation.

Fig.19

Approach diagram

Wearable prototype

Ideation

As we moved on to use DWM1001 developement boards with an ultrawideband (UWB) transceiver module, the board needed a battery and a hull that can be carried by a visitor in order to measure their distance to an artwork.

Every third or fourth person in the crowd receives a wearable and can influence the artworks. In order to make the correlation between movement of the wearable and reaction of an artwork, the wearable is made noticeable, highly refractive and reflective. It is worn as a necklace in contrast to e.g. a wristband in order to make it closer to a preson‘s heart, enabling them to move their whole body if they wish to move the wearable.

The wearable was designed in collaboration with Tatjana.

Fig.

Prototype

The first prototype consists of acrylic plastic hemisphere and 3D printed backplate that also serves as a DWM1001 board mount. One ring is used to screw the backplate to the sphere while the other is used for nylon yarn attachment. A small 850 mAh battery is located under the board mount, which is elevated 1cm from the backplate and is screwed onto 3D printed threadings with 3D printed nuts.

In order to create more complex reflections and refractions, both sphere surface and backplate are made highly glossy materials.

After handling this prototype the following conclusions were made: the rings need to be rotated 90 degrees in order to ensure battery opening is pointed upwards, the light is too bright so the hemisphere will be spray painted lightly & the hole for the rope around your neck has to be rotated 90 degrees to

Fig. 21: wearable prototype 1

Fig. 22: DWM1001 board

Fig. 23

wearable prototype 1 on a person

Final Implementation

Wearable

The final design for the wearable consists of of mainly the same elements as the first prototype. Some adjustments were made: the plastic hemisphere has been spraypainted to dim the light of the DWM1001. Furthermore the upper ring has been flipped 90 degrees to make sure that the chain is not obstructed by the pendant. Lastly, during the final calibrations before the exhibition, a change in necklace was proposed. The height of the pendant influences the borders of the artwork significantly. When Clemens (1.90m+) approached the artwork everything went smoothly, yet since the artwork is partially made of silver it could obstruct the signal to the anchor, therefore the chain is now adjustable to change the height of the pendant to make sure everyone uses it at approximately the same height.

Fig. 24: final implementation of wearable

Outlook

Generally speaking, the collaboration with Tatjana and the teachers was a comfortable and fruitful experience. The final product does almost exactly as we intended it to work in our concept phase.

Further additions to the concepts would be to try to add for example a heartbeat sensor in a subtle way, mainly to also attach the experience to the endocrine communicational system. A heartbeat sensor in particular adds to the entire story in the spirit of awareness. From a more intuitive view point we discussed the feeling capacities of the heart and how feeling calm or stressed is often perceived in the heart. From a more scientific view point this also makes sense as cortisol, the stress hormone, acts on the heart muscles to determine a faster or slower pace.

Tatjana has invited us to continue working together. The first aim is to be present at the RESI DIGITAL: NEUE SINNLICHKEIT exhibition at the residenz theater in Munich. A future exhibition in June has also been mentioned, but we are still in discussion about this.

Collaboration

The light is a child of darkness, as sound dwells in silence. In the black of darkness, our senses orient themselves from the outside world inwards. Your own perception becomes your only compass. The night initiates the day, the darkness determines the light.

In this tension, the artist Tatjana Busch designs the architecture of darkness and explores its perception. With her works, she gives great importance to the topic of „space“, dives into the geometry of space, its properties - always on the lookout for new spaces of reflection. This not only creates new physical light locations, but also subjective spaces in the perception of the viewer behind the eyes. This is where the actual work is created. The actual sculpture is modulated there, with the light as consciousness material.

Tatjana Busch is no stranger to the Munich art public. In 2007 she received the Haus der Kunst prize and last year one of her objects was even auctioned off at the PIN Party for art for triple the gallery prize. In her expansive works, Tatjana Busch examines light as a material for consciousness in connection with objects and in the perception of the viewer.

Noah Lokocz

Bachelor computer science

noah.lokocz@tum.de

Hlib Novosolov

Master architecture

glebnovosyolov@gmail.com

Literature

Andrew Reisner, Phillip A. Shaltis, Devin McCombie, H Harry Asada, David S. Warner, Mark A. Warner; Utility of the Photoplethysmogram in Circulatory Monitoring. Anesthesiology 2008; 108:950–958.

Asanowicz, A. (2002). Hybrid Design Environment. Proceedings of the 20th International Conference on Education and Research in Computer Aided Architectural Design in Europe (ECAADe). https://doi. org/10.52842/conf.ecaade.2002.572

Baradaran Rahimi, F., Levy, R. M., & Boyd, J. E. (2018). Hybrid Space: An Emerging Oppor-tunity That Alternative Reality Technologies Offer to the Museums. Space and Culture, 24(1), 83–96. https://doi. org/10.1177/1206331218793065

Cambridge Dictionary | English Dictionary, Translations & Thesaurus. (2023). https://dictionary.cambridge. org/

Doscher, James. „Accelerometer Design and Applications“. https://web.archive.org/ web/20081213001559/http://www.analog.com/en/technical-library/faqs/design-center/faqs/CU_faq_ MEMs/resources/fca.html 13.12.2008,

E. A. Pelaez and E. R. Villegas, „LED power reduction trade-offs for ambulatory pulse oxime-try,“ 2007 29th Annual International Conference of the IEEE, 2007, pp. 2296-2299

Evaluation of an Infrared Distance-Measuring Sensor, Randy Butturini, U.S. Consumer Product Safety CommissionOffice of Hazard Identification and Reduction, 07.06.2014

IGI Global: International Academic Publisher. (n.d.). https://www.igi-global.com/

InformedHealth.org [Internet]. Cologne, Germany: Institute for Quality and Efficiency in Health Care (IQWiG); 2006-. What is an electrocardiogram (ECG)? 2019 Jan 31. Available from: https://www.ncbi. nlm.nih.gov/books/NBK536878/

National Oceanic and Atmospheric Administration(NOAA) Coastal Services Center. 2012. “Lidar 101: AnIntroduction to Lidar Technology, Data, and Applications.”Revised. Charleston, SC: NOAA Coastal Services Center.

NOAA. What is sonar? National Ocean Service website, https://oceanservice.noaa.gov/facts/sonar. html, 26.02.21

P. H. Charlton et al., „Breathing Rate Estimation From the Electrocardiogram and Photople-thysmogram: A Review,“ in IEEE Reviews in Biomedical Engineering, vol. 11, pp. 2-20, 2018

Shelley, Kirk & Shelley, S.. (2001). Pulse Oximeter Waveform: Photoelectric Plethysmography

What Is CogniFit? (n.d.). CogniFit: Brain Training - Mental Health Made Simple. https://www.cognifit. com/

What the FAQ are VR, MR, AR, DR, AV, and HR? – Clive Maxfield. (n.d.). Retrieved November 5, 2022, from https://www.clivemaxfield.com/coolbeans/fundamentals-vr-mr-ar-dr-and-hr/

Zheng Y, Poon CC, Yan BP, Lau JY. Pulse Arrival Time Based Cuff-Less and 24-H Weara-ble Blood Pressure Monitoring and its Diagnostic Value in Hypertension. J Med Syst. 2016 Sep;40(9):195.