Playful Complexities - Vol.1

perception

The Prelude Introduction

evolutionary computation 1.1 Of loops and flows 1.2 The body and its environment 1.3 Designing our own realities 1.4 The beauty in indeterminacy

In a world dominated by the ocular sense, can intuitive interactions (consciously or Interactive Music-Making

2

2.2 Homeostasis 2.3 Autopoiesis 2.4 Alvin Lucier’s Feedback Systems 2.5 Processing auditory information through information design 2.6 Curating music through self-expression

Survival of the fittest

3 4 5 6

interactive music systems

4 6 7 8

3.1 The birth of Genetic Algorithms 3.2 Workflow of a GA 3.2.1 The process of GA’s 3.2.2 The Genetic Algorithm puzzle 3.3 System Optimisation 3.4 Interactive Evolutionary Computation 3.4.1 GenJam 3.5 Tackling User Fatigue

The body as a medium to curate sound 4.1 Liquid Identities: A fluid exoskeleton 4.1.1 Initial approaches 4.2 Prototyping 4.3 Building the IEC of the prototype 4.4 Real-time interaction with an audience

Inferences 5.1 Visual Feedback 5.2 Physiological Data 5.3 Learning through Exploring 5.4 Understanding Motives

non-consciously) curate a sound-driven environment?

11 12 13 14 16 17

The first volume of Playful Complexities is a guide for interaction designers and artists who are new to the field of music-making, in pursuit of using it as a major aspect of their design process. It aims to find solutions through technology to make their end-users interact with a sonic environment. Positioning music in an interactive medium, it investigates the technologies that can be incorporated to

19 20 21 22 23 24 25 25

allow sensory interactions between the body and a wearable to create an emergent sonic ecosystem.

‘Liquid identities [influx]’ (Interactive Architecture Lab, 2021) is an exploration of an emerging soundscape that is modified by an individual as one explores a multisensory environment. Through playful interactions with technology, participants

26

are encouraged to explore their own bodies as mediums to alter the elements 29 29 30 30 32 33

35 36 36 37 37

Towards playful complexities

39

Bibliography | List of Figures

40 | 41

of a soundscape. This embodiment facilitates them to be part of a bigger whole, using an abstract tool (wearable) to devise this sonic aggregate together. As the first prototype, a liquid wearable composed of vinyl fabric and ink was used as a communicating agent, resulting in an emergent soundscape.

playlists

1

sonic emergence

1 3

Using this prototype as a starting point, vol.1 will try to determine the type of interactions that will facilitate the participant to become a performer through physical movements and curate music based on them. Using the man-machine-environment relationship and Interactive Evolutionary Computation as theoretical standpoints, it will also branch out to determine whether there is an ‘optimal solution’ for interactive systems in the highly subjective field of music.

the.prelude 1

As one of the most significant forms of media in our everyday lives, music has the ability to create a profound impact on us. With a wide range of genres and ever-changing technology, the techniques of making music are constantly evolving. Whether one is a professional musician or simply enjoys making music on occasion, creating our own composition instils a sense of fulfillment within us.

Using music becomes an integral part of our process as creatives by adding a distinctive persona to our work. As an amateur musician, I was inspired to write this thesis to provide a first-hand reference to other interaction designers and artists who want to incorporate interactive music in their work.

With an initial approach of finding solutions to amalgamate musicmaking and interaction, this guidebook investigates how users would be able to interact with a sonic environment to produce music through

introduction

evolutionary processes. Describing each segment as a playlist, it begins by attempting to map human learning processes, which are subsequently applied to theories of interactive music-making.

Frequently seeking examples from unique ways of creating music, it dives deeper into understanding how a basic systemic process may result in distinctive outputs to build an interactive link between the user and the interface.

It follows the journey of studying self-sufficient systems and the development of a prototype with my teammate Simon Volt for our project “Liquid Identities [influx]”, where we evaluated the potential

playlist 1

of creating music tailored to users’ interests based on their physical movements.

Echoing Simon’s interest in self-expressing one’s own identity, the idea to generate user-centric music became an area of interest that was considered through exploring techniques previously deployed in feedback mechanisms to develop a relationship between human interaction and sonic outputs.

Playful Complexities [Vol.1]

3

1.1 Of loops and flows We adapt and learn to live in the environment around us as humans. According to Hunt, Hermann and Pauletto (2004), infants investigate each object given to them by experimenting with numerous ways to interact with it. They throw it around, feel its texture, and drop it to see what happens next. This inquisitiveness leads to a better grasp of the feedback they receive because of performing a specific activity. As their contacts with diverse scenarios become more frequent, they begin to recognise the many types of interactions they have with the numerous activities they participate in, and they progressively learn how to fulfill each one. As individuals, we tend to gain knowledge of a set of rules and processes in this way.

While most of our daily lives involve conscious decision-making, some processes transport the mind into a disembodied state. Mihaly Csikszentmihalyi terms this form of disconnection as “flow” (Csikszentmihalyi, 2000), where the brain switches off for a moment and forgets what it was doing since the actions happen non-consciously. He also elaborates by explaining that such instances are more prominent in children when they are more freely able to interact with the world without any formal approach to it, and these moments start to become lesser in adult life. As described by Hunt, Hermann and Pauletto in their research on interactive sonification systems (2004), most musicians on stage slip into this disembodied phase and often find themselves suddenly conscious of what notes their fingers are playing amidst a performance, bringing it to a standstill.

With such instances being rare encounters as adults, we mostly face situations which require us to make a sound decision. While some choices in making these decisions may give us more liberty, some are meant to be followed robustly to perform a specific task. This often categorises our lifestyles to be constantly inside a control loop, which forces us to follow a formal procedure to acquire desired results.

Similarly, traditional musical notations also follow the notion of control loops with rules and procedures that have been set out to write scores, i.e. the human brain needs to understand how to produce a specific melody by following a predefined set of information. The composer

Fig. 1

Trad

Music itional

ns

Notatio

Musicians, however, tend to enjoy re-interpreting pieces that have already been written by others with their own flair. While improvisation may occur with or without a predefined composition, what if there is a possibility to create a sonic environment where the users can improvise the score in their own ways? Further, what if they could create compositions that are personal to their liking, based on other forms of information (unlike standard notation) that do not have any customary rules?

Does the music composer lose complete control over the final piece or give up a certain amount of control to offer a freedom of expression? Are we in or out of the control loop in this case?

herein dictates the way a musical piece should be played.

4

Playful Complexities [Vol.1]

5

1.2 The body and its environment While navigating through decision-making, the human body also actively responds to

Brain responding to stimuli based on previous experiences

the environment it is placed in and derives its own perception of it. Placing a similar

Perception

experience in two different environments hugely impacts the way it is perceived. Using a personal anecdote as a reference, the experience of listening to a live recorded version of Coldplay’s Yellow (Martin, Buckland, Berryman, and Champion, 2000, track 2) at home held power to invoke a sense of emotion amongst its listeners. While

Human body collecting sensory information from its immediate environment

Renewing Perception

listening to this alone, my association with this song was related to the bittersweet nostalgia of love.

Fig.2 Human Perception: a self-renewing process (created by the author, 2021)

A decade later, watching the band perform live and singing along with the crowd elicited a deeper emotional connection based on the audience’s collective response. The mind always recalled the affective experience during the concert (a

1.3 Designing our own realities

prior experience) whenever the song was played afterward, producing a renewed

Bringing back an interesting discussion from my previous study of perception of audio-

association with the musical piece.

visual environments (Tadinada, 2021), French philosopher Maurice Merleau-Ponty in his book Phenomenology of Perception (1945), deals with the ideology of an objective and

“Our perceptual system is an ongoing learning system which is constantly evolving from previous foundation, that is, instead of auditory perception “here and now”, it is rather “until here and now”.

experienced world. Sensing, as opposed to knowing, according to him, is a

“living communication with the world that makes it present to us as the

(Pickles, 2016)

familiar place of our life” (Merleau-Ponty, 1945).

Furthermore, the medium via which it is conveyed through loudspeakers versus

Expressing his understanding about it, he describes that while a specific sensation seems

portable home speakers (or any other source) made a difference. The way we perceive music, which is extremely subjective and dependent on a person’s preferences, may also be influenced by the physical environment and medium in which it is propagated.

direct and obvious, it is often mistaken to be as straightforward. To develop multi-sensory responses to external stimuli, the physiological and psychological elements of the human body interact and influence each other. Merleau-Ponty also backs the Gestalt psychology

As designers, it becomes very important to define an environment the work is

that describes perception as a complex process that cannot be easily simplified as the

placed in, catering to the type of response the music tries to create through intuitive

sum of its parts. It is rather the relationship between these parts and the influence of our

interactions.

own experiences that helps the brain interpret sensory inputs to form a certain response.

We hence try to create our own realities of the phenomenon we perceive based on our experiences (Tadinada, 2021).

6

Playful Complexities [Vol.1]

7

1.4 The beauty in indeterminacy Because human perception is a work-in-progress that evolves as it encounters newer experiences, the focus of this study shifts towards the discovery of wholly new results to further understand how users would react to such interactions.

Can users’ decisions yield results that provide a probabilistic outcome rather than a predetermined one? Can designers create outcomes that are inherently unpredictable, even by them?

John Cage, a pioneer in experimental and avant-garde music becomes a prominent example as a practitioner of aleatoric music (music based on chance). His interests stretched far beyond music, one of which was towards creating new possibilities using chance and indeterminacy. He also ventured into experimental ways of creating such musical pieces.

In his work Music of Changes (composed in 1951 for his friend David Tudor), Cage

Fig.3 The 64 Hexagrams, used to determine chance in I Ching: The Chinese Book of

borrowed techniques from I Ching: The Book of Changes (1950) – a Chinese manuscript

Changes

that explores the concept of chance as a primary theme and attempts to discover a structure in it, using a divination system. One of the principal charts was made utilising 64 hexagrams (shown in Fig. 3).

What if users had the ability to craft their outcomes along the lines of indeterminacy?

To determine pitch, length, and other factors of these musical notations published as 4 volumes (1951), Cage was heavily influenced by the idea of indeterminacy from the book

8

and flung coins onto a set of musical charts for them to make a decision. He then created

Placing interaction in the region of chance provides users the autonomy to

a musical piece with standard notations, utilising these charts as his guide. This marked

compose their own music and challenges the systems-thinking capabilities

the creation of music that had no preset technique and was reliant on probability, which

of the designer. Moreover, it becomes critical to comprehend how interactive

is beyond human control.

music systems are created to build a system as intricate as this one.

Playful Complexities [Vol.1]

9

Music-making has always involved a close association between physical, visual, and technological tools, from the manufacturing of the instrument to musical notations. Music as a medium of interaction has witnessed a significant advance in the typologies of approaches, of which Interactive Music systems are an area of relevance.

Interactive music systems, as defined by Agostino DiScipio, are

interactive music-making

“computational tools that are capable of reacting in real-time upon changes in external conditions” (DiScipio, 2003).

For designers who yearn to create interaction through music, the external changes would have to be created by agents - the users or performers, who facilitate in providing the type of data required to perform a specific task. Throwing light on the feedback loop in interactive system design, he divides this process into four segments.

According to DiScipio (2003), an ‘agent’ firstly initiates the process by creating an external change. Secondly, it communicates this through a ‘control device’ (any form of physical interaction with the computer to perform a specific task). Thirdly, this task creates

playlist 2

a sonic output that is perceived by a user. Lastly, another response is awaited from the control device to repeat the process. This way, interaction develops its definition of creating a relationship between the user and the interface since the state of the computer is dependent on the changes made by the user and vice versa.

Playful Complexities [Vol.1]

11

control device

2.2 Autopoiesis The emphasis from homeostasis moved towards a new concept: autopoiesis during the 1960s. This time, the system didn’t burn out but kept regenerating itself.

“agent” or performer

In their book Autopoiesis and Cognition, Maturana and Varela (1972) describe an autopoietic machine to consist of a series of processes involving the transformation and destruction of its constituent elements. This machine continually renews itself and has system output (sound)

the ability to comprehend the processes that created it.

Homeostasis

Fig.4 DiScipio’s system of Feedback Loops in Interactive Systems (DiScipio, 2003 - recreated by the author, 2021)

Interactive Music Systems, although well defined by Agostino Di Scipio have had a history of how they came into existence, which can briefly be summarised into the establishment

Autopoeisis

of Homeostatic and Autopoietic systems to begin with.

2.1 Homeostasis Katherine Hayles (1994) defines Homeostasis as a principle by which all organisms thrive as closed-loop systems, repeatedly inspecting and revising their current state. During the 1940s, the concept encouraged the rise of ‘circuit-based electronic instruments’ by Bebe and Louis Barron, and Herbert Brun (Dunbar-Hester, 2010). These machines were

Lifespan

Regeneration

designed to function until their lifespan, i.e., until they completely burnt out. The era later inclined towards incorporating some amount of intelligence into them by introducing a

Fig. 5 Homeostatic vs. Autopoietic Systems (created by the author, 2021)

basic set of rules for the composition. These rules would have to be appropriate enough to produce a rational musical outcome.

Described by Hayles as reflexivity (1994), these systems also considered the observer as their core element of the system, as opposed to the myopic approach of Homeostasis

However, the notion of devices only functioning until their life cycle moved towards

which involved external sources observing the functionality of it. This leads to valuing

understanding how systems could be incorporated to introduce an interaction between

the concept of feedback mechanisms in a system.

man and machine without the interference of the latter’s lifespan.

12

Playful Complexities [Vol.1]

13

2.3 Alvin Lucier’s Feedback Systems Alvin Lucier is another imminent musician whose techniques have been crucial in his contributions towards feedback systems in sonic interactions. Retracing a comment by a Philosopher of Science Andrew Pickering, his work also mainly emphasises on

“real-time feedback coupling between performer and performance” and “the decentring of the self into a technological apparatus” (Pickering, 2010).

Fig.6 Alvin Lucier’s I am Sitting in a room at the Museum of Modern Art in December 2014.

Interested in the varying outputs of a musical piece, he based his work on the interaction between three main elements - body, machine, and the surroundings.

As explained in his narration, this recorded voice was replayed

I am Sitting in a room (Lucier, 1969) is his most straightforward yet striking work

through speakers and subtly modified by the acoustics of the

of efficiently using feedback systems. Using microphones, he used his own voice

environment, with each repetition further abstracting it. As the

to be recorded and replayed into a room with the following narrative, which also

depth of the voice grew fainter, and a new sonic landscape began

describes the work:

to emerge - a feedback system between the propagated and

“I am sitting in a room different from the one you are in now. I am recording the sound of my speaking voice and I am going to play it back into the room again and again until the resonant frequencies of the room reinforce themselves so that any semblance of my speech, with perhaps the exception of rhythm, is destroyed. What you will hear, then, are the natural resonant frequencies of the

produced sound was created. In this case, a brand-new sonic output was generated from a bizarrely dissimilar input, giving the physical elements of the space a major role in crafting the soundscape. The experimental approach of producing unknown outcomes through basic interaction stands out in his work.

room articulated by speech. I regard this activity not so much as a demonstration of a physical fact, but more as a way to smooth out any irregularities my speech

Alvin Lucier also comments on his colleague Cage’s work

might have.” (Lucier, 1969)

mentioning that “they have nothing to do with self-expression but to do more with new discoveries” (Lucier, 2012). In this case,

In such situations, one is outside the control loop. While the user contributes to

the user’s preference takes a backseat whilst the machine is given

providing data to begin an iteration, there is no control over what is produced since

more power over the outcome of the musical piece. The idea

the non-human elements of the environment take over the fabrication of novel

of giving cues to yield completely novel outcomes becomes an

sonic landscapes. There is a decision being taken, but there is no concrete output

interesting theme to craft a sublime interaction between the user

that led to the decision.

and the interface. Furthermore, are there more unique ways of creating music?

14

Playful Complexities [Vol.1]

15

2.4 Processing auditory information through Information Design

2.5 Curating music through self-expression

In order to explore such ways of interpreting musical notations, it becomes essential to think

Wearables of all types are heading towards becoming a personalised means of communication

about how auditory information can be processed through other forms of media and design.

through which we may express ourselves. Imogen Heap, a technological innovator, and artist

Cornelius Cardew’s “Treatise” (Cardew, 1967) is a score that is not scripted as musical notations

in collaboration with a group of engineers created a pair of MIDI gloves, also known as Mi.Mu

but as a set of graphics to be inferred by a musician.

Gloves (2010) that serve as musical instruments to generate music through the gestures and movements of the user (Heap, 2013). The major aspect of this wearable is to give the user

“Just as no two interpretations of a graphic can be the same, no two performances of a

the leverage to improvise music based on the gestures of their hands. The gloves, which

graphic score can be the same. If the musicians are true to the score, each performance

work over wifi connectivity are comprised of flex and motion sensors that allow the system to

would feature different notes but would share a similar form and composition.”

anticipate the “bend in the fingers” to produce sonic outputs. A location-based algorithm also

(Forrester, 2019).

facilitates them to become the designers of their own space through visual movements. These gestures must be programmed to play specific notes and melodies that the user can select in advance using a software that maps musical notations to these gloves. Vibration motors are also embedded in the gloves to provide haptic feedback to the user, depicting change.

As a transformative way of making music, Heap (2013) tries to introduce the association between musical creations with gestures and movements of the body. This wearable also provides a direct association between self-expression and the curation of music.

Fig.7 Page 41 and 183 depicting music scores in Treatise (1967)

Defying the conventions of sheet music, no instructions or rules were provided except graphical symbols, which gave the authority to the person reading it to interpret it in their own way.

Anyhow, Treatise is still a composition - it is written in the format that abstractly still reminds the reader of one. Although a line runs through the centre of each page, it is altered intelligently with the use of symbols, reminding the reader of something familiar. Perception of these symbols is the basis of the creation of a composition. This work demonstrates a radical shift in the way musicians think about the linguistics of sound, which has predominantly always been

“In granting the performer authority over their body, space, and time, the audience becomes an active participant in the experience of live electronic musical performance through integrative communication techniques” (Forde Thompson, Graham, Russo, 2005) This example brings us to the possibility of devising a wearable devoid of any rules of how musical outputs are generated. Combining ideas from processing aural information through mediums other than regular musical notations, can self-expression through bodily movements generate new outcomes without establishing any formal rules of how to use the wearable?

notations. It challenges the ordinary and visually urges the reader to creatively implement novel ways and decipher it. Treatise is a ‘familiar unfamiliarity’ that serves as a great representation to look at scores differently.

Fig.8 (background) The Mi.Mu Gloves (2013)

16

Playful Complexities [Vol.1]

17

The decisions we make in our day-to-day lives are not set in stone. Most of the ones we make are under our control, and our preferences are usually determined by what best fits our situation. Similarly, the evolution of all living things also works on a selection process.

A natural selection, as explained by Charles Darwin and Alfred Russell Wallace (Darwin & Wallace, 1858), can be abridged into some basic ideologies. They explain that evolution takes place only when there are individuals that have the capacity to replicate themselves. Once this replication begins, it is not perfect - which leads to a certain degree of disparity in them. As a result of these variations between individuals, some of them possess the quality to have an advantage over others in

survival of the fittest

terms of survival.

Generation A

Generation B

Evolution

playlist 3

Fig.9 Replication to evolution: the survivors (created by the author, 2021)

These ‘survivors’ are selected based on their edge over others, which depends on a ‘fitness value’. This value certifies them as individuals suitable enough to move on to the next generation (Takagi, 2001). The process of evolution hence cements the concept of self-generation to create better individuals with each iteration.

Playful Complexities [Vol.1]

19

In the Oxford Handbook of Computer Music (McCormack, Eldrige, Dorin, and Mcilwain, 2009), terms such as genotype, substrate, and phenotype were utilised to explain a generative process, drawing inspiration from biology. According to it, the genotype specifies a process through a certain set of rules or instructions, which is then performed by a mechanism (for example, the user in this case). A substrate, as defined in their research, plays the role of maintaining the current state of a system, which can be modified by the mechanism (the user). The changes made by the mechanism to the substrate generate phenotypes, which are also often referred to as ‘offspring’. This phenotype is the final output which is perceived by the user. A graphic elaborating this generative process has been recreated as fig.10 below.

Genotype

Phenotype Mechanism Substrate

User/Audience

Artist

Fig.11 Biomorphs from Dawkins’ The Blind Watchmaker (1986) Creation

Generative Process

Generation

Experience and Feedback

Fig.10 A simple generative process (recreated by the author, 2021)

3.1 The birth of Genetic Algorithms

3.2 Workflow of a Genetic Algorithm Each genetic algorithm, as explained by J.H. Holland (1975), begins with a set of genotypes that are part of a larger generation (for instance, Generation A). The primary

This theory of evolution led to the advent of genetic algorithms. A Genetic Algorithm

idea behind a GA is that individuals from this population with more adaptation are

(referred to as a GA henceforth) is simply an optimization strategy that mimics a

picked to survive with a higher chance each time. After one iteration, these solution

selection process similar to natural selection in the evolution of living beings. Richard

possibilities (phenotypes) become the population of the following iteration (Generation

Dawkins, in his book The Blind Watchmaker (Dawkins, 1986) was one of the earliest

B). This process continues until an optimal solution (explained in page 22) is found.

practitioners to use a genetic algorithm with human intervention. Inspired by Darwin’s evolution as his model, he created a set of programs that would let users generate

Genetic Algorithms are thus employed to develop the most optimum ‘solution

characters like insects, also known as biomorphs (Dawkins, 1986), and choose their

candidates’, through a process which is explained further in the next section.

offspring with the features that were preferred by the user. Biomorphs stand out as a compelling example of how genetic algorithms can be used to determine results through a user-selected procedure.

20

Playful Complexities [Vol.1]

21

3.2.2 The Genetic Algorithm Puzzle

3.2.1 The process of GA’s

Print this page out to create your own set of preferential numbers!

Puzzle: Fill in the boxes with numbers to create a preferential generative algorithm.

Initialistaion

This involves the selection of a random population from a generation, generally set within a certain range of limits (an upper and lower limit).

Select 8 different numbers of preference between 1-30 and fill them into each blob

Evaluation

This stage comprises of deciding a “fitness value” for the iteration. The

Z

fitness value is the most essential criterion for selection of individuals since it decides the survival capacity of individuals.

Y

If you had to select 2 numbers from each set, make a choice of how to filter them. (even numbers, prime numbers, your favourite numbers,etc.)

X

Selection

This step involves the selection of best set of individuals from the population based on the fitness value to move ahead. Fitter individuals move on to the next generation.

Crossover Fill in the two selected numbers in each box respectively corresponding to the one they previously belonged in

Crossover is the stage wherein there is a selection of the best set of individuals from the population based on the fitness value. Fitter individuals move on to the next generation.

Next, fill the numbers based on the color coding from the previous step

Mutation

This step creates new individuals by combining individuals selected from the previous generation. The gene of this “offspring” is usually a combination of the parent genes.

Add/subtract 1 this to the number Fill them below.

Termination

This step terminates the algorithm when an optimum solution is obtained. A solution candidate may be considered to have reached at its “optimum” when the difference in evolution and

A new set of numbers based on an individual preference has been created

mutation of offsprings from different generations starts to become negligible.

22

Fig.12 Puzzle (created by the author, 2021)

Playful Complexities [Vol.1]

23

3.3 System Optimisation

3.4 Interactive Evolutionary Computation

Our musical preferences, like all our other preferences, may vary and evolve

From his research on using various systems to create an interactive music system,

throughout time. Even more so, they might be influenced by how we are feeling at

Takagi (2001) introduces the concept of Interactive Evolutionary Computation

a particular time of day. We all have different ways of waking up to an alarm. Some

(referred to as IEC henceforth) as

people prefer to wake up to peaceful music first thing in the morning, even if they prefer to listen to techno music later in the night at a club. Some people prefer loud music to wake them up, but they might not like music at such high frequencies later during the day. As a result, when it comes to music, preference for a certain genre

“an optimization method that adopts evolutionary computation (EC) among system optimization based on subjective human evaluation.”

(Takagi, 2001).

is very subjective, with many variables beyond a designer’s control. Even though improvements in technology have given us the ability to create various types of

This method is simply a replication of an Evolutionary Computation technique, where

music, it is still vital to understand user preferences when it is employed as a form

a quantitative fitness value is substituted with a human user. The process allows

of interaction. Since preferences in response to music may be dynamic, a robust

preferential selection of music that is tailor-made for users based on their liking.

system needs to be inculcated which would be able to change efficiently along with

The added advantage of a composition evolving in real-time is often preferred over

this response. According to Hideyuki Takagi (Takagi, 2001), there are two ways to

one predefined selection, giving them a choice to change their mind between the

optimise any type of system. They can either be quantitative (numerically defined) or

duration of the piece as well. To tackle user-fatigue and personalise the experience

subjectively evaluated.

of music-making, IEC lets users obtain favourable outputs from interactive systems.

Quantitative

Subjective

Initialisation

Evaluation

Selection

Crossover

Mutation

Termination

Fig.14 Subjective Evaluation through user intervention (created by the author, 2021) Evaluation

Fig.13 The two types of evaluation (created by the author, 2021)

3.4.1 GenJam

Genetic algorithms can easily be used to perform a task with the former evaluation i.e., using a quantitative method to derive outputs. While evaluating interactive

GenJam, created by J. Biles (1994), is a music generator that is built on the principles

music systems, Hideyuki Takagi (2001) argues that using such evaluation makes it

of Interactive Evolutionary Computation, which takes in chord progressions and

impossible to produce favourable results and thus pushes towards using human

generates jazz melodies. In this example, the listener chooses the fitness value by

sensory reception as an input.

selecting good and bad keys when the music is played. The results from this were then combined to create an Audio CD (Biles, 1995) to create a jazz music piece. Apart

24

Additionally, choosing just one preference may not be the ultimate solution for an

from the recording, it was also available to be played in real-time. Since too much

interactive system. When the users begin to interact with the soundscape for a longer

user-fatigue is generated when the listener needs to concentrate for longer periods

period of time, they experience ‘user-fatigue’, a term that is used to describe the

of time, GenJam allows the user to subjectively evaluate the musical piece in chunks

weariness when one undergoes a subjective evaluation repetitively (Takagi, 2001).

of note-sequences instead of evaluating to an entire melody at once.

Playful Complexities [Vol.1]

25

3.5 Tackling User Fatigue Using evolutionary algorithms in IEC, however, does produce user-fatigue due to the repetitiveness of the task, no matter how easy the designer makes it for the users to evaluate. While explaining the concept of user-fatigue, Takagi (2001) also mentions that an IEC often encounters its users getting weary easily due to constant subjective evaluation.

This exhaustion, as explained through various references by Fukomoto (2010) in his research, has practically been reduced by using the following strategies, alongside using Evolutionary Algorithms: Visualising the output of IEC: According to Hayashida and Takagi (2002), visualising the output as a landscape allowed users to clearly understand the changes being made by them. A visual landscape (as used by them) served as visual cues for the users during the iterations of the IEC. Using Physiological Information: Based on their conclusions (Hayashida and Takagi, 2002), gathering physiological data to calculate fitness values also helps detect the psychological changes in the user and monitors the user-fatigue to significantly reduce it throughout the runtime of the algorithm.

If user-fatigue decreases, solution candidates would keep renewing themselves even more based on constant user evaluation to produce more optimised results each time. It now becomes crucial to understand what an ‘optimum’ is, for which an interaction with users becomes essential.

26

Playful Complexities [Vol.1]

27

Inspired by the evolution of using systems-thinking in relation to the manmachine-environment concept, investigating the body as a medium to curate music becomes an important aspect of this study. Deriving data from physical movements to generate varying results and curating the sonic ecosystem is further explored in my studio project ‘Liquid Identities’ [influx], in collaboration with Simon Volt at the Interactive Architecture Lab, University College London (2021). The design for the sonic interaction took major inspiration from interactive music techniques, as elaborated in the previous chapters.

the body as a medium to curate sound

4.1 Influx: A fluid Exoskeleton ‘Liquid identities [influx]’ (Interactive Architecture Lab, 2021) is an exploration of self-expression through an emerging soundscape that adapts to an everchanging ‘liquid’ wearable. The playful interactions between technology, sound, materiality, and the body create a multi-sensory experience and push for unique ways of embodied collaboration between the users and the sonic outputs generated by their physical movements. The goal of this project is to make the wearable an autopoietic system, based on the physical movements of the user.

Revisiting the question posed earlier, the project tried to scrutinise whether music could be produced through information processes that were constructed by human intervention. If music were to be produced, can it suit

playlist 4

the user’s taste and preferences, in this case, their own ways of expressing themselves?

One of the solutions from studying previous works is the use of Interactive Evolutionary Computation. The movements of the body were thus used as an indication of ‘preference’ for the IEC of this interactive wearable.

Playful Complexities [Vol.1]

29

4.1.1 Initial Approaches The main goal was to reduce user fatigue while this exoskeleton was being worn by a single person. To collect data from users, several sensor choices were investigated for the wearable, including the usage of heartbeat sensors and brainwave data. The prototype would have a very brief interaction period for users to experience during an exhibition it was going to be showcased at, which would make it really challenging to derive inferences from the data obtained. As a result, the emphasis shifted away from physiological inputs and inclined more towards understanding how users interact with the wearable as a first step.

4.2 The Prototype The prototype of the wearable was made of vinyl fabric with free-flowing ink inside it. The liquid was allowed to follow its own way through numerous pockets and channels to flow freely inside the sleeve based on how the user moves (as indicated in fig.15). Photoresistors were implanted inside the layer beneath this liquid wearable, which collected the brightness values of it based on the presence of ink around them. These photoresistors were attached to a micro-controller (Arduino Nano) for data inputs.

The main goal was to draw inspiration from concepts of chance and indeterminacy of John Cage and Alvin Lucier’s works, giving authority to external factors to influence human intervention. Adding a subtle twist to Lucier’s use of physical qualities in I am Sitting in a room (1969), the concept of the prototype was designed as an accessory that can be worn to freely express one’s own identity. While some would prefer to wear it on their arms, some preferred it as a bodysuit and some as a neckpiece. Hence, the physical appearance of it would change based on the shape and position of the wearable on the body. This would in turn refashion the location of the photoresistors embedded inside it. The prototype tries to

fitness value of the wearable:

provide the physical elements of the wearable (in Lucier’s work, the architectural quality of

cumulative brightness < 100

the space) with a significance to initiate a feedback mechanism. Using Cardew’s Treatise as a metaphor for information design, the users would have to devise their own ways to navigate through these pockets of ink to lower the brightness of the photoresistors, with no rules of how the ink needs to reach a specific target.

The brightness of these photoresistors was considered a major criterion to evaluate user preference. For instance, in the prototyping stage, the cumulative brightness of all resistors reaching a maximum quantitative value of 100 was considered to be the fitness value.

30

Fig.15 The liquid exoskeleton (photograph by the author, 2021)

Playful Complexities [Vol.1]

31

4.3 Designing the IEC for the prototype

4.4 Real-time interaction with an audience

Inferring conclusions from previous examples and devising a physical

This concept of IEC was exhibited at The Ars Electronica

approach to obtain data inputs from the user, Liquid Identities [influx]

Festival in September 2021 (Ars Electronica London

uses the concept of selecting an initial population of 2 sets consisting

Garden), to observe whether the methodology from

of 8 integer values each, ranging between 1 and 52 (upper and lower

previous studies works with real-time feedback from

limits). The numbers were then scaled to musical notations and run

the audience. Fifty visitors who attended this exhibition

through the iterations of a basic IEC. Taking cues from GenJam, the

Fig.17.1 Prototype A: interactive sheet (image by the author, 2021)

agreed to try this prototype, and each participant was

project also tested an 8-note and 16-note sequence for the user

allowed to use the prototype individually. Although

to evaluate instead of assessing an entire melody at once, thus

the physical quality of the space where the wearable

significantly reducing ‘user fatigue’.

was positioned could not be altered, feedback from participants regarding what they had felt experiencing the prototype was noted.

Max MSP was used as the software for this GA, and the inputs received from the users were then programmed to be generated in real-time, receiving live data from an Arduino Nano microcontroller. The fitness

Two prototypes were created for this exhibition:

value was set to all the photoresistors’ cumulative brightness to be

Prototype A (fig 17.1) was an interactive liquid sheet.

100 or lesser, i.e., all the photoresistor pockets have ink running on a

In this prototype, the wearable was laid out as a two-

layer above them.

dimensional sheet, which allowed the users to navigate the liquid towards different photoresistors to understand Wearable

the fluid characteristic of this wearable. Prototype B (fig

the control device devide

17.2) was a wearable sleeve, like the conceptual ideas used

Ableton

Processing the notes received from MAX MSP to desired music type

for the project. Since the audience interaction was less

Arduino Nano

than 60 seconds long, the 8-note sequence was chosen to

Collecting Real-time data from user fitness value evaluation

demonstrate this algorithm. The live data obtained from

MAX MSP

the users was updated every 5 seconds.

Creating the IEC

Fig.16 Software used to build IEC (created by the author, 2021)

This exhibition was divided into two parts, each of which corresponded to a different visitor. For Group 1, the When the IEC was designed, no optimum solution candidate was

audience was told how a basic IEC works and given a

considered, leading to an interesting question whether the system

clear knowledge of the fitness value, which states that if

might perform successfully with merely a shift in direction based on

the brightness levels of all the photoresistors add up to a

the users’ non-conscious feelings.

maximum numerical value of 100, a sequence is accepted as the fittest individual to move forward. In Group 2, the

The produced melodies were then processed in Ableton to generate

process was more non-conscious. Users were allowed

a sonic output.

to explore the sheet/wear the prototype without having any knowledge of a complex outcome of the resulting Fig.17.2 Protoype B: the wearable (image by the author, 2021)

32

melodies.

Playful Complexities [Vol.1]

33

The exhibition served as a platform for feedback and learning to develop the prototype further.

On a

generalised note, visitors of Group 1 were initially driven towards harmonising the melody, while Group 2 was trying to devise their own techniques of movement to improve it.

Although the aural response to their movements was a success, some users from Group 1 struggled to understand how they were creating this music even after the process was explained to them. Since every user may not be a keen listener, an important inference from this prototype was that the users were looking for another cue to inform them of their actions. As most music is usually predefined,

inferences

they found it difficult to familiarize themselves with the fact that they were composing and monitoring the values of these sequences in real-time. Hence, as described above (Chapter 3, page 26), a visualised output of the IEC in terms of change in characteristics of the liquid (for example, colour, viscosity, direction of flow, etc) may be suitable to provide a visual prompt to its user that a change has occurred.

All users agreed to be enjoying control over the music they were generating, as opposed to responding to a piece of music that was already pre-determined by the

playlist 5

designer. In addition to this, a significant observation with regards to the movement of the users interacting with the wearable and the note sequence was made.

Some interactions from the exhibition can be viewed by clicking here.

Playful Complexities [Vol.1]

35

Although the movements determined the fitness value of the algorithm, the wearable was unintentionally designed in such a fashion that users were also required to move

5.3 Learning through Exploring

their arms a certain way to let the ink shut all the photoresistors off. In this case, they

Most of the users from Group 2 began to create their

were non-consciously moving only in a distinct direction to define the fitness value of

own rules from their brief learning of using prototype

the melody - a phenomenon where the system takes over the choreography of the

A – the interactive music sheet. They presumed some

user’s body movements.

areas of the wearable corresponded with high and low notes. When the IEC produced certain high or low notes,

However, little amount of user-fatigue was observed even for such a short duration,

they drew their own conclusions that this was because of

which corresponds to definitely implementing the techniques explained in Chapter

letting the ink flow through a specific region on the sheet.

4, in the future.

This way, they primed themselves to believe that they

5.1 Visual Feedback

had comprehended the mechanism of the prototype.

Fig. 18.1

Simultaneously, it was observed that some users liked the notion of just creating melodies through their movements

Providing a visual tool of communication to the users also unravels the possibility

with prototype B - the wearable. They underwent a sense

of exploring lighting elements around this emergent music generator. Since all the

of accomplishment of having control over this completely

sensory data is presently obtained from the photoresistors embedded inside the

novel object that was being solely dictated by them.

wearable, the results also push the project towards understanding the environment it could be set in. As further research, it aims to try and create similar scenarios with lighting techniques that adapt to the data obtained from the wearable and map the differences between the two outcomes. For instance, the initial tests could be as

5.4 Understanding Motives

simple as using different RGB values of light in response to the movements of the

Despite the fact that one of the groups was aware of a

user, which would also prime them to move based on the colours they produce.

general IEC’s goals, neither of them looked forward to

5.2 Physiological Data

Fig. 18.2

finding the optimum solution after a brief interaction with the prototypes. They seemed to enjoy improvising note sequences more than getting closer to an optimal

If the duration of the interaction were to be significantly increased, it also opens

outcome. This marks a triumph in the attempt to create

doors to a system that could benefit from adding an element of physiological data

an IEC with no optimum. As a result, the question whether

inputs such as heart rate, to enhance the experience of this interaction. It would be

an optimal solution is necessary for an interactive system

interesting to further practically understand how internal processes can amplify the

in a subjective discipline like music arises.

quality of preferential outcomes.

In addition to observations during the exhibition, two important questions were posed ahead of it to understand user feedback better – how the users learned to interact with the prototype and whether they understood the motive and intentions

Fig. 18.3 Fig. 18.1, 18.2, 18.3 Arm Movements to try and create different routes for the ink to flow. (images by the author, 2021)

of the interaction.

36

Playful Complexities [Vol.1]

37

Although there are numerous techniques to create interactive music systems, using music as a form of media becomes highly subjective due to human intervention. For interaction designers seeking to incorporate music as a parameter in their work, Interactive Evolutionary Computation provides a gateway to efficiently fabricate melodies suited to the user’s preference. One of the benefits of utilising such an algorithm is the element of surprise that comes with each new iteration. While users believe they are in control of the system, the beauty of this method is that the designer nevertheless becomes the chief composer of the piece by developing the foundation of the system itself. Even though the progressions are unpredictable, Interactive Evolutionary Computation’s true triumph is to give users a feeling of ownership over their creations.

towards playful complexities

“When music is primitively compelling, it forces its way into the conscious; when music is structured in such a way that engages analytical and predictive processes, it invites itself into the conscious.” (Keane, 1989)

After constructive feedback from the users during their interaction with the prototypes, this research arrives at a conclusion that there does not need to be an optimum solution in artistic creativity. Through intuitive interactions and adaptive learning mechanisms, it can also be deduced that users try to determine their own rules for the way a system works, trying to acclimatize their actions based on varying perceptions to unpredictable results. While some may try to decipher the workflow of

playlist 6

the system, some will just submit to their accomplishment of producing an outcome navigated by them. In both cases, there is a learning curve that is purely based on their understanding of it, devoid of any boundaries to the possibilities of using it.

With each ‘familiar unfamiliarity’ that they encounter, users participate as vital agents in closing the control loop in unique ways, advancing towards manifesting their own playful complexities.

Playful Complexities [Vol.1]

39

Biles, J.A (1994). GenJam: A Genetic Algorithm for Generating Jazz Solos.

Biles, J.A (1995). Al Biles virtual quintet, genjam. [Compact disk DRK-144].

Cage, J (1951). Music of Changes, for piano. Peters Edition EP 6256, EP 6257, EP 6258, EP 6259 (4 volumes).

Cardew, C (1967). Treatise Edition EP 7560.

Fig.1 Bent, I. D. (2021). musical notation. In Encyclopedia Britannica. [Online Image] [Accessed 10th September 2021] https://www.britannica.com/art/musical-notation

Csikszentmihalyi, M (2000) Beyond Boredom and Anxiety: Experiencing Flow in Work and Play, reprint, Jossey Bass Wiley

Fig.2 Author’s Illustration. 2021. London.

Darwin, C. R. & Wallace, A. R. 1858. On the tendency of species to form varieties, and on the perpetuation of varieties and species by natural means of selection. Journal of the Proceedings of the Linnean Society of London, Zoology, 3(9): 45-62.

Fig.3 Heng, D. M. (2020, June 9). How to use the Yijing (I-Ching) for better Decision-Making? ILLUMINATION. [Online Image] [Accessed 20th September 2021] https://medium.com/illumination/how-to-use-the-yijing-i-ching-for-better-decision-making-67e3f1e2cf08

Dawkins, R. (1986). The blind watchmaker: Why the evidence of evolution reveals a universe without design. WW. Norton & Co. Di Scipio, A. (2003). ‘Sound is the interface’: from interactive to ecosystemic signal processing. Organised Sound, 8(3), 269-277. Cambridge: Cambridge University Press.

Fig.4 Re-illustrated by the author. 2021. London

Dunbar-Hester, C. (2010). Listening to cybernetics: Music, machines, and nervous systems, 1950-1980. Science, Technology, & Human Values 35(1), 113-139.

Fig.5 Author’s Illustration. 2021. London.

Eldridge, A. C. (2002) Adaptive systems music: Musical structures from algorithmic process. In C. Soddu (Ed.), Proceedings of the 6th Generative Art Conference, Politecnico di Milano University, Milan, Italy.

Fig.6 Amanda Lucier (2015). Alivn Lucier records I am Sitting in a room at the Museum of Modern Art, New York [Accessed 12th July 2021] https://www.moma.org/wp/inside_out/wp-contentuploads/2015/01LucierMoMA_122014_005.jpg

Forrester,J. (2019) https://medium.com/nightingale/treatise-a-visual-symphony-of-information-design-2ced33ef01a0 Access Date: 6 July, 2021 Fukumoto, M. (2010). Interactive Evolutionary Computation utilizing subjective evaluation and physiological information as evaluation value.

2874 - 2879. 10.1109/ICSMC.2010.5641932.

Fig.7 Cardew, C (1967). Treatise Edition EP 7560. page41, page 183 [Accessed 27 July 2021]

Hayashida,N & Takagi,H. (2002) “Acceleration of EC convergence with landscape visualization and human intervention”. Applied Soft Computing.vol. 1.

Fig.8 Unknown (2014) Mi.Mu gloves [Online Image] [Accessed 10th September 2021] https://mimugloves.com/gloves/ Fig.9 Author’s Illustration. 2021. London.

Heap, I. (2013, January 11). Imogen Heap Performance with Musical Gloves Demo. Lecture presented at Wired 2012. https://www.youtube.com/watch?v=6btFObRRD9k, accessed 10 August, 2021.

Fig.10 Di Scipio, A. (2003). ‘Sound is the interface’: from interactive to ecosystemic signal processing. Organised Sound, 8(3), 269-277.

Cambridge: Cambridge University Press

Holland, J. H. (1975). Adaptation in natural and artificial systems: An introductory analysis with applications to biology, control, and artificial intelligence. U Michigan Press.

(Image recreated by the author, 2021)

Hunt, A & Hermann, T. & Pauletto, S. (2004). Interacting with sonification systems: closing the loop. 879 - 884. 10.1109/IV.2004.1320244.

Fig.11 Dawkins, R. (1986). The blind watchmaker: Why the evidence of evolution reveals a universe without design. WW. Norton & Co.

Koga, Shinpei & Inoue, Takafumi & Fukumoto, Makoto. (2013). A Proposal for Intervention by User in Interactive Genetic Algorithm for Creation of Music Melody. IEEJ Transactions on Electronics, Information and Systems.

Fig.12 Author’s Illustration. 2021. London.

Lucier, A. (2012) Music 109, Notes on Experimental Music. Middletown, CT: Wesleyan University Press.

Maturana, Humberto; Varela, Francisco (1980) [1972]. Autopoiesis and Cognition: The Realization of the Living (2 ed.).

McCormack, J & Eldrige, A & Dorin, A & Mcilwain,P. (2009). The Oxford handbook of computer music. In Dean, R. T. (Ed.), Generative Algorithms for Making Music.

Merleau-Ponty, M. (1945) Phenomenology of Perception - accessed 1 December 2020.

Pickering, A. (2010). The cybernetic brain: Sketches of another future. Chicago: University of Chicago Press.

Pickles, D. (2016) Cybernetics in Music. Unpublished PhD Thesis. Coventry: Coventry University, accessed 16 July 2021.

Roth, G., & Schwegler, H. (1981). Self-organizing systems : an interdisciplinary approach.

Takagi, H. (2001). Interactive evolutionary computation: Fusion of the capabilities of EC optimization and human evaluation. Proceedings of the IEEE. 89.

Tadinada, P. (2021). Maneuvering through Perception in Audio-Visual Installations. Contextural Theory Term 1. University College London.

Thompson, W., Graham, P. & Russo, F. (2005). Seeing music performance: Visual influences on perception and experience . , 2005(156), 203227. https://doi.org/10.1515/semi.2005.2005.156.203 Wilhelm, R., Baynes, C. F., & Bollingen Foundation Collection (Library of Congress). (1950). The I ching: Or, Book of changes. New York: Pantheon Books.

40

Fig.13 Author’s Illustration. 2021. London.

bibliography

Martin, C & Buckland, J & Berryman, G & Champion, W. (2000). Yellow. [Recorded by Coldplay]. On AM [MP3 File]. Rockfield (Monmoth, Wales): Parolophone

Fig.14 Author’s Illustration. 2021. London.

Fig.15 Tadinada.P, Volt.S (2021) [Original Photographs] Fig.16 Author’s Illustration. 2021. London.

Fig.17.1 Tadinada.P, Volt.S (2021) [Original Photographs] Fig 17.2 Tadinada.P, Volt.S (2021) [Original Photographs] Fig. 18.1, 18.2, 18.3 Tadinada.P, Volt.S (2021) [Original Photographs] Chapter Cover Pages and Cover Page illustrated by the author, 2021, London.

list.of.figures 41