"Music to Form Mappings in Product Design"

Music to Form Mappings in Product Design By Carlos Abraham Ornelas Aispuro

Submitted to the program M.Sc. MediaArchitecture at Bauhaus-Universität Weimar

November 2016 Š Carlos Abraham Ornelas Aispuro All rights reserved

2

Music to Form Mappings in Product Design By Carlos Abraham Ornelas Aispuro Submitted to the program M.Sc. MediaArchitecture at Bauhaus-Universität Weimar November 2016 © Carlos Abraham Ornelas Aispuro All rights reserved

Abstract The cognitive basis for the correspondences between music composition and visual representations are generally associated with personal artistic interpretations, making it difficult to predict generalize findings.

To identify systematic basis for music-to-form mappings, I collected from different research Intuitive Strategies (associative bindings, inter-modal analogies and emotional bindings)1.

I examined specific relationships between sound structure of music and meanings related to visual perception of shapes, and I combined them both through an emotional dimension of valence and arousal.

To be more specific a Cup was selected as a product design object to analyze many possible form variations, and then map them into an emotional diagram.

The emotional analysis of musical composition is mainly based on research from Changing Musical Emotion2, and the emotive design grammar of a Cup is based on psychological research on shapes perception and other intuitive design strategies.

Using tools from grasshopper I generated the variables of this emotive design framework, synthesize them with musical features and tested the outcomes of two different songs.

In order to evaluate the first outcomes of the system I choose a Two-alternative forced-choice task method to test if participant were able to perceive a correlation between musical composition and form composition.

First supervisor: Professor Reinhard König Second Supervisor: M.F.A Jason Reizner Advisor: M.Sc. Saskia Kuliga

1

Haverkamp, Michael, Synästhetisches Design – Kreative Produktentwicklung für alle Sinne, 2008.

Livingstone, Muhlberger, Brown, Thompson. Changing Musical Emotion: A computational Rule System for Modifying Score and Performance.

2

3

4

Contents Contents ……………………………………………………………………………………………………………………… 5 Acknowledgements……………………………………………………………………………………………………... 7

CHAPTER 1 INTRODUCTION ………………………………………………………………………………….. 9 1.1. Thesis Overview ……………………………………………………………………………………………………. 11

CHAPTER 2 BACKGROUND ………………………………………………………………………….……….. 12 2.1. Music-visual correspondences in historical examples ….………………….…………………….…... 12 2.2. Audiovisual perception …………………………………………………………………………….………..……. 15 2.3. A model of cross-sensory linkage …………………….……………………………………..……….….……. 17 2.4. Emotion in music ……………………………………………………………………………………..…….………. 18 2.5. Designing emotive shapes ………………………………………………………………………………….……. 19

CHAPTER 3 A LANGUAGE FOR AUDIOVISUAL DESIGN ………………………………..…..….…. 21 3.1. Mapping music to emotions ……………………………………………………………………….….…..….…. 21 3.1.1 Dimensional emotion model …………………….……………………………………………………..…..…. 21 3.1.2 Mapping features to the primary emotional diagram ……………….………………..……….…….. 22 3.1.3 Analyzed musical features …………………………………….………………………………….…….………. 24 3.2. Mapping topological form to emotions………………………………………………………….…..………. 26 3.2.1 Background of the emotive perception of shapes ………………………………………….….………. 26 3.3. Qualitative taxonomy of an emotive Cup …………………………………………….………..….…….…. 30 3.3.1 Topological variations …………………………………………………….………………………………..…….. 31 3.3.2 Intuitive strategies ……………………………………………………………………………………....….…….. 32 3.4. Quantitative taxonomy of an emotive Cup …………………..…………………………..….…….………. 37

5

CHAPTER 4 SYNTHESIZING MUSICAL FEATURES AND FORM …..……..…………………….…. 47 4.1. Mapping musical features to form: An audiovisual design language correlation ……….………. 47 4.2 Two songs and two Cups ………………………………………………………………………………….…………. 53

CHAPTER 5 EVALUATING THE CORRELATION OF TWO SONGS AND TWO CUPS ….….. 56 5.1 Evaluation method …………………………………………………………………………………………………….. 56 5.2 Evaluating quantitative results and discussion …………………………………………………..………….. 59

CHAPTER 6 CONCLUSION ……………………………………………………………………………..…………. 68 CHAPTER 7 REFERENCES & APPENDIX ……………………………………………………………..…..…. 70 7.1 References …………………………………………………………………………………………………..…….…….. 70 7.2 Appendix ………………………………………………………………………………………..………………..………. 72

6

Acknowledgements

I would like to express my gratitude to my supervisor Reinhard Kรถnig for the useful comments, remarks and engagement through the learning process of this master thesis.

Furthermore I would like to thank Saskia Kuliga for guiding me in the topic of emotional evaluation, and Jason Reizner as well for the support on the way.

Also, I like to thank the participants in my survey who have willingly shared their precious time, and many thanks to all my family members and friends without whom I would not be where I am today.

7

8

Chapter 1 Introduction “Musica est exercitium arithmeticae occultum nescientis se numerare animi“ 3 Gottfried Wilhelm Leibniz 1712

Listening to sound, music or spoken language induces visual sensation, which mainly consist of abstract shapes, colors, spatial and motional information4. Thus, Paul Klee, Wassily Kandinsky and Joseph Albers developed interesting theories about music and arts at Bauhaus School in the mid 1920’s.

New results in neurosciences have now aroused the psychological and philosophical discussion on perception and formation of concepts in audiovisual arts & design, and some of this research imply that we share an unconscious understanding of cross-sensory meanings, which unlike genuine Synesthesia5, are similar for most of the human population.

In this same direction a design approach is undertaken by Haverkamp to integrate a multi-sensory perceptual theory, by creating systematic models which allows the linkage of different modalities.

The assimilation of different stimuli in our body creates an emotional reaction, and even though this reactions are strongly diversified6, they could play a unifying role to associate stimuli objects from different senses.

3

German translation „Musik sei eine Zahlenübung des Unterbewussten“, Nerlich, Luise, KLANG tektonik: Entwurfsgrammatik in Architektur und Musik, p. 74 4 Haverkamp, Michael. Look at that Sound! Visual Aspects of Auditory Perception, Granada 2009 5 Rare neurological phenomenon in which stimulation of one sensory or cognitive pathway leads to automatic, involuntary experiences in a second sensory or cognitive pathway, is claimed that only 1 in every 2000 people in the world have synesthesia.

6 Haverkamp, Michael, Visual Representation of Sound and Emotion, IV International Conference Synesthesia: Science and Art, Almeria, 2012, p. 6

9

Figure 1. Haverkamp’s model of multi-sensory perceptual object7

Objects are able to express symbolical meanings through their formal characteristics, and their forms convey a combination of both logical and expressive meanings that we recognize through their overall structure8, in a greater way music can produce a range of emotions and meanings by manipulating musical components. Under these variables designers and musicians take advantage of their intuitive perception, to manipulate the meanings their creations communicate.

To explore this connections, I created a hypothetical design grammar which synthesize perceptual bindings between music and form, and performed an evaluation to expand the discussion about audiovisual linkage in art & design.

7

Haverkamp, Michael, Synästhetisches Design: Kreative Produkentwicklung für alle Sinne (Munich: Hanser, 2008), p. 125 Blijlevens, J., Creusen, M. E., & Schoormans, J. P. (2009). How consumers perceive product appearance: The identification of three product appearance attributes. International Journal of Design, p. 27-35 8

10

1.1. Thesis Overview

This thesis work attempt to investigate correlations between music and form structures taking in consideration intuitive strategies of audiovisual perception, to create a design grammar that would be able to connect both fields systematically.

I will start providing a background into how music and form correspondences are linked in historical examples through different art-design fields, and how audiovisual bindings could be understood nowadays with new models of cross-sensory linkage and perception (Chapter 2).

I will then present a summary of stablished research in music-emotion theory and shape-emotion theory, and propose an emotive deign taxonomy to bind this areas of research together (Chapter 3).

Using this taxonomy as an underlying design framework, I developed components in Grasshopper to connect analyzed data from music to the grammar design variables, enabling a synthesizing method to define connections between music characteristics and design features (Chapter 4).

The effectiveness of the selected intuitive design strategies are evaluated using a Two-alternative forced-choice task method and Fuzzy Logic to test relevance of form characteristics and emotional traits of audiovisual associations (Chapter 5).

The thesis will close with reflections on findings from this evaluations and suggestions for future work, to expand the understanding of intermodal bindings in audiovisual and emotional design (Chapter 6).

Additional support material and references referred in this thesis are included at the end of this document (Chapter 7).

11

Chapter 2 Background The theory about perception and interconnections between visual representations and music have a long tradition, and this couplings have been expressed within a framework of historical knowledge. The following section gives an introduction to how music have conveyed multi-sensory meanings to form, in different artwork fields like paint, sculpture and architecture.

2.1. Music-Visual correspondences in historical examples In antiquity, the Pythagorean theory of harmony was considered the universal standard of musical composition and, in derivation from this, also the standard of architectural design, taking in consideration for example, the arrangement of columns and the dimensional ratio of the groundplan of buildings, to express harmonic patterns through rhythm and proportion.9 In another era, music has written itself into the history of painting in many ways, this interest in intermodal associations could be related from early manifestations around 1800 were musical aesthetics had a strong influence on painters, first as a representation of musical subjects and motifs, then searching for inspiration at the abstract level of general art theory10. Reflections of this nature become more common throughout Europe, reaching a climax in the debate about abstract painting that took place in the 1920’s11.

9

Daniels, Naumann. Audiovisuology: see this sound. p.43 Gottdang, Andrea. Painting and Music, Audiovisuology: see this sound. p.248 11 Gottdang, Andrea. Painting and Music, Audiovisuology: see this sound. p.246 10

12

Figure 1. Wassily Kandinsky, No. III, Die Kleine Welten, 1922, four-color lithograph, UNL-F.M. Hall Collection.12

One of the most famous metaphors in the field of architecture as “frozen music”13 was denominated in the romantic thesis “Philosophie der Kunst” from Friedrich Wilhelm Joseph Schelling (1772-1829), were the space sequence and different harmonic elements in the buildings are perceived as “plastic musicality”.14 Likewise I would like to refer the sculptural work of Heinrich Neugeboren in the 20th century, where he made formal representations on the symphonic works of Johann Sebastian Bach (Figure 2).

Figure 2. 1928 | Neugeboren, Heinrich Fuge_stereometric presentation of Bach, Johann Sebastian Music.15

12

Kennedy, Sharon, Painting music: Rhythm And Movement In Art, University of Nebraska-Lincoln, 2007 The original denomination is German is “erstarrte Musik” 14 Nerlich, Luise, KLANG tektonik: Entwurfsgrammatik in Architektur und Musik, p. 75 15 Nerlich, Luise, KLANG tektonik: Entwurfsgrammatik in Architektur und Musik, p. 111 13

13

In the work of KLANG tektonic, Luise Nerlich mention different scenarios by which architecture and music are connected in a historical context. This transformations could be through Associative Basis, Space-Sound Installations and Mathematic Basis.

The concept of Associative Basis is defined basically as intuitive and subjective interpretations where a musical piece inspires a design base on drawings and visual representations, Space-Sound Installations are based on interactions related to specific acoustics strategies but, also to philosophical background connecting music and space. Transformations on Mathematical Basis are mostly regulated concepts, were the methods laid on mathematical structure analysis.16 Also the design process could be applied in both directions, taking music as an inspirational input for architectural design, and vice versa, architecture inspiring music compositions. One example of the second type is the work of the composer Claus-Streffen Mahnkopf where he created a series of compositions as a tribute for architects.

In the 1950’s two revolutionary artworks based on Mathematic Basis were developed by the architect and composer Iannis Xenakis: On the one hand the composition of Metastaseis in 1953/54 (Figure 5), on the other hand the Philips-Pavilion for the world exhibition in Brussels in 1958 (Figure 3). In the design process for composition and architecture Xenakis worked seemingly sketches about space, in which hyper-paraboloid and conic elements are used to represent a continuous tone course (Figure 4).

Figure 3. Philips-Pavilion in Brussel 1958.17

Figure 4. Axonometric representation of sound paths.18

16

Nerlich, Luise, KLANG tektonik: Entwurfsgrammatik in Architektur und Musik, p. 10,11 Nerlich, Luise, KLANG tektonik: Entwurfsgrammatik in Architektur und Musik, p. 183 18 Nerlich, Luise, KLANG tektonik: Entwurfsgrammatik in Architektur und Musik, p. 217 17

14

Figure 5. Metastaseis Sketch, Draw: Xenakis, Iannis.19

2.2. Audiovisual perception “In contrast to genuine synesthesia, which is consider absolute, multimodal integration is dependent of the context; if one sense provide too little or unclear information, other senses enter in as a corrective.�

Gerhard Daurer

Nowadays, in a media oriented society, the correlations of images and sounds has become inescapable. Through audiovisual technology, not only hearing and seeing, but also the aesthetics, technology, and economy of the visual and the auditory have become connected with one another in multiple ways.

Early manifestations can be found since the mid-eighteenth century, where color organs have represent a kind of pre-history of audiovisual media, they anticipated image-sound effects that later emerge as experimental or innovative uses of audiovisual media, but it was not until 1920 that it was possible to represent images and sound as analog, electrical oscillations; from the

19

Nerlich, Luise, KLANG tektonik: Entwurfsgrammatik in Architektur und Musik, p. 285

15

1960’s as audio-video signals; and from the 1980’s as digital code; with this innovations it also became possible to inter-transform, generate and manipulate images and sound.20 Technical mass media have now enable the conservation and reproduction at will of auditory and visual sensory impressions on film, video and DVD. Nonetheless, live performance is currently experiencing a renaissance: especially in the live visuals found in club culture and media art, celebrating this immediacy more than ever before.21 The synthesis of hearing and seeing over the course of natural evolution and their subsequent cultural conditioning, is an aspect of cultural evolution that is represented by multimodal integration, modifying and enhancing the perceptual capacity of the individual.22 A research approaching this cultural conditioning was performed by Stein and Meredith, where they report an experiment in which the sensitivity of individual multimodal neurons was tested, they were able to show particularly strong effects with coincidental bimodal stimuli23, explaining in part the perception-intensifying effects of combinations of images and sound.

Certain shapes have been shown to evoke similar audiovisual perceptions. Kohler’s ‘maluma’ and ‘takete’ experiment where he showed that the majority of people matched the nonsense word ‘malumba’ to a smooth rounded shape drawing and the word ‘takete’ to a spiky angular shape (Figure 6) – show that simple shape drawings can evoke almost universal cross-sensory perceptions24.

Figure 6. Kohler’s ‘takete’ (spiky shape) and ‘malumba’ (rounded shape) experiment.

Personally I find very interesting a synesthesia25-based theory proposed by Ramachandra and Hubbard, to give a solution to one of the oldest puzzles in psychology of how language evolved.

20

Daniels, Naumann. Audiovisuology: see this sound. p.11 Daniels, Naumann. Audiovisuology: see this sound. p.07 22 Daurer, Gerhard. Audiovisuology: see this sound. p.329 23 Stein and Meredith, The Merging of Senses, MIT Press, 1993 24 Kohler, W. (1947). Gestalt psychology. New York: Liveright.

25 Synesthesia is a curious condition in which an otherwise normal person experiences sensations in one modality when a second modality is stimulated. It’s also a genuine perceptual phenomenon, not an affect based on memory associations from childhood or on vague metaphorical speech.

21

16

In this paper is mentioned that even Chomsky, the founding father of modern linguistics, has expressed the view that, given the complexity of language, it could not have possibly evolved through natural selection. Although in this theory proposal, they suggest the existence of a kind of sensory-to-motor synesthesia, which may have played a pivotal role in the evolution of language.

A familiar example of this is dance, where the rhythm of movements synesthetically mimics the auditory rhythm.26 They also express diffusively, that there may be an extensive cross-wiring between brain regions that represent abstract concepts, which would explain the link between creativity, metaphor and synesthesia (and the higher incidence of synesthesia among artists and poets).27

2.3 A model of Cross-Sensory Linkage

“Only a comprehensive look at the diverse manifestations of multisensory perception… will lead to more profound knowledge and make it possible to use them systematically in design”28 Haverkamp, Michael

During the last two decades, when research on genuine synesthesia experience the extensive nature of this phenomenon, is became a common approach to strictly separate it from other perceptual processes of cross-modal linking. An interesting approach for demonstration of those perceptions is to sketch visual phenomena which are stimulated by sound.29 The cognitive basis for the correspondences between music composition and form are generally associated with personal design interpretations. To identify systematic basis for music-to-form mappings I refer the work of Michael Haverkamp, whose “model of linkage levels” (Figure 7) represents an effort to systematize specific mechanisms of linkage between different sensory modalities.

Basically Haverkamp categorized five different base mechanisms by which stimuli of different modalities are link to our system of perception. The processes involved can occur simultaneously and also influence, complement or restrict one another. Also the processes can be spontaneous

26

Ramachandra, V.S., Hubbard, E.M., Synaesthesia – A Window into Perception, Thought and Language, JCS, 8, N. 12, 2001, p. 3-34.

27 Ramachandra, V.S., Hubbard, E.M., Synaesthesia – A Window into Perception, Thought and Language, JCS, 8, N. 12, 2001, p. 3-34.

28 Haverkamp, Michael, Synästhetisches Design: Kreative Produkentwicklung für alle Sinne (Munich: Hanser, 2008), p 115 29 Haverkamp, Michael. Look at that Sound! Visual Aspects of Auditory Perception, Granada 2009.

17

and genuine (fixed) or spontaneous and context dependent. The three levels in the middle play a crucial role in perception and in the judgment of objects or atmospheres.

The label “intuitive strategies” derives from the fact that these basic mechanisms are particularly well suited to establishing lines between the senses that seems evident to us and can be interpreted easily.30

Figure 7. Model of cross-sensory coupling

2.4 Emotion in music Most of us also listen to music to experience emotions. The specific mechanisms through which music evokes emotions is a rich field of research, with a great number of unanswered questions.

Music has been reported to evoke the full range of human emotion: from sad, nostalgic, and tense, to happy, relaxed, calm, and joyous31.

The capability of musical features to be manipulated in the expression of different basic emotions received considerable interest in the 1990’s. In a study of performance, Gabrielsson (1994, 1995) asked performers to play several known tunes, each with six different emotional intentions and the performers and the performers were found to vary the works’ overall tempo, dynamics, articulation and vibrato in relation to the emotion being expressed.32

30

Daurer, Gerhard. Audiovisuology: see this sound. p.337 Juslin, P. N., & Laukka, P. (2004). Expression, perception, and induction of musical emotions: A review and a questionnaire study of everyday listening. Journal of New Music Research, 33, 217–238.

32 Livingstone, Muhlberger, Brown, Thompson. Changing Musical Emotion: A computational Rule System for Modifying Score and Performance. Computer Music Journal, p.42 31

18

“Apparently it may be a “code” for communication in the structure of music, and this code is common to performers and listeners, with similar acoustic features used when encoding and decoding emotional intentions.”33 For this research, the results and conclusions of Changing Musical Emotion will play an important role on defining the basis to configure musical features in the arousal-valence circumplex, and to develop components to analyze music.

The experiments performed with the system CMERS (Computational Music Emotion Rule System)

by Livingstone and its group, supported their main hypothesis analyzing emotional perception in music. Although the current system is specific to Western classical music, and its effectiveness on other genres is unclear.34

2.5 Designing emotive shapes “A favorite object is a symbol, setting up a positive set of mind, reminder of pleasant memories, or sometimes an expression of one’s self. And this object always have a story, a remembrance, and something that ties us personally to this particular object, this particular thing.”

Donald Norman35

The symbolic qualities of man-made objects and the meaning that they can convey is the study of product semantics, and these symbols can be any information that is perceptible to the human interaction with forms, acting to engage our cognition and emotions. Designers use these symbols by way of “a nonverbal product language as the vehicle for communication”.36 In words of Desmet, this process is described in the following sentence; “through cognitive processes, like interpretation, memory retrieval, and associations, we are able to recognize metaphors, assign personality or other expressive characteristics, and asses the personal or symbolic significance of products”37.

33

Livingstone, Muhlberger, Brown, Thompson. Changing Musical Emotion: A computational Rule System for Modifying Score and Performance. Computer Music Journal, p.42 34 Livingstone, Muhlberger, Brown, Thompson. Changing Musical Emotion: A computational Rule System for Modifying Score and Performance. Computer Music Journal, p.61.

35 Norma, Donald. Emotional Design: why we love (or hate) everyday things. P.06.

36 Krippendorff, K. and R. Butter (1984). "Product Semantics: Exploring the Symbolic Qualities of Form." Innovation Spring 1984, p. 4-9.

37 Desmet, P. M., & Hekkert, P. (2007). Framework of product experience.International Journal of Design, 1(1), 57-66.

19

I am particularly interested in the symbolic characteristics that can convey cross-sensory analogies or emotional bindings to sound and music compositions, which will create another filter to choose specific shape features along this crossed design field.

One of the contributor to the meaning an object can convey is form, which is constructed by a combination of elements; points, lines, planes surfaces and volumes. These elements of form create a design. This process was broke down by Stiny, into a set of rules that define a language of design. The geometry of the shapes and forms, and rules that describe the design in terms of its meaning and function38. Many resulting designs may be created, using this “shape grammar” of geometric special design rules and associative descriptions.

One these initial set of rules for shape grammars and descriptions are created, design algorithms can then combine them to39 “produce an object as a work of art in response to some initial conditions”40.

38

Stiny, G. (1980). Introduction to shape and shape grammars. Environment and planning B, 7(3), 343-351.

39 Mothersill, Phillipa. The Form of Emotive Design. MIT, 2014, p. 16 40 Tiny, G., & Gips, J. (1978). Algorithmic aesthetics: computer models for criticism and design in the arts. Univ of California Press.

20

Chapter 3 A Language for audiovisual design “The audiovisual is now located both in human senses and in things”.

Dieter Daniels and Sandra Naumann “Ähnlich wie eine Sprache, in der komplexe Sachverhalte mit großer Gestaltungsfreiheit durch Sätze aus Wörtern zusammengesetzt werden, werden hierbei für das Entwerfen Regeln aufgestellt… Eine solche reglementierte Methode wird hierbei als Etwurfsgrammatik bezeichnet.“ 41 Nerlich, Luise

Designers carry out almost a synthesis of a wide range of cognitive, emotional and sensory inspirations to create an object whose design attributes symbolize these same experiences. The challenge in this section laid on using intuitive strategies which are similar for a wide human population range, creating thus an inclusive cross-modal system.

3.1. Mapping music to emotions

3.1.1 Dimensional emotion model Dimensional emotion models define emotions based on a set of continuous scales. Common dimensions in theories proposed in this area include valence – indicating the positivity or negativity of the emotion – and arousal – indicating the excited or calm emotional state. Russell’s circumplex model of affect creates a two-dimensional bipolar valence-arousal space, onto which words with different emotional meaning can be mapped (Figure 8).

41

Nerlich, Luise, KLANG tektonik: Entwurfsgrammatik in Architektur und Musik, p. 08

21

Figure 8. Russell’s circumplex model of affect.42

Russell’s dimensional circumplex allows these emotions to be analyzed in the quantitative twodimensional space of valence and arousal, which will provide a useful terminology related to the design of forms.

3.1.2 Mapping music to the primary emotional diagram

In Changing Musical Emotion researchers use the term music-emotion rule to represent the variation and application of a musical feature to bring about a specific change in musical emotion.

A music-emotion rule has a type, a variation and a set of emotional consequences.

An example of a music emotion rule is: “Mode minor = sad, angry.” In this example the rule type is “Mode”, the variation is “minor” and the emotional consequences are “sad”or “angry”.43

42

Russell, James A. A circumplex model of affect. Journal of personality and social psychology 39.6 (1980): 1161-1178 43 Livingstone, Muhlberger, Brown, Thompson. Changing Musical Emotion: A computational Rule System for Modifying Score and Performance. Computer Music Journal, p.44

22

Figure 9. Mapping of structural music-emotion rules for Quadrants 1 and 3 of the 2DES. For example, Tempo fast maps to Quadrant 1 (emotional consequent).44

Figure 10. The set of Primary Music-Emotion Rules mapped on to the 2DES. Adapted from Livingstone and Thompson (2006).45 44

Livingstone, Muhlberger, Brown, Thompson. Changing Musical Emotion: A computational Rule System for Modifying Score and Performance. Computer Music Journal, p.45 45 Livingstone, Muhlberger, Brown, Thompson. Changing Musical Emotion: A computational Rule System for Modifying Score and Performance. Computer Music Journal, p.48

23

Figure 10 displays a relationship between rule variations and their emotional consequences. All ruled variations alternate between the expressions of high versus low arousal, or positive versus negative valence. For example, to express high arousal in music, the tempo should be increased, the articulation should be made more staccato, pitch high raised, and timbral brightness should be increased.46

3.1.3. Analyzed Musical Features The music analysis is performed using algorithms from FFT (Fast Fourier Transform) library tools in Processing, where signals from mp3 files are selected and sent to Grasshopper in order to be analyzed.47 It was not possible to analyzed all characteristics described in the Set of Primary Music-Emotion Rules, nevertheless most of the relevant features were obtained via Processing-Grasshopper or using another music analysis software like FL Studio.

Tempo Is well known as the speed of a musical piece and it could be described as fast or slow. In the western modern music is usually indicated in beats per minute (bpm). Music sequencers use bpm system to denote tempo.

Loudness Is described as the physical strength (amplitude) and is also related to the attribute of auditory sensation, in terms of which sound can be ordered on a scale extending from quiet to loud. This feature was analyzed using FFT libraries in Processing and further sent to Grasshopper, obtaining average loudness of the song and Pitch loudness as well.

Pitch Pitch is a major auditory attribute of musical tones (frequencies) and is the quality that makes possible to rate sounds as “high “or “low” in these sense associated with musical melodies. This attribute was as well analyzed using Processing libraries and components from Grasshopper.

Mode

46

Livingstone, Muhlberger, Brown, Thompson. Changing Musical Emotion: A computational Rule System for Modifying Score and Performance. Computer Music Journal, p.46 47 Research and Grasshopper components to analyze musical features were undertook and developed by Majd Murad, as part of a free project named FFT Chairs: Music & Design, at Bauhaus-Universität Weimar, in winter semester 15/16 for the program M.Sc. MediaArchitecture.

24

Mode generally refer to a type of scale, couple with a set of characteristics melodic behaviors. In the western tradition modes’ is strong associated with valence: major mode is associated with happy and minor mode is associated with sad.48 FL Studio software was used to get this attribute.

Harmony Harmony is a complex concept to understand in music theory, nevertheless it can be generalized as the simultaneous used of pitches (tones, notes) or chords and their construction and chord progression and the principles of connections between them.

In our case the way to describe if harmony is simple or complex is by analyzing the amount of matches of harmonic ratios in music composition, mostly focusing in the musical intervals corresponding with a frequency ratio 3:2, better known as the Perfect fifth. This ratio was analyzed using components in Grasshopper.

The figure below (Figure 11) shows the summary of musical features being analyzed in this project.

Figure 11. Primary Music-Emotion Rules being analyzed in this thesis.

48

Livingstone, Muhlberger, Brown, Thompson. Changing Musical Emotion: A computational Rule System for Modifying Score and Performance. Computer Music Journal, p.42

25

3.2. Mapping topological form to emotions The form of a product can be used to communicate information and important aspect of product design is defining the aesthetic and emotional value of it.

Shapes and their characteristics such as roundness, angularity, simplicity, and complexity have been postulated to affect the emotional responses of human beings in the field of visual arts and psychology. In the following pages, I present some of the research on emotive shape theories and show how a quantitative emotive design framework can be generated within the topological variations of a cup, thus connecting emotions to specific three-dimensional shapes.

3.2.1 Background of the emotive perception of shapes There has been a variety of work examining the relationship between emotions and form.

Research have been carried out to qualitatively formalize our perception of emotive forms by mapping various elements of shape to different expressive adjectives, and this adjectives can be linked to our descriptions of different emotions Poffenberger studied stimulus-response mapping of adjectives to small, medium, and large angular or sinusoidal waves.49 These results were replicated by Collier and mapped onto valencearousal dimensions (Figure 12).

49

Poffenberger, A.T. and Barrows, B.E. 1924. The feeling Value of Lines. Journal of Applied Psychology.

26

Figure 12. Collier’s diagram mapping Poffenberger’s shapes onto the valence-arousal dimensions.50

In this diagram of valence-arousal circumplex, it is suggested that our perception of the affective character of the lines is related to the type of movements we use when expressing the related emotions. Downwards facing lines represent relaxed low-energy emotions such as sadness and trust, while upwards-facing lines express more powerful and uplifting emotions such as anger and joy. Curves with a higher rhythm, i.e. more inflections, show more movement and, therefore, higher energy. Angular curves express a rigidity associated with negative emotions, while the softness of smooth curves are associated to more positive emotive movements.

Recent studies have further endorse the existence of a universal perception of shapes, for example The Sensual Evaluation Instrument provides users with a set off-hand size, ambiguous physical objects meant to afford a channel of emotionally evaluative communication through dialog between user and designer (Figure 13). The creators of the instrument documented consistencies in response of users to shape characteristics across cultural contexts.

50

Collier, G. L. (1996). Affective synesthesia: Extracting emotion space from simple perceptual stimuli. Motivation and Emotion, 20(1), p. 1-32.

27

Figure 13. Three-dimensional shape used by Isbister et al.51

An important reference to understand better the emotional value of shapes, is the summary of design elements proposed by Phillipa Mothersill in The Form of Emotive Design, where the qualitative taxonomy is broken down into the four quadrants of the valence-arousal circumplex: 







Positive, low-energy emotions are represented by top-heavy forms with smooth big and medium curves leaning slightly back or horizontal, and with round, flat and wide aspect ratios.

Positive, high-energy emotions are represented by middle or bottom-heavy forms with smooth or angular medium and small curves leaning forwards or horizontal, and with tall, round slender aspect ratios.

Negative, high-energy emotions are represented by middle or bottom heavy forms with angular small medium curves leaning heavily forwards or backwards, and with tall, round and slender aspect rations.

Negative, low-energy emotions are represented by top-heavy forms with smooth or slightly angular medium and big curves leaning heavily backwards, and with round, flat and wide aspect ratios.52

51

Isbister, K., Höök, K., Laaksolahti, J., & Sharp, M. (2007). The sensual evaluation instrument: Developing a trans-cultural self-report measure of affect International journal of human-computer studies, 65(4), 315-328 52 Mothersill, Phillipa. The Form of Emotive Design. MIT, 2014, p. 31

28

Figure 14. Visualization of design characteristics for primary emotions.53

Figure 14 above suggest a visual representation of each of the qualitative design characteristics, mentioned in the last paragraph.

Furthermore, a relevant aspect to take in consideration for mapping in the arousal-valence circumplex, is Fuzzy logic54, which allows to create variables between polar attributes like positivenegative and active-passive. Figure 15 illustrates clearly fuzzy variations between visual representations rated as less active and more active.

53

Mothersill, Phillipa. The Form of Emotive Design. MIT, 2014, p. 38 Is a form of many-valued logic in which the truth values of variables may be any real number between 0 and 1, considered to be “fuzzy�.

54

29

Figure 15. Graphic associations for activity.55

3.3. Qualitative taxonomy of an emotive Cup

In order to make sense out of this existing emotive shape research into a qualitative taxonomy suitable to be integrated into a parametric design, is important to understand which are the topological variation limits of a cup. Many different characteristics of cup design were analyzed to identify relevant elements or form characteristics, which could be located in the valence-arousal dimensions. In the following section, I will discuss how these descriptive design attributes can be transformed into specific design variables that will contribute to my proposed quantitative taxonomy of emotive Cup.

55

Lindauer, MĂźller, Experimentelle Gestaltung, niggli, 2015, p. 71

30

3.3.1 Topological Variations

Topology is almost the most basic form geometry that distinguishes different kind of geometries between each other, and is also in the kind of transformations that are allowed before you really consider something changed.56

Figure 16. Existing cups located intuitively in a valence-arousal circumplex.

The semantic image analysis of exiting cups was important to start defining the relationships between design attributes and the expressive meanings of topological variations.

56

Bruner, Robert. What is Topology, 2000.

31

I located different types of cups in the valence-arousal diagram as an intuitive task, taking in consideration proportion (height - width), form complexity and texture, to assign emotional traits.

3.3.2 Intuitive strategies As I have mention before, Haverkampf’s model for cross-sensory coupling (Figure 17) will play an important role in the categorization of my design grammar for a cup, in which the intuitive strategies like sematic correlations, concrete associations and cross-sensory analogies will be used as a filter to stablish the basic components of the design grammar.

Gerhard Daurer argued in the model below, that spontaneous contextual strategies have emotional bindings57 that could be mapped in the valence-arousal dimensions, thus allowing a quantitative framework.

Figure 17. Haverkampf’s model for cross-sensory coupling

I extracted the following qualitative shape/form characteristics to map them in a design taxonomy defined by valence and arousal dimensions.

Dimension Ratio (active/passive) The basic topological variation in a form is the dimension, which is defined by height/width ratio, the first abstract correlation according to conceptual form analysis from the title Emotion Gestalten58, is that wide-short forms are associated with; calmness, rest, security and heaviness,

57 58

Daurer, Gerhard. Audiovisuology: see this sound. p. 336 Roth, Saiz. Emotion Gestalten: Methodik und Strategie für Designer. Basel: Birkhäuser, 2014, p. 153

32

adjectives that I relate with passiveness, and slender-tall forms are linked to; ascent, rise, erected, adjectives that associate with activeness. (Figure. 18)

active

passive

Figure 18. Sketch Dimension Ratio

Base/Top Proportion (active/passive) By proportion I mean the relation between the base and the top of the form, in the forms were the top is larger than the bottom, it could express an ascendant tendency (active) and the cases were the base is larger than the top, it could be interpreted as descendent (passive) 59. (Figure. 19)

active

passive Figure 19. Sketch Base/Top Proportion

Axis Tilt (sad [passive/negative] happy [active/positive]) We can comprehend the twisting and bending of a simple sack of flour with a wide range of emotional expressions has human have (Figure 20). In this specific case I relate the form of a cup as a character expressing a feeling, by making a simple modification in the axis. I consider that a short curved axis express a sad feeling, and a long curved axis express the feeling of joy (Figure 21).

The range between sad emotion and happy emotion in the valence-arousal diagram is a diagonal crossing the passive-negative quarter to the active-positive quarter.

59

Mothersill, Phillipa. The Form of Emotive Design. MIT, 2014, p. 37

33

Figure 20. The animated flour sack conveying a range of emotions from “The Illusion of Life: Disney Animation” by Johnston & Thomas, 1995.60

sad

happy Figure 21. Sketch Axis Tilt

Segmentation (active/passive) Thinking about segmentation in a cup, as a possible expressive texture, I’m referring again the figure association made in Emotion Gestalten, taking vertical lines as an active expressive shape and horizontal lines as a passive expressive shape. (Figure 22)

active

passive Figure 22. Sketch Segmentation

60

Mothersill, Phillipa. The Form of Emotive Design. MIT, 2014, p. 11

34

Regular/Irregular (positive/negative) This function as an abstract feature could be applied either to form or to segmentation. Generally regular geometries or textures are linked to positive feelings and Irregular or complex geometries are correlated to negative feelings. (Figure 23)

positive

negative Figure 23. Sketch Regular/Irregular

Summary of form characteristics in a qualitative circumplex.

The break down of these design elements will help me to distill further a taxonomy, usable for a base design grammar. The valence and arousal position of the emotions were overlaid into this diagram, from which a hypothesis of shape, topology and texture could be found. To help visualize this qualitative design elements I integrate them in the diagram below (Figure 24).

35

Figure 24. Qualitative design taxonomy for an emotional cup

36

3.4. Quantitative taxonomy of an emotive cup The hypothetical work described above demonstrates the existence of a tentative universal perception of forms, however, much of the design elements categorizations are qualitative and intuitive.

For this type of taxonomies to be integrated into a parametric model CAD tool, a more quantified, and therefore computational version of this taxonomy is required.

Visual programming components from Grasshopper are a proper tool to create design elements and perform multiple variations, which later could be connected to music data analysis.

Components like gHowl allow to receive data through an IP code, from mp3 analyzed files in Processing, in order to use them as an input for any component.

Furthermore, most of the design elements described in the last chapter are mapped into the valence-arousal dimensions, defining polar limits in form features that goes from positive to negative, or active to passive. As a design strategy I create a continuous deformations from polar associations to define variations in between under the principles of fuzzy logic.

Topological construction The first step was to create an axis to define the minimum-maximum values of height, then a circle radius define the values of width, followed by its location along the axis. Afterwards the circles are scaled using a matrix of data to resemble different variations of a Cup design, and finally the curves are connected through a loft function creating a surface. (Figure 25)

Axis definition

Division

Base circle

Location

Scaling

Surface

Figure 25. Topological geometry construction steps

Segmentation and Deformations After defining the base topological construction, a reconstruction of the geometry through control points is performed to redefine the characteristics of the segments and the types and levels of deformations. The loft will hold a new creation of points along the defined surface, same points that could be added or removed for creating a base segmentation.

37

Following this, I defined components to handle the matrix of points, that allows different types of variables in shifts, displacements and deformations. (Figure 26)

Figure 26. Segmentation and deformations based on control points.

The variables required to generate a three dimensional designs using the process described above, were developed using Grasshopper components from Rhino software. The resulting design were evaluated in order to define ranges between the polar attributes. The resulting selection of modifiers include: Topology Variations (Table 1) 





Dimension Ratio: The dimensions can be determined by changing an slider that control a Ratio of minimum height (80 mm) with maximum width (R 44 mm), and maximum height (120 mm) with minimum width (R 38 mm).

Base/Top Proportion: In this feature I developed two shape options; Option A, defined by a continuous linear deformation and Option B, defined by a parabolic curve. The set of data is created to define variations between a wider Base-Top proportion assigned as 0, and a smaller Base-Top proportion assigned by 5, generating a total of 6 variation for each type.

Axis Tilt (Table 2): As I mentioned before in the qualitative description above, the blend of the axis tilt is defines by emotional features Sad/Happy, crossing diagonally along the valence-arousal diagram (passive/negative to active/positive). For this case the variations are created as a combination between the dimension ratio, and the displacement of the central axis point, therefore causing the bending of the Cup form. 30 variations were generated, establishing value -15 as Sad, 0 as Neutral and 15 as Happy.

Segmentation Variations (Table 1) 

Vertical Segmentation Ratio: This feature defined the proportion of the segments by modifying the number of vertical subdivisions. The minimum values is 5 divisions, were the segmentation aspect remain in a horizontal proportion (passive associations), and the maximum values is 40 divisions, were the segmentation aspect resembles a vertical proportion (active association).

Regularity/Irregularity Variations (Table 3) 

Horizontal Shift: Points located in the section rings are shifted in a “cull sequence” (1,0,1,0) for achieving variations in terms of irregularity. Positive side remains without shift, and the modifications starts from neutral points to the negative side. Variations are developed in rational number from 0.000 (neutral) to 0.300 (Negative).

38







Vertical Shift: Points in the located in the vertical columns are shifted in a “cull sequence” (1,0,1,0) for achieving variations in terms of irregularity. Positive side remains without shift, and the modifications starts from neutral points to the negative side. Variations are developed in rational number from 0.000 (neutral) to 5.000 (Negative).

Horizontal Segment Displacement: Points located in the section rings are displaced opposite to the axis in a vertical “cull sequence” (1,0,1,0) for achieving variations in terms of deformation. Variations are developed in rational number from 1.000 (positive) to 1.200 (Negative).

Vertical Segment Displacement: Points located in the vertical columns are displaced opposite to the axis in a “cull sequence” (1,0,1,0) for achieving variations in terms of deformation. Variations are developed in rational number from 1.000 (positive) to 1.200 (Negative).

Table 1. Topology variations

39

Table 2. Axis Tilt variations according to characterized emotional binding

Table 3. Regularity/Irregularity shift and displacement variations

40

Following the basic geometry construction, a Mesh Surface modifier is applied to create a smooth surface. Sharp edged finish fits well for the active-negative quarter of the arousal-valence dimension, but due some technical difficulties to generate fuzzy variations between a smooth surface and a sharp edged texture, I decided to take it out from this quantitative taxonomy.

The figure below (Figure. 27) shows quantitative form variables across the valence-arousal dimensions.

Figure 27. Quantitative taxonomy of an emotive cup

The following images present possible outcomes from nine equally distributed points of this quantitative taxonomy.

41

Figure 28. Passive-negative possible outcome

Figure 29. Passive-neutral possible outcome

42

Figure 30. Passive-positive possible outcome

Figure 31. Neutral-negative possible outcome

43

Figure 32. Neutral-neutral possible outcome

Figure 33. Neutral-positive possible outcome

44

Figure 34. Active-negative possible outcome

Figure 35. Active-neutral possible outcome

45

Figure 36. Active-positive possible outcome

46

Chapter 4 Synthesizing musical features and form “How is an archimusic created within a data world? Within the computer’s memory resides digitalized microcosm of images, sound and symbols, whose organizations intersects with that inherent in the machines structure forming new dynamic interference patterns.“ 61 Marcos Novak

In the following section I will explain how quantitative data from music analysis and form features are combined together in order to match the emotional bindings from each diagram. Connections are based on the valence-arousal dimensions, defining in most of the cases correlations from passive to active emotions and positive to negative emotions.

4.1. Mapping musical features to form: An audiovisual design correlation There seems to be a broad consensus about the term “musical imagery” as denoting images or associations of musical sound in our minds, there are obviously many opinions on the nature of such images and on their relationship to perception and memory62. Nevertheless the previous emotional diagrams allow us to define a threshold that could be complemented with more findings from psychoacoustics research.

The next diagram (Figure 37) clarify how music features and form characteristics are interconnected.

61 62

Martin, Elizabeth. Architecture as a translation of music. Princeton Architectural Press, 1994.

Godoy R. and Jorgensen H., Musical Imagery, Taylor & Francis, New York, 2001.

47

Figure 37. Connections between music features and form characteristics

Tempo In modern western music tempo is usually indicated in beats per minute, taking the beat as a fundamental unit of temporal structure of music. The amount of beats is a specified fraction of a minute, therefore the greater the number of beats per minute, the smaller the amount of time between successive beats.63 For this project I defined a range between 40 and 150 bpm as maximum and minimum values respectively. Tempo will be directly connected to Dimension Ratio, Base/Top Proportion and Vertical Segmentation Ratio. (Table 4)

63

Miguel A., Bertrand D. and Richard G, Tempo and Beat Estimation of Musical Signals, ENST-GET.

48

Table 4. Tempo synthesized to form modifiers

Loudness Loudness is the subjective measure of perceived sound intensity. This sensation, which is mainly related to sound pressure, allows us to distinguish between loud and soft sounds. Loudness will trigger the Vertical Segment Displacement, thus creating a perception of expansion through intensity. (Table 5)

Table 5. Loudness synthesized to form modifiers

Pitch According to the American National Standard Institute, pitch is the auditory attribute of sound according to which sound can be ordered on a scale from low to high. In this special case, pitch will affect the Horizontal Segment Displacement by defining the location on the vertical axis. By this I mean, that lower frequencies will be mapped to the lower horizontal segments and the higher frequencies will be mapped to the higher horizontal segments. (Table 6) Afterwards, the displacement will be triggered by pitch loudness of the frequencies, categorized in four sections.

49

Table 6. Pitch synthesized to form modifiers

Mode and Harmony Based on the studies of Changing Musical Emotion Mode and Harmony basically define the negative/positive values on the valence dimension.

On the one side Mode will work as a Boolean toggle, depending if the song is defined as Minor (negative valence) or Mayor (positive valence), and in this components will affect the 25% of the input data for Horizontal Shift and Vertical Shift features. (Table 7) On the other side Harmony is more complex and the analysis components can rate from complex (1 unit) to simple (12), therefore this music characteristic will play a bigger role defining the location of the song in the valence axis, with a value 75% of the input data for Horizontal Shift and Vertical Shift features. (Table 8)

Table 7. Mode synthesized to form modifiers

50

Table 8. Harmony synthesized to form modifiers

The following figures shows how musical features and form modifiers are mapped on the valencearousal dimension. (Figure 38 & 39)

Figure 38. Form modifiers mapped on the valence-arousal dimension

51

Figure 39. Musical features mapped on the valence-arousal dimension

52

4.2 Mapping two songs and two cups Bringing all the above research and experimentation together, design for any song can be created.

I analyze two songs with different characteristics in order to test the associations with two different outcomes from the design grammar.

The selected two songs were download from the website Free Music Archive under CC (creative commons) licenses. The first song Can’t let her go by Lobo Loco have a CC BY-NC-SA (AttributionNonCommercial-ShareAlike) license and the second song is Rouge by Raw Stiles with a CC BY-NCND (Attribution-NonCommercial-NoDerivatives) license.

These forms provide a testable output for my hypothesis, and will be evaluated in user studies discussed later in this thesis.

Song: Can’t let her go64 Artist: Lobo Loco Tempo: 60 bpm Average Loudness: 2942 Pitch: [8.7] [22.2] [58.45] [109.58] Mode: Minor Harmony: [Perfect 5th: 4] [Major 3rd: 6]

Figure 40. Form outcome from the song Can’t let her go

64

Online link to listen audio file https://www.youtube.com/watch?v=WifqeSTrPw4

53

Song: Rouge65 Artist: Raw Stile Tempo: 100 bpm Loudness: 7088.2 Pitch: [34] [44.61] [81.09] [186.77] Mode: Major Harmony: [Perfect 5th: 8] [Major 3rd: 4]

Figure 41. Form outcome from the song Rouge

Figure 42 provide a quantitative position of the previous form outcomes on the valence-arousal dimension, locating the first cup on the passive-negative quarter and the second cup on the activepositive quarter.

65

Online link to listen audio file https://www.youtube.com/watch?v=DyZPQ3gDkX4

54

Figure 42. Location of form outcomes in the quantitative valence-arousal dimension

55

Chapter 5 Evaluating the correlation of two songs with two cups The following survey was carried out in order to evaluate if my hypothesis for a quantified design taxonomy is correct and to asses possible trends regarding to; song-form correspondences, relevance of form characteristics, emotional or arouse/valence ratings.

In the next pages, I describe the evaluation method, and propose as well an interpretation and discussion of the results.

5.1. Evaluation method Experiment The aim of this study is to identify if participants are able to perceive a correlation between musical compositions to form composition, and also perceive a connection between form, music and emotions, and improve the design taxonomy hypothesis based on the results.

Hypothesis According to the use of “Intuitive strategies” to define a design grammar of correlation between music and form, participant should intuitively know a specific relationship between two different audio files and two different forms.

This knowledge is defined as well as “spontaneous contextual” so I’m expecting they will be able to achieve this task when they have two elements to compare with two songs, which defines the contextual parameters to correlate.

Method Participants The data collected in this study include the responses of 25 participants from Bauhaus-Universität Weimar community and students from Hochschule für Musik FRANZ LISZT. All participants are students, between 18 to 40 years old and with different backgrounds regarding to nationality.

56

From all participants 64 percent of the respondents are female gender and the other 36 percent are male gender.

Materials -

Audio file 1 (Can’t let her go) located in the passive-negative quarter of the emotional diagram.

Audio file 2 (Rouge) located in the active-positive quarter of the emotional diagram.

Object A (coffee mug) that corresponds with the mapped location active-positive quarter in the emotional diagram.

Object B (coffee mug) that corresponds with the mapped location passive-negative quarter in the emotional diagram.

Procedure

In order to avoid “experimenter bias” and reach objectivity, participants will be alone in a room, following instructions and completing the tasks on a computer screen.

Two-alternative forced choice task Task 01 One audio file-two objects With this first task I’ll know if participants are able match an audio file with an object, just by having one audio file as a reference.

Participants in the forced choice task will be asked to decide whether Audio File 1 match with Object A or Object B. Participants will seat in front of a computer with two Objects on the sides. The trial will begin with the play of the Audio File 1, then the participant will be asked to choose a form that correspond to the abstract composition of the song.

Task 02 Two audio files-two objects With this second task I’ll know if participants are able match two audio files with two objects.

Participants in the forced choice task will be asked to decide whether Audio File 1 or Audio File 2 match with Object A or Object B. The trial will follow by playing the Audio File 2, then the participant will be asked to choose the form that correspond to the abstract composition of each song. With the given instruction to correct the first task, in case they have change their mind.

Task 03 Participants will be asked to rate from 1 to 5, which form characteristics were relevant for their decision in Task 01 and Task 02. Taking 1 as not at all and 5 as very strong.

The following form characteristics will be display; Height/Width Ratio, Straight/Curve Form, Axis Tilt.

Horizontal/Vertical Segmentation, Regular/Irregular Segmentation and Bump Displacement.

57

These characteristics will be complemented with images to make them clear.

Task 04 Participants will be asked to what extend they judge the object’s (Form A and Form B) as: active, passive, positive, negative, aggressive, joyful, calm, melancholic.

Arousal traits are displayed in a scale from 1 to 10, taking 1 as passive and 10 as active and, valence traits are also displayed in a scale from 1 to 10, taking 1 as negative and 10 as positive.

The scale to judge specific emotional traits (aggressive, joyful, calm, melancholic) are from 1 to 5, taking 1 as not at all and 5 as very strong.

Task 05 Participants will be asked to what extend they judge the audio clips (audio clip 01 and audio clip 2) as: active, passive, positive, negative, aggressive, joyful, calm, melancholic.

Arousal traits are displayed in a scale from 1 to 10, taking 1 as passive and 10 as active, and valence traits are also displayed in a scale from 1 to 10, taking 1 as negative and 10 as positive.

The scale to judge specific emotional traits (aggressive, joyful, calm, melancholic) are from 1 to 5, taking 1 as not at all and 5 as very strong.

3D Object “A”

3D Object “B”

58

5.1.1 Evaluating Quantitative Results As described above, the data collected in this study included 25 responses. These values are used in this analysis section to evaluate how effective the quantitative design taxonomy is at creating forms that could be related to specific songs.

In the first task One audio file-two objects, results shows that 80% of the participants relate Sound clip 01 [passive-negative quarter] to Form B [passive-negative quarter] and only the 20% of the participants relate it to Form A [active-positive quarter] (Figure 43).

In the graph below (Figure 44) 66, the number of responses67 which connect Sound clip 01 to Form B is indicated with a red line, and according to binominal statistics calculated for 25 participants, the amount of responses is out of the curve of random chance, therefore I could define this responses as a trend.

Figure 43. Graph of the Results from Task 01, One audio file-two objects.

Figure 44. Graph generated by Binominal distribution calculator, DI Management

66

Graph generated by Binominal distribution calculator, DI Management. http://www.di-mgt.com.au/binomial-calculator.html 67 20 matching responses out of 25 participants.

59

In task 02 Two audio files-two objects, results shows that 84% of the participants relate Sound clip 01 [passive-negative quarter] to Form B [passive-negative quarter] and only the 16% of the participants relate it to Form A [active-positive quarter]. (Figure 45) A change of opinion coupling sound clips and forms between task 01 and Task 02 was recorded only in one of the participant’s responses, possibly indicating on the one hand that the contextual reference of two songs was not relevant to make a choice, or on the other hand, that the participants tried to be consistent in their responses.68

Figure 45. Graph of the Results from Task 02, Two audio files-two objects.

Figure 46. Graph generated by Binominal distribution calculator, DI Management

68

There may be a third reason for the similar outcome in Task 01 and Task 02. In Task 02 there was a note indicating “You can change the decision you made in Task 01, if you think should be corrected”. The purpose of this note was to let participants know that it was acceptable to change their choice in Task 02, about the same question in Task 01.

Nevertheless this was an online survey, where it was possible to scroll back to Task 01, and change the choice that they already did.

60

In order to make sense out of this trend I may consider possible causalities for this results. The first scenario could be that the hypothetical design grammar is at a certain level correct, and the majority of the participants found an association between Sound clip 01 and Form B, and Sound clip 02 and Form A. The second scenario could be that the majority of the participants found an association only between one Sound clip and one Form, and the other Sound clip and Form were matched because they were the left options.

In the following tasks (Task 04 and Task 05) I will analyze more specific treats of 3d Objects and Sound clips separately, to narrow down a possible explanation of the trend found in previous tasks.

Task 03 The main goal for Task 03 is to identify which form characteristics had more relevance for the decision in the Task 01 and Task 02. With this information I could identify specific elements to give priority in further development of the quantitative taxonomy, as well as to expand the understanding of perception about separated characteristic which integrate a unified form.

In this section was necessary to complement the characteristic name with an image to make clear visually, what the name is actually intending to mean.

Rating in the scale from 1 to 5, considering 1 as not relevant and 5 as very relevant, Height/Width Ratio was rated as the most irrelevant feature with an average score of 1.68. The role of this characteristic was to convey visual meaning by coupling wide-short form to passive traits and slender-tall proportion to an active property. Straight/Curve Form had an average score slightly above the medium rank with 3.44, possibly coupling soft sounds with a curve and louder sounds to a straight line.

In the case of Axis Tilt I got an interesting result, where the two major charts shows different picks, unlike the other result in this section. We can see in the graph below (Figure 47) that 37.5% of the participants rated this feature with number 1 (the lowest value), and 33.3 with a relative high value of 4. Possibly indicating a divide opinion, some of them didn’t find any association at all, and others found a relative strong association. The roll of this features was to convey an emotional association by creating a feeling of sadness when the tilt is blend. Nonetheless the average score remains in 2.5, lower than the medium rank.

Figure 47. Relevance graph results of Axis Tilt feature.

61

Horizontal/Vertical Segmentation had a similar rank as Straight/Curve Form remaining in an average of 3.44, conveying vertical lines to active attributes and horizontal lines to passive traits.

The most relevant features where Regular/Irregular Segmentation scoring an average rank of 4.28 and Bump displacement with an average rank of 4.52.

The only feature which is connected to valence traits is Regular/Irregular Segmentation, coupling irregular segmentation to negative traits and regular segmentation to positive properties. Although was rated as the second most important trait, there may be a chance that the image representing this characteristic in the survey, could be related as well to Bump Displacement, influencing thus the perception of this feature relevance.69 The average result of all form characteristics could be appreciated in the graph below (Figure 48).

Figure 48. Average results of form characteristics relevance responses, to relate the 3D objects (A or B) to Sound clips (01 or 02).

Bump displacement was ranked by the participants as the most important feature to make an association between a Sound clip and a Form. This characteristic is strongly related to wave length association, commonly use to represent visually the properties of sound. With this data a further development for the shape grammar could be undertaken by giving priority to features which were rated as more relevant.

Task 04 & 05

Valence and arousal ratings in the Task 4 allowed me to make a referenced location in the circumflex diagram to compare, with the one I used in the quantitative design taxonomy, to define the connection between musical analysis and shape/form grammar.

69

This image could be checked in the Appendix section, page 76

62

The following graphs shows the ultimate location extracted from the ratings performed in this survey, mapping the scales passive to active and negative to positive in the circumflex diagram. Form A and Form B were relocated, and the previous location is shown with a dimmed layer.

Analyzing this result I realize that arousal ratings are much more consistent that valence ratings, the graphs show peaks at the moment to define a form or a sound clip as active or passive, unlike valence, where the graphs shows almost equal bars distributed over many ratings, evidencing an inconsistent pattern. (Figure 49)

Figure 49. Arousal rating results of Form A.

In Figure 50 the location extracted from the survey result indicates that Form A was rated more active and less positive than the hypothetical location from the designed taxonomy, and in the case of Form B, we can see a match in the passive quarter, even though was rated less passive than the designed taxonomy, but regarding to valence rating, it didn’t match at all, therefore being relocated in the positive quarter.

The following graph (Figure 51) shows the valence-arousal ratings of Sound clip 01 and Sound clip 02, positioning the first one in the active-positive quarter, and the second one in the passive-positive quarter.

Afterwards we can compare the similar outcomes in Figure 52 about the ratings from participants on Form A & B and Sound clip 01 and Sound clip 02.

63

Figure 50. Form location extracted from the results of valence and arousal ratings

Figure 51. Sound clip location extracted from the results of valence and arousal ratings

64

Figure 52. Form and Sound clip location extracted from the results of valence and arousal ratings

The survey development was planned to try to avoid influence from one task to another, and taking in consideration that music is much more powerful conveying emotions than form, the task arrangement was structured as follows; Task 01 and 02 to relate form to sound clips, then Task 03 to focus the attention in form characteristics, giving a pause to some emotions being aroused by music. Then Task 04 follows to rate emotionally Form A and Form B, and finally Task 05 to rate emotionally Sound clip 01 and Sound clip 02. Although this considerations doesn’t guarantee the lack of perceptual influence between tasks.70

In this section participants were asked to rate separately the 3d objects and sound clips on specific emotions that corresponds respectively to the four different quarter of the valence-arousal circumplex71.

The graphs below (Figure 53) we can confirm some consistencies in the arousal traits, for example, ratings regarding active emotions (Joyful and Aggressive) are high in Sound clip 02 and Form A, the same ones which were rank as active in the previous analysis and by the quantitative taxonomy. In the same way, ratings on passive emotions (Calm and Melancholic) are high in Sound clip 01 and Form B, previously ranked as passive.

70

71

Another option to avoid perceptual influence between tasks was to create surveys for separate groups.

Russell, James A. A circumplex model of affect. Journal of personality and social psychology 39.6 (1980): 1161-1178

65

Figure 53. Ratings of Forms A-B and sound clip 01-02, on specific emotions

However, ratings on emotions corresponding to valence traits are not consistent between them, for example, Sound clip 01 is rated high as melancholic and also high as calm, but according to the valence circumplex72 melancholic correspond to the negative side of emotions and calm to the positive side, and a similar example can be referred in Sound clip 02 between Joyful and Aggressive emotions. Form A and Form B shows as well the same inconsistencies in valence ratings.

Inconsistencies can be found also between this specific emotion analysis and the valence analysis.

As a first example we can see that Form A was rated as positive in the valence rating, but was also rated in a greater extend as aggressive, an emotion that corresponds to the negative side of the

72

Russell, James A. A circumplex model of affect. Journal of personality and social psychology 39.6 (1980): 1161-1178

66

circumplex. Furthermore, Sound clip 01 was rated high as melancholic, and emotion linked to the negative side in the circumplex, but was also rated as positive according to valence traits.

Personally I think the problematic issue about valence ratings is that, positive and negative traits could be associated not only to specific emotions, but also to other polar concepts like pleasant/unpleasant or good/bad. Thus, we are able to perceive negative emotions like sadness as pleasant, making difficult to map linearly traits in two opposite poles.

Despite the inconclusive results on valence dimension, is possible to use arousal results to confirm a causality for the identified trends in Task 01 and Task 02. Most of the participants were able to related respectively two different songs with two different cups, by making abstract emotional associations of activeness and passiveness.

67

Chapter 6 Conclusion

“We never see the same thing when we also hear; we don’t hear the same thing when we see as well�.

Chion, Audio-Vision. Sound on Screen, XXV.

In this document a presented a look to emotional analysis of music and also a compendium of visual references to show the expressive meaning of form. Later on I used this findings to define a cross-sensory design grammar. One of the founding assumptions of this research is that, there is a common perceptual understanding of music and form, which can be used to connect them systematically.

In the introduction and background I discussed historical references which linked, in one way or another music with art/design fields, and also findings on audiovisual perception which were complemented with a model to integrate a design process.

Furthermore I described the process to analyze valence and arousal responses in music, however the technological capacity to quantify emotional values from music is a complex research field, which is currently under development. Then I defined a qualitative design grammar for a cup and based on this elaborated a parametric design process.

The outcome of two analyzed songs were 3d printed in order to perform an evaluation of the system. Analyzing quantitative result, I was able to identify a trend about people relating respectively two specific songs with two specific cups, and the emotional rating allowed me to identify that they were capable to associate arousal traits to perform their choice. However valence ratings were very inconclusive, therefore, better design strategies to convey emotions and to analyze them should be integrated.

68

Evaluating emotions is a complex task to measure, is possible to find similarities in different tasks, however, is impossible to exactly compare feelings between individuals73.

The complexity of the research lays on multiple cognitive and emotional associations between music-emotions, form-emotions and finally music-form, making difficult to track mapping failures to improve the system, although some improvements can be achieved after analyzing results on form characteristics relevance.

Music is experienced over time, while form is expressed as a solid abstraction. One dismissed approach in this process was a linear sequence to represent experience over time, the reason for this was because there is too much information in music, and is very difficult to express it through form, especially using a cup surface as a canvas.

The design proposal developed in this research is just one possible way to map music to form out of other multiple options. Nevertheless, form is not the only aesthetic quality to convey associations in objects, design attributes such as texture, material and color play an important role in our perception as well.

This research has hopefully presented an attempt to connect systematically music with product design by embracing qualitative design knowledge into a more quantifiable design process.

73

Haverkamp, Michael, Visual Representation of Sound and Emotion, IV International Conference Synesthesia: Science and Art, Almeria, 2012, p. 6

69

Chapter 7 References & Appendix 7.1 References Achiche, Ahmed. Mapping shape geometry and emotions using fuzzy logic. ResearchGate, 2008.

Blijlevens, J., Creusen, M. E., & Schoormans, J. P. (2009). How consumers perceive product appearance: The identification of three product appearance attributes. International Journal of Design.

Bruner, Robert. What is Topology, 2000.

Collier, G. L. (1996). Affective synesthesia: Extracting emotion space from simple perceptual stimuli.

Motivation and Emotion, 20(1).

Daniels, Naumann. Audiovisuology: see this sound. Köln: König, 2015.

Desmet, P. M., & Hekkert, P. (2007). Framework of product experience.International Journal of Design.

Dieter, Naumann. Audiovisuology: See this sound. Köln: König 2015.

Godoy R. and Jorgensen H., Musical Imagery, Taylor & Francis, New York, 2001.

Haverkamp, Michael, Synästhetisches Design – Kreative Produktentwicklung für alle Sinne, 2008.

Haverkamp, Michael, Visual Representation of Sound and Emotion, IV International Conference Synesthesia: Science and Art, Almeria, 2012.

Haverkamp, Michael. Look at that Sound! Visual Aspects of Auditory Perception, Granada 2009.

Hensel, Menges, Weinstock. Techniques and Technologies in Morphogenetic Design. Helen Castle, 2006.

Isbister, K., Höök, K., Laaksolahti, J., & Sharp, M. (2007). The sensual evaluation instrument: Developing a trans-cultural self-report measure of affect International journal of human-computer studies, 65(4).

Juslin, P. N., & Laukka, P. (2004). Expression, perception, and induction of musical emotions: A review and a questionnaire study of everyday listening. Journal of New Music Research.

Kalogerakis, Chaudhuri, Koller, Koltun. A Probabilistic Model for Component-Based Shape Synthesis.

Stanford University.

Kennedy, Sharon, Painting music: Rhythm And Movement In Art, University of Nebraska-Lincoln, 2007.

70

Kohler, W. (1947). Gestalt psychology. New York: Liveright.

Krippendorff, K. and R. Butter (1984). "Product Semantics: Exploring the Symbolic Qualities of Form."

Innovation Spring 1984.

Lindauer, Müller, Experimentelle Gestaltung, niggli, 2015.

Livingstone, Muhlberger, Brown, Thompson. Changing Musical Emotion: A computational Rule System for Modifying Score and Performance. Computer Music Journal, pp.41-65, MIT Press, 2010.

Lu, Suryanarayan, Adams, Li, Newman, Wang. On Shape and the Computability of Emotions. The Pennsylvania State University.

Martin, Elizabeth. Architecture as a translation of music. Princeton Architectural Press, 1994.

Maurer, Pathman, Mondloch. The shape of boubas: sound-shape correspondences in toddlers and adults.

The Authors, Journal compilation, 2006.

Miguel A., Bertrand D. and Richard G, Tempo and Beat Estimation of Musical Signals, ENST-GET. Mothersill, Phillipa. The Form of Emotive Design. MIT, 2014.

Nerlich, Luise. KLANG tektonik: Entwurfsgrammatik in Architektur und Musik. Bauhaus Universität Weimar, 2012.

Norma, Donald. Emotional Design: why we love (or hate) everyday things. New York: Basic Books, 2005.

Poffenberger, A.T. and Barrows, B.E. 1924. The feeling Value of Lines. Journal of Applied Psychology Ramachandran, Hubbard. Synaesthesia-A Window Into Perception, Thought and Language. San Diego, Journal of Consciousness Studies, 2001.

Roth, Saiz. Emotion Gestalten: Methodik und Strategie fur Designer. Basel: Birkhäuser, 2014.

Russell, James A. A circumplex model of affect. Journal of personality and social psychology 39.6 (1980).

Stava, Pirk, Kratt, Chen, Mech, Deussen, Benes. Inverse Procedural Modeling of Trees. Adobe Systems Inc., USA, University of Konstanz, Germany, Shenzhen Institut of Advanced Technology, China, Purdue University, USA, 2014.

Stein and Meredith, The Merging of Senses, MIT Press, 1993.

Stiny, G. (1980). Introduction to shape and shape grammars. Environment and planning B, 7(3).

Stiny, G., & Gips, J. (1978). Algorithmic aesthetics: computer models for criticism and design in the arts. Univ of California Press.

Woolman Matt. Seeing Sound: Vom Groove der Buchstaben un der Vision vom Klang. Mainz: Schmidt 2000.

71

7.2 Appendix

7.2.1 Online survey, Music to Form Mappings in Product Design 7.2.2 Online survey results, Music to Form Mappings in Product Design

72

7.2.1 Online survey, Music to Form Mappings in Product Design

73

74

75

76

77

78

79

7.2.2 Online survey results, Music to Form Mappings in Product Design

80

81

82

83

84

85

86

87

88

89

90