01
Artist Statement During the last decades the scientific field of artificial intelligence has undergone a major change in direction. The general cognitive paradigm of constructing an artificial humanoid thinking machine was becoming more and more problematic. It turned out that creating an intelligent robot capable of communication is not just a matter of computational power. As a direct answer to this the 90s saw the rise of a promising and remarkable research domain in artificial intelligence called affective computing. In this field, originally founded by Prof. Dr. R. Picard at MIT, researchers are studying innovative emotional man-machine interactions. From these efforts a new breed of humanoid robots including the KISMET (first generation) and Nexi (second generation) was developed at MIT. The groundbreaking vision affective computing offers has also had a profound influence on the field of computational and cognitive musicology. Specifically it has renewed interest in the connections between music and emotions. Since the tentative studies of K. Hevner in the 1930s this connection could only be thought of in terms of classical psychology. But by applying the field of affective computing a whole new perspective opens up. This point of view lies at the foundation of the construction of the EMO-Synth. In implementing the system numerous new principles and ideas have also come to
02
surface. Two seminal ideas in this context are the following: + The EMO-Synth can be seen as a brand-new virtual tool by which the artist can extend and complement his own creative process. By using the system it becomes possible for the user to understand his own personal creativity on a new level. + By using the system a totally new virtual platform arises in which the boundaries between artist, audience and the generated artifacts are redefined. To what level are these artifacts a creation ofthe artist or the audience? To which extent can we speak of artistic input when the EMO-Synth truly participates in the creative process? Of course the last two statements start from a certain perspective on the artistic process. It is my personal point of view that art communicates through human emotions. Every work of art implies a certain kind of expression that can be understood intuitively rather than rationally. It is this intuitive and emotionally loaded component that the EMO-Synth tries to uncover and
simulate. The realization of the EMO-Synth project also implied a radical break with the old notion of static art. Doing this is not a new idea but it was only through recent developments on both the methodological and technical level that the implementation of systems like the EMO-Synth becomes possible. Recent artificial intelligence techniques and techniques stemming from affective computing give a whole new insight into both the esthetic or emotional content of art objects. In developing the EMO-Synth I feel it as my personal goal that the project will redefine and reshape part of the whole new discussion about the place of automatically generated art in the general artistic discourse. Valery Vermeulen Producer & Founder of the EMO-Synth
03
EMO-Synth The EMO-Synth is an interactive multimedia system where the emotional manmachine interaction plays a central role. During performances and demonstrations the system automatically generates and manipulates sounds and images to direct the user in certain predefined emotional states. The development of the EMO-Synth relies on different techniques stemming from a broad range of domains including artificial intelligence, affective computing, psychophysiology, automatic music composition and sound and image generation. In the current setup emotions are quantitatively measured using biosensors that monitor psychophysiological parameters such as heart rate (ECG or electrocardiogram), stress (GSR or galvanic skin response) and electrooculogram (EOG). During operation the EMO-Synth will subsequently generate audiovisual artifacts and analyze the resulting emotional impact on the user. Using techniques stemming from artificial intelligence like interactive genetic programming and HMM’s the EMO-Synth will try to generate audiovisuals to bring the user in a specific emotional state.
04
The concept of the EMO-Synth is based on the link between the apparent rational logical and creative process. Recent technological and scientific experiments have provided mounting evidence of this link. As a consequence more and more opportunities arise for the artist to use scientific systems in his own creative process. I first came into contact with these ideas during a four-year research project I undertook between 2001 and 2005 at the Institute for Psychoacoustics and Electronic Music (IPEM, University of Ghent). These were a primary inspiration to start the EMO-Synth project.
05
Prototypes Between 2004 and 2007 the first four prototypes of the EMO-Synth were developed to explore the possibilities of the system and to design possible blueprints for future developments. Prototype 01 was developed for the multi media project SKANNER (collaboration with Belgian artist Angelo Vermeulen) that mainly dealt with the concept of human fear. The project was shown on various locations including the Atomium (expo Interface, Brussels, Belgium, 2004), W139 (Amsterdam, Netherlands, on invitation of Belgian artist Jan Fabre, 2004), Beursschouwburg (Subliminal Festival, Brussels, Belgium, 2005), STUK (Artefact festival and Kulturama 2005, Leuven, Belgium, 2005), Z33 (expo Feel the young, Hasselt, Belgium, 2004). With this prototype first experiments were elaborated in the field of the sonification of the psychophysiological data. Prototype 02 was developed in 2006 for the exposition PERSEVERANCE at the Godshuis in Sint Laureins in Belgium. A first automatic music generating system was implemented with this prototype. During the exposition an EMO-Synth performance was broadcasted live on the national Belgian radio station Studio Brussels.
06
Prototype 03 developed new possibilities to extend the generating capabilities of the EMO-Synth. The music generation engine was complemented with generated light stimuli. Prototype 03 was shown at the expo SPOTLIGHTS at cc Hasselt in 2007 in Belgium. Prototype 04 was developed between 2007 and 2008 in close collaboration with the young Belgian curator Astrid David and the collective Office Tamuraj. The musicgenerating engine was renewed while constructing this prototype. Music generation in this new prototype was mainly based on an automatic mixing engine. Under the supervision and lead of Astrid David experiments were also being carried out with different forms of presentation and confrontations between the EMO-Synth and other artistic disciplines. This resulted in the so-called test-cases. Testcase 1 was held at the artist’s office in 2007 in Ghent, Testcase 2 was hosted by art space Croxhapox (Ghent, Belgium, 2007), Testcase 3 was realised in collaboration with art organization X=10c and was held at the art fair Lineart 2007 (Expo Hallen, Ghent, Belgium, 2007). Testcase 4 was held both at art gallery One Twenty Gallery (Ghent, Belgium) and at Muhka Media (Antwerp, Belgium) featuring the renowned Belgian psychotherapist Wilfried Van Craen.
Prototype 05 is the current prototype of the EMO-Synth and has been under development since 2009. The creation of this prototype is being supported by the Flemish Audiovisual fund in the seminal project entitled “Montage cinema revived by the EMOSynth�. This project can be seen as a logical sequel on the different test cases held between 2007 and 2008. As to the technical side Prototype 05 includes a renewed sound generation engine which incorporates all previous technologies and a first implementation of an image generation system.
07
PROTOTYPE 05 Montage cinema revived by the EMO-Synth
Introduction The general idea behind the EMO-Synth is to create a dynamic work of art that evolves automatically. Using a natural evolutionary process the computer will pursue the creation of ideal optimized and personal audiovisual material. The EMO-Synth thus embraces a fasciniting new paradigm that opens the discussion on the deprecation of the unique character of a piece of art. Referring to earlier art history; this is a recurrent discussion that also appeared at the rise of photography at the start of the 20th century. At that time the general concern was the loss of the aura that surrounds artwork because reproduction of photographic material severely questioned artistic authenticity. But as history thought us it turned out artistic potential is not equal to its medium. Montage cinema revived by the EMO-Synth is a project based around the EMO-Synth where experimentation with the visual application of the EMO-Synth is a central key point. The project was a result of a series of performances so called test cases – which Office Tamuraj elaborated between 2007 and 2008 and where I acted as curator. In this respect “Montage Cinema Revived by the EMO-Synth” could be seen as a project in which the experience and techniques used in these testcases are merged. Themes explored in testcases 1 to 4 included automatic music and image generation, the use of color- and light dynamics and
08
psychophysiology. During this process as a curator I safeguarded the audiovisual result for which the film constitutes the apex Having a background in art sciences I looked for ways to promote the thesis that the EMO-Synth project belongs to the domain of audiovisual arts. In order to get this statement in the public artistic debate I organized dialogues and reflective moments. In my point of view the EMO-Synth performances refer to developments in the Fluxus movement of the 70s. The fact that Valery Vermeulen visualizes heart rate frequencies and statistical data shows a lot of similarities to how Bernar Venet put together very explicit seance-performances with mathematicians and scientists from 1966 to 2006. Bernar Venet moved from Nice to New York where he joined a group of people with backgrounds ranging from nuclear physics to math and music. In this respect Office Tamuraj can identify itself to the early group of inventors of multimedia and the pioneers of video-art with John Cage, Nam June Paik, Yoko Ono and Bruce Nauman as leading figures. Among these the work of Nam Jun Paik seems to be most related to Vermeulen’s artistic approach. Like Nam Jun Paik, Valery Vermeulen has the talent to renew the arts on a technical, visual and auditory level. The artist Valery Vermeulen continuously switches between three domains by applying scientific
research to his creative process. He consciously puts pure mathematics in the background to serve the artistic result. As the development of the EMO-Synth project has an outspoken interdisciplinary character the collective Office Tamuraj was founded to elaborate the various aspects of the project. Along the way these collaborations seemed to be seminal for the creation of several prototypes. The addition of programmer Stif Labarque turned out crucial for a high end implementation of the current prototype, the editing of the film by Arpad Dorozmai was essential to be able to talk of montage cinema and the collaboration with Liard Kranen helped to bring the EMO-Synth to what it is today. In collaborating with Valery I’ve always pursued my ambition to realize a work in team setting where each member plays his own crucial role. Office Tamuraj has tried to conquer new terrain in the audiovisual arts with its computer controlled work of art. The drive to
capture the human emotion by means of music is perhaps romantic, but it could lead to new to new artistic disciplines where the visitor can engage in a direct and concrete relationship. The purely observational vanishes in favour of a real interactive component. Astrid David Freelance Curator
If you want to build a ship, don’t drum up the men to gather wood, divide the work and give orders. I n s t e a d , t e a c h t h e m t o y e a r n fo r the vast and endless sea. Antoine de Saint Exupery
09
PROTOTYPE 05 Montage cinema revived by the EMO-Synth
project outline The concept of montage cinema was conceived by a number of Russian filmmakers in the 1920s and its core ideas were rooted in the Soviet revolution a couple of years prior. Different filmmakers, such as Vertov, Pudovkin and Kuleshov, all had quite different ideas about what montage cinema ought to be. Therefore the word “montage”, which literally means editing, is used to describe the Russian cinema of the 1920s. In this project though, we started off with the theories and views of Sergej Eisenstein who claimed that montage was “the very essence of cinema”. His idea of “montage cinema” was that filmmakers needed to compose a series of exciting events in order to emotionally stimulate the viewer. This meant that individual shots couldn’t just follow each other, but had to collide. He denounced the classical Hollywood way of editing in which temporal and spatial continuity were quintessential. Shots had to be opposed to one another instead in a ‘dialectical way’. This idea was derived from the Marxist theory in which thesis and anti-thesis combined into syn-thesis and thus determined the course of history. This collision of shots could be based on conflicts of scale, volume, motion and speed, both within and between different frames. These conflicts result in a new mental concept which forms itself inside the viewer’s mind. This mental concept cannot originate from the different shots coexisting, but emerges from the overall interaction between shots. This enabled the director to raise abstract concepts, such as specific emotions, in the mind of the audience. Genuine emotions could not be felt
10
through emphatic identification with an actor/actress playing a certain role and acting out an emotion. True emotions could only be felt by the viewer through the dialectical process of images colliding and thereby generating meaning, superseding the sum of its parts. A frequently used example to explain this theory of how cinema works is the famous scene from Eisenstein’s film Strike in which we see workers who are on strike and then get attacked by law enforcement agencies. Subsequently we see bulls who are guided towards the slaughter-house. The juxtaposition of these shots creates a metaphor in the viewers mind about the workers being treated as cattle and stirs up rage and discord about the way the workers are humiliated. The metaphor in this example acts as a catalyst of emotions that could not have been set off by merely showing images of the workers and the reaction of the establishment to their uprising against the system.
“Montage cinema revived by the EMO-Synth” is mainly focused on the development of Prototype 05 of the EMO-Synth in collaboration with the collective Office Tamuraj. Right from the start of the development of the EMO-Synth there were already some ideas to use it in a movie context. After the initial developments of the first four prototypes funds provided by the Flemish government (supported by the Flemish Audiovisual Fund, www.vaf.be) now make such a realisation possible.
“Montage cinema revived bythe EMO-Synth” also denotes the implementation of our first image generating system. It is basically an automatic video montage system in this case. During performances the EMO-Synth will automatically start producing a new montage of the given movie, while preserving semantic content. Implementing this first image generating system has already proved to be crucial to the further evolution of the image-generating component of the EMO-Synth.
The main goal of “Montage cinema revived by the EMO-Synth” is to explore and implement modules for the EMO-Synth so that it can be used in a cinematographic setting. The general idea – in line with the general ideas and philosophy of the project – is to radically abandon the concept of a static movie. Instead we programmed the EMO-Synth to generate new versions of any given movie with the appropriate, personalized soundtrack. The audiovisual material is generated in a way that maximizes the emotional impact on the user.
“Montage Cinema Revived bythe EMO-Synth” is a brand-new interactive cinema experience that meets the expectations of the viewer as closely as possible.
For Prototype 05 images were needed that could provoke strong emotional states during the performances. To meet this requirement we decided to work with an old classic silent movie. After some internal debate, Office Tamuraj chose Julian Ruperts 1925 version of The phantom of the Opera to be the footage we would work with.
11
PROTOTYPE 05 Montage cinema revived by the EMO-Synth
The Phantom Of The Opera
12
The Phantom of the Opera is a 1925 film directed by Rupert Julian. The film is about Eric, a musical genius who hides himself because of his deformed face. After he has heard Christine, the young opera singer, he becomes hopelessly in love with her. This is the beginning of a tragedy. Christine is engaged to the handsome young count Raoul de Chagny and cannot feel love for a deformed freak. Driven by extreme anger and jealousy the phantom kidnaps the young soprano. As most of the work of the montage cinema school is protected by legal issues we looked in the project for alternative film material apt to being applied in the "montage cinema" context. Extensive research lead us to The Phantom of the Opera. Even though it was made in roughly the same period as the montage film-school flourished, it has few similarities at first glance. But closer examination reveals that it contains all the elements to be used for montage purposes. On one hand there is the world of the phantom, which is dark, gloomy and driven by negative emotions. On the other there is Christine’s world which is bright, happy and has overall positive connotations of love and devotion. The emotional elements within the shots are one-dimensional and don’t really evoke complex emotions and abstract concepts in the viewer’s minds. But a clever re-editing, executed by the EMO-Synth which follows the guidelines stipulated by the director of the project, adds a new dimension to the film. The juxtaposition of shots of both “worlds” will lead to the emergence of new and more complex and abstract emotions which will be reinforced by the new soundtrack that is composed to the slightly re-edited parts of the film. The audience will experience empathy for the villainous character of Eric. This empathy is usually not associated to malicious characters in films and can only be achieved by means of using montage techniques. These elements present in The Phantom of the Opera make it the ideal film to use in a montage context without being a montage film itself.
13
Office Tamuraj Because the elaboration of the different aspects of the EMO-Synth requires various skills the dynamic collective Office Tamuraj was founded in 2007. The collective includes people from different backgrounds including programmers, musicians, designers and people with an artistic aspiration. Our main philosophy as a dynamic collective is to allow people to join in or drop out freely. Being a collective also implies we want to voice a single message: Office Tamuraj aims to translate recent developments in affective computing to the world of music and image generation. In doing so it acts as a bridge between the world of purely academic research and the artistic world. Of course we are aware that we are not the only players trying to attain this goal. But in gathering people with a common spirit and state of mind we want to establish a discussion platform for the possibilities of creating a new era where man and machine can communicate on an emotional and artistic level. For the creation of “Montage Cinema Revived bythe EMO-Synth” Office Tamuraj consists of the following people: Valery Vermeulen – Electronic musician, composer, producer and developer of new interactive multimedia software and founder of the EMO-Synth project. He obtained a PhD in mathematics at the University of Ghent where he conducted fundamental research in the area of algebraic group theory and geometry (PhD in Pure Mathematics in 2001). Between 2001 and 2005 Vermeulen worked at the Institute for Psychoacoustics and Electronic Music (IPEM, University of Ghent) on a research project focusing on the link between music and emotions. Meanwhile he started writing and recording music in his own production studio. Since 2004 Vermeulen has been working on the development of the EMO-Synth and other art projects focusing on sound and music. Besides being involved in the EMO-Synth project Vermeulen works currently as a statistical expert and consultant and studies music at the Royal Conservatory of Ghent. Astrid David – Astrid David – Studied social-cultural work at the Social College Ghent, art-sciences at the University of Ghent, followed by an internship at the Metropolitan University of Manchester. She worked as cultural policy coordinator on two accepted cultural policy plans for Tielt and Eeklo, organized the exhibition Persévérance in the “Godshuis” of Sint-Laureins in 2006 and was associate at the André Simoens Gallery from 2006 to 2009. More info on www.david-vzw.be. Arpad Dorozmai – Graduated in Communication Sciences from the Catholic University of Leuven in 2004 with a dissertation entitled “A filmtheoretical research on the socialcriticism in Elia Kazan’s films”. In 2004 he entered a Master-class in Film Studies and Visual Culture at the University of Antwerp. The following year he produced a short documentary film entitled “Kandarock". For his master degree, he worked on a short film entitled “Nowhere to hide”. The subsequent two years he worked for the “European Film Festival in Brussels”. He came into contact with Valery Vermeulen at Testcase 1 of Prototype 04 and eventually became a member of Office Tamuraj.
14
Liard Kranen – After graduating from Waldorf Education with a project and a paper on ‘DOGMA95’ Liard enrolled in Economy and Arts at the Utrecht School of the Arts. He graduated in 2006 with honors. Liard was involved in numerous projects as a producer such as the short fiction film ‘Hotel Paradijs’ from Jan Krüger and the long-play fiction ‘De Legendarische Scheiding’ from Lars Trim and the documentaries ‘Hooligans’ from Nikita Sutyrin and ‘Kresty’ from Allard Detiger. After his these experiences in film Liard obtained a Master of Arts degree in the EMMA European Media Master of Arts program of the HKU. In meantime he worked as a project manager for the Changing Societies project of BUZA (Dutch ministry of Foreign Affairs) the HKU and VGIK (All-Russian State Institute of Cinematography Moscow). His graduation project was called “Disnified” (www.disnified.org) an online interactive documentary film about the “entertainmentization” of society and the thoughts of young Europeans on this topic. Liard became one of the members of Office Tamuraj art collective in 2007. Furthermore Liard is working on a photographic documentary and an audiovisual installation from the Disnified project. Stif Labarque – In 2004 Stif Labarque obtained the degree of Industrial Engineer specialized in Information Technology. For his thesis he wrote an application that served as an online data loader to provide access analysis for a video streaming website for La nueva España (a Spanish newspaper). Since then he has been developing several web-based and standalone applications using Microsoft’s .NET technology. He was contacted by Valery for the development and implementation of Prototype 05 of the EMO-Synth. Through the use of cutting edge technologies he has seamlessly integrated the new EMO-Synth’s core and it’s subsystems into one application.
Valery
Astrid
Arpad
Liard
Stif
15
16
DVD not here? Go to the media section at www.emo-synth.com
17
Montage cinema revived by the EMO-Synth was supported by
Flemish Audiovisual Fund www.vaf.be
Flanders Image www.flandersimage.com
Mindmedia B.V. www.mindmedia.nl
Montage cinema revived by the EMO-Synth was produced by
Office Tamuraj www.myspace.com/OfficeTamuraj
More info about the EMO-Synth
www.emo-synth.com
v.u.: Valery Vermeulen - OfficeTamuraj@gmail.com design: www.jon-ibe.be