2009 ISEA

Interactive Cinematic Experience Supported by Real-time Simulation System: Technology, Methodology and Ideology Mark J. Chavez∗ Ina Conradi† Liu Linyi‡ Jamie Telford§ Nanyang Technological University, School of Art, Design and Media

Figure 1: Happiness and Fear.

Abstract

1

Our research aims to create an active cinematic narrative generated in a 3D simulation engine, postulating the method and approach to addressing the emerging language of interactive cinematic experiences.

This project aims to create an interactive cinematic narrative generated by a 3D real-time simulation engine. Traditional watching experience in a cinema is mostly a passive involvement for the audience, which means no matter what the viewer thinks about the movie, their emotional state will influence the development and presentation of characters and plot. Interactive or Active Cinema, will be part of the final result of the research. Although the approach is not new, the intent of the research is to provide for an immersive engagement for the viewer beyond passive cinema. And thus finally, it should be supposed to improve audience’s satisfaction by implementing their anticipation of the movie.

Traditional watching experience in a cinema is mostly a passive involvement for the audience, which means no matter what the viewer thinks about the movie, he or she cannot influence the development and presentation of characters and plot even when they feel bored and dissatisfied at given session. Interactive cinema, the final production of whole project, although the name is not new, in this regard, could entirely provides a possibility for viewers to choose being engaged, or not to. And thus finally, it should be supposed to improve audience’s satisfaction by implementing their anticipation of the movie.

This project is made possible through a grant funded by the National Research Foundation, Media Development Authority cospace initiative in Singapore and with the aid of the first graduating class of the Nanyang Technological University’s School of Art, Design and Media, current undergraduates in the school and university.

CR Categories: J.1 [COMPUTERS IN OTHER SYSTEMS]: Real time—; K.8 [PERSONAL COMPUTING]: Games— [H.5.1]: INFORMATION INTERFACES AND PRESENTATION— Multimedia Information SystemsArtificial, augmented, and virtual realities; Keywords: interactive cinema, new media, animation, games ∗ e-mail:

mchavez@ntu.edu.sg

† e-mail:inaconradi@ntu.edu.sg ‡ liulinyi@pmail.ntu.edu.sg § JGTelford@ntu.edu.sg

Introduction

The project postulates the method and approach to addressing the emerging language of interactive cinematic experiences. Principle to the interactive experience three key areas are identified and presented in layered contextual order.

As part of our intial deployment we will develop a proof of concept installation piece. The focus of the work will be on integrating computational intelligent agents designed as character archetypes within a dynamically changeable world. This world is created to adapt along a possibility of multiple narratives. Within this structure we plan to develop a method to use the emotive state of the viewer to drive the narrative, thereby authoring a Meta-plastic virtual world. [Mura 2008] In our research we define the plastic figure as composed of the following elements: • Rhythmic structure: structural equilibrium as composed by dynamic relations of form (including audio), distributed along various dimensional • Form Variations: are rhythmical balance relations between form weights. The form variations and sequences are the components of the Rhythmic form • Form Motion: details of character and behavior as understood by idiosyncratic motion. Motion is perceived and gener-

ated within the construct of dynamic rules and balances. The figures movements are generated by dynamic rules balances within different relation levels equilibrium of structural and aspect qualities. The quality motion is built upon an equilibrium of structural form and time based changes • Visual Quality: the appearance of the figure as determined by • Color: indicating the chromatic qualities; • Visual Appearance: the level of qualitative consistency of the silhouette as well as the hue and value;

2.1

Story Development

Rather than a linear and predicted narrative, the story presented by the system will be based on certain structure but motivated by audience’s emotion and attention, which may lead the story to different endings. But the target audience of interactive storytelling is normal people, whose watching experience is only gained from traditional linear narrative, which means that we have to avoid any confusing content, jumpy narratives and unfamiliar film languages in this structure and storytelling. We refer to the most digestible narrative structure, avoiding art-house style and blurring syntax. Thus,

• Hermetic Quality: accessibility and understandability of the content.

Primary objectives To create a System Repository is where characteristics of animation design are addressed within a library of physically based modeling structures and a collection of design primitives. These elements will be independently developed per content idea and are Motif based. [Louwerse and Van Peer 2002] Interdependent models of behaviors and design gathered into a Primitives Repository. The above elements will plug into our graphics and animation system Platform where agents react to stimulus outside of the virtual world; where characteristics of the experience are absorbed by the system and adaptively simulate an understanding of how the audience interacts with it. Having a sense of awareness and behavior provided by the structures built and within the scope of our agent technology.

Technology - developing state of the art 3D real time simulation engine supporting a predetermined repository of designed primitives.

Methodology - using branching storytelling methodology as a construct in response to the needs and wants of the audience in conjunction with interactive narrative representations layered upon the emerging story dictated by the personality and emotive responses of the audience to the resultant unfolding presentation.

Ideology - bringing the technology and methodology together with intelligent agents manifested by a controlling drama manager to manipulate the audience through narrative and story as to make choices that fit within the agency of the system.

2

Social Science Research on Interactive Narratives

This project seeks to further develop and sophisticate the language of interaction. This process is indicative of the maturation of a technology used in the mechanism for interactive narratives and story, such that the audience no longer holds fascination about the mechanism of the presentation (how to tell a story) but moreover in the content that is the presentation (content). This can be compared with the development of cinematic language in the early 1900’s as the audience transitioned from a technological fascination to a more complex language of cinematography. [Miller et al. 2005]

Figure 2: Sample illustration. ten movies have been chosen for sequence-and-scene breakdown from narrative perspective. Selected titles are specifically clustered around the theme of revenge, which is probably involved with action, crime and thriller, considered as a genre with strongly motivated actions and highly organized narrative lines; productions from United States and Asia e.g. Japan and Korea, were included in order to facilitate the analysis of similarities and differences among diverse cultural background. The consideration for selecting such a theme and titles comes from the universal acceptance upon judgment between good and evil, cause and result, karma and dharma, and so on.

After document research on narrative structure and breakdown for existing films, we found that there is a conventional narrative structure embedded in the genre. The visual design and development was concurrent followed the story structure. A senior art director is designated to be in charge of a group of undergraduates and fresh graduates, who are capable of conceptual sketches.

As a result, we are able to crystallize the conventional structure from this study, which eventually supports the interactive storytelling in our final product.

2.2

Fundamental Study on Perceptions of EmotionProvoking Images

Using existing narrative methodology or language familiar to the audience along with existing and emerging methodology from the games industry the project will explore and invent unique and individual narrative experiences for the audience.

As a part of the research project, we develop a study on the audience’s emotional perception towards abstract images, in order to understand audience’s reaction and emotion when they are exposed to given images. This multi-discipline collaboration is involved with artists’ creative work, information retrieval science, and mediaeffect studies. A cognition-affect model [Bolls and Muehling 2007] grounded in social psychological theories will be included as the approach for examining the process that how perception and cognition influence the emotion.

This project identifies that story (content) and narrative can be treated as two ways of interacting with an audience. Story interaction is focuses on the needs and wants of the participant while narrative interaction focuses on the personality and emotive disposition of the participant.

The starting point for this study image generating was research into diversity of stylistic methods of abstract art: abstract expressionism (Franz Kline, Philip Guston, Hans Hofmann, Barnett Newman, Ad Reinhardt, Robert Motherwell, Arshile Gorky, Joan Mitchell), painterly automatism with free creation of imaginary forms, action

Figure 4: Jealousy and Mortification.

Figure 3: Illustrations by Wong Jin Cherng.

painting, (Jackson Pollock, Willem de Kooning), expressive markmaking (Cy Twombly), techisme, Lyrical Abstraction (the Cobra group artists Japan’s Gutai group), Color field painting with highly articulated and psychological use of color (Morris Louis, Mark Rothko, Clyfford Still, Ad Reinhardt, Helen Frankenthaler, Richard Diebenkorn, Sam Francis, Mark Tobey) and calligraphy. [Tuchman et al. 1999] Real Flow and fluid dynamics, Maya Paint Effects and digital photography were used in generating imagery simulating paint tactics of flowing, dripping, and pouring. However emphasize was not on technique or procedure employed rather on artist’s originality and authenticity. Expressive results of techniques employed were layered into compositions manifesting energy of various emotion categories. In case of digital photography rather than focusing on representation of recognizable figurative object, (sunset, flowers, people) focus was on presentation of an abstract elements (line tone etc.) as an independent emotion equivalent. An abstract visual language and its linear complexity, energy, suspense and coloristic brilliance were used to express dynamic relations between different emotion categories. They differ in terms of color palette from overly of saturated pinks reds and greens in disgust category, following with extreme contrast of morbid black and white hues in anger, dissolving hues in sadness to delirium of contrasting blues reds and yellows in happiness. Linear sweep also changes from ornate swellings and curved spillings in fear, articulate linear dribbles and whorls in anger, arabesques and splashes in joy to stains and splotches in disgust.

Abstract art challenges the viewers in a particular way: they are required to look with fresh eyes at pictures that are different. They have to discard old habits, such as the desire to recognize something; abstract art does not imitate, it represents in a different way.

Viewers find no affirmation of themselves in what they see. They are denied the satisfaction of re-encountering a known realityOne of abstract art’s great discoveries is undoubtedly to have made re-

ality’s energetic side visible again. It helps us to comprehend that Nature is just as invisible, immaterial and dynamic as it is tangible, concrete and static. The importance of the in-between is rediscovered. The abstract representation of reality is founded in the tow-way flow of visual energies. Gottfried Boehm. [Gooding 2001] The combination of technology and methodology represents a first order interface that facilitates audience interaction with the ongoing narrative and story. Second order interface involves the interactivity between the system as an agent interacting with the audience.

3

Implementation.

Principle to the interactive experience are three key areas identified for the implementation and development in phases.

3.1

Technology

Hardware layer developing state of the art 3D real time simulation engine supporting a predetermined repository of designed primitives.

Software layer this system contains the input/output program, drama manager program, and real-time rendering system. Control of the system is implemented through the Drama Manager. The Drama Manager is a program that coordinates and facilitates the human interaction feedback loop.

3.2

Methodology

Narrative Story is broken into object orientated episodes. Each episode represents different points along a time line posited with dynamic thematic narrative. Incidental characters have motivation that runs parallel linking in a linear fashion dependent on the outcome of the current scene. The outcome is determined by a computer vision method and driven by the audience in the first scene.

The narrative will be affected by nature of audience feedback.

Interactivity In our system ’Narrative’ is layered on story structure providing a higher level of interaction with the story flow. Where story interaction works more on a branching scene structure, our system will allow the narrative element to directly manipulate the visual appearance of the scene through traditional cinematic and other animation techniques. An interactive narrative relies on or drama manager determined outcomes of the audience agent feedback loop. Upon determining the persona of the audience the visual presented is morphed accordingly giving a unique story experience.

3.3

Ideology

Multi agent systems The system relies on an agent based model.

For this project four agents are identified and are used to propel the audience through any given story path.

• Character / avatar agent • Environment agent • Story agent • Narrative agent Agents can be both the environment and character working in parallel or in juxtaposition. These agents directly represent the more passive story and narrative agents.

Agents ultimately work for the drama manager which at all times works to keep the audience on any given story trajectory. Typically the drama manager will provide the illusion that the audience is in control of the story outcome at any given moment. This is important to maintaining audience immersion in the presentation.

4

Conclusion

Our research initiated in an academic setting and supported by undergraduate and freshly graduating students uses what have become to be known as traditional image-making techniques, e.g. drawing, painting and sculpting. These uses of technology are not limited however to the pencil, paint-brush and clay and include digital rendering and capture techniques. Our emphasis however is not in the technical execution of the work but rather the meaning embedded therein. Using this approach we hope to move our students beyond the role of the artistic technician so prevalent in industry, to a role that embraces the larger function of an artist in society today.

Our multi-disciplinary approach will grow with effective and committed collaboration among all stakeholders striving to achieve established goals. All relevant researchers contribute to the final production in both a theoretical approach and an industrial (production) pipeline. As the theoretical foundation, research in information science provides a perceptual test bed whose results will be the basis for further study on interaction between cinema and its audience. The results of our collaboration managed by a methodology with vested objectives will promote communication between various parties whose standpoints are differentiated in design and implementation.

As such, our system will allow us to further the understanding of image in relation to cinema and innovate with cinematic techniques.

This is both an authoring tool and a sensor based method to analyze audience reaction to narrative, providing yet another rich stream of data for analysis. This research project will create knowledge about subtle meaning in visually inspired narrative, what are the typical and atypical forms and provide a means for the animator/artist to mix classic narrative forms and style with contemporary notions and techniques thereby leveraging the existent material to creating new content streams and methods.

Acknowledgments To co-PI’s Dr. Miao Chun Yan, Visiting Assistant Professor Shen Zhiqi and Assistant Professor Chang Yun-Ke.

References B OLLS , P., AND M UEHLING , D. 2007. The Effects of Dual-Task Processing on Consumers’ Responses to High-and Low-Imagery Radio Advertisements. Journal of Advertising 36, 4, 35–47.

G OODING , M., 2001. Movements in Modern Art: Abstract Art. H EYLIGHEN , F., AND J OSLYN , C. 2001. Cybernetics and second order cybernetics. Encyclopedia of physical science & technology 4, 155–170.

L OUWERSE , M., AND VAN P EER , W. 2002. Thematics: interdisciplinary studies. John Benjamins Publishing Company.

M ILLER , T., G OVIL , N., M C M URRIA , J., M AXWELL , R., WANG , T., S HOHAT, E., S TAM , R., B ORDWELL , D., T HOMP SON , K., AND C ORRIGAN , T. 2005. Global Hollywood 2. Pearson Longman Author: Dina Iordanova Title: Cinema of Flames: Balkan Film, Culture and the Media Publisher: British Film Institute Note: All of the above titles are also requested for reserve for another course. Author: Geoffrey Nowell-Smith, ed. Title: Oxford History of World Cinema Publisher: Oxford University Press ISBN#: 0 19 811257 2 DATE: 1996.

M URA , G. 2008. The Metaplastic Virtual Spaces. Lecture Notes in Computer Science 4820, 166.

T UCHMAN , M., F REEMAN , J., B LOTKAMP, C., AND G EMEEN TEMUSEUM , H. 1999. The spiritual in art: abstract painting 1890-1985. Angeles County Museum of Art.