Page 1

Electronic Integrated Arts (EIA) MFA thesis, School of Art and Design, NYSCC at Alfred University, Alfred, NY, 2010

Moyi Zhang Mixed Reality

The simulacrum is never what hides the truth-it is truth that hides the fact that there is none. The simulacrum is ture. ——Ecclesiastes


Foreword Introduction Chapter 1 The way to Augmented Reality About my research Chapter 2 Being and surroundings “My Baby” Chapter 3 Perception in communication “My best friend” Chapter 4 None-bodyment and bodily form “The New Diamond Sutra” Chapter 5 Subjectivity and Selfhood “Me and I” List of pieces included Bibliography Technical Experience


Every invention or creation is crystalline in nature, having many sides. Although they look opaque they

are nothing other than the reflections of something else. They are mediums that enable us to see past the illusionary state of existence, and help us to understand reality on a deeper layer or higher levels that leads to a place out of which our memories and thoughts evolve. Every creation is like a telescope. It focuses on truths that are out there, and compel us to ask where and what are they or what they are. Through this process we begin to examine some of the more historical questions like where are we in relationship to the space that lies beyond the familiar boundaries of our world and what or where is our true place in the universe at large. These creations force us to think constantly about their reality, even though we don’t know what they are or what it is exactly. We have to vigorously attempt to seize their meaning. Perhaps it is us who are ultimately seized within the grasp of their meaning. Although we may explain more through the virtue of our technology in this digital age, these answers only raises more questions, making us even more aware of old adage: “That the more we learn, the less we know”. The “horizon of not knowing” moves, but is not removed.

Conversely, “What has been is what will be, and what has been done is what will be –done; there

is nothing new under the sun” Eccl. 1:9 RSV. It could means although people claim to create new inventions they are not creating new. Nothing is original or unique. All being that exists has already existed. They are just repetitions that exist in a different context with a different form. People can say: “This is something new” in the present at least. The sense of present means ‘where I am’ and now means ‘when I am’. Thus, the common experience is that the present is inextricably linked to oneself. The notion of the present is when we feel the body’s existence in a certain time and space. This gives raise to the creation of the sense of selfhood, other selves, or a

universal self. Around ourselves, we create reality. So reality is actually something we could perceive within the domain of our consciousness and it extends to the range of wherever our consciousness is able to reach. Imagine we are sitting in the center of our own reality, and then beyond this is a bigger range that is virtuality.

Much of the literature on virtuality contains a false assumption, that reality is the actual and virtuality

is the illusive in opposition to reality. The Oxford English Dictionary lists one of the definitions for virtuality as “essential nature of being, apart from external form or embodiment. So reality could be the representation as a form of an actual entity. It is an actualization to our perceptions, but it doesn’t mean “real”. The word virtuality derives from the Latin virtus, which means strength and virtue. It refers to the strength of a non-physical thing in light of the fact that such things can affect people deeply. Virtuality is invisible; concealed by reality, but it doesn’t mean it is “unreal”. Although it has a great impact on us that may not stay within our awareness. There is another false assumption that virtual space could only exist in cyberspace and virtuality is a new term specific for the digital age. However, virtuality should exist in everything everywhere anytime. Virtuality doesn’t present itself, we have to have a deep sense of what are we are experiencing to encounter virtuality.

Artists are very sensitive question beings, especially in this particular age when people’s lives are dra-

matically changed by digital media. Consequently artists should embrace and undertake the various challenges presented by virtuality.


My investigation is the unpacking of virtuality as a dimension of our experience.

I am looking at virtuality as it is embedded deeply within perception, memory, identity, and communication, in contemporary life. This dimension of experience is intensified during the information age. Therefore my investigation engages experiences of desire, doubt and confusion about what is “real”. In my works, series narrative fictions were created. They are allegories to represent or convey meanings beyond the literal. They reveal the internal relationships and unsolvable contradiction between our understanding of virtual and reality. The digital figures in the fictions create a mixed reality. They observe and experience “reality” through their own perspective. They are representations of un-present selves. They are put into the physical world by augmented reality technology. The AR figures’ present in “reality” rely s on a maker’s presence in the scenes. They have to be seen through the computer version, the extension of our physical visions. Their digital simulated bodies are non-physical but do exist.

Thesis Committee: Peer Bode (chair), Xiaowen Chen, Andrew Deutsch, Barbara Lattanzi

Chapter 1 The way to Augmented Reality, About my research

The first year of my research, the deconstruction and reconstruction of time and space

My first year of making art in Alfred follows two separate but related themes. The first is an investiga-

tion of how our understanding of reality is affected by our perception, memory and altered mental states such as trauma, panic, anger, paranoia and depression. The second examines our understanding of reality and its relation to time and space. I did many works researching issues along these two lines, with the approach of deconstruction and reconstruction I can demonstrate how existing stereotypes may be overturned.

Two of the pieces being made at the first semester using Finalcut Pro and After Effect involve the

recordings from the university observatory and a playground. “Time Game” is a site specific performance video. It plays with the idea of time overlapping and the notion that time exists simultaneously. By questioning the illusion that time is linear, and creating a space in which time and movement are at play together, the viewer observes not only the passing of time but time in reverse, time stuttering and time looping back on itself. Observing the layers of time within a static environment reveals the complexity of times multi-dimensional nature. My performance and movement through the site visualizes time moving in multiple directions and in multiple ways. The performer literally carries time with her as she performs and moves through the observatory space. An observatory is a location used for observing celestial events. It is home to telescopes. Because light takes time to travel, we see objects not as they are now but they were at the time when they release the light that has traveled across the universe to us. So we can look farther back through time through telescopes. They are “time machines” to help us to travel million light-years away. So our perceptions of time and space may not as we normal perceive.

“Circle” is another site specific performance recorded at a playground. “Circle” piece is a narrative

that, just like a circle, is without a beginning or end. The ambiguous event between the two figures is happening over time and space while the camera is going around itself. Instead of following the people, the camera is scanning the surroundings, spinning by the scene. The landscape, which is almost static, becomes an empty space, because there are no other events to catch the viewer’s eyes. The viewer could only see the event when the camera meets the figures at a certain point. All these points are being constructed with our imagination, generating narrative structures. This video work is trying to explore the relationship between the changing image and the looping of time. The work poses the question about our understanding of time in us, time in machine and time in nature.

The piece “The doll house” is a video installation also plays the idea of time and space. The frame of

the video disappears. It is the structure being built holds the space and time. The projected imagery on the back of the house is the digital simulated space while the house itself is the physical simulated one. It was multiples of myself walking and carrying things from inside to outside of the house through doors which are actually the projected image in each of the sections. The doors they entered open up an imaginative space which are out of people’s visions. The house has an intricate structure combines digital, physical and imaginative space in one. All the shots were collected in Harder Hall in Alfred, NY. This building is the place where my studio located in these two years. The structure of Hard Hall is like a labyrinth also related to the idea of this “doll house”. There is always a way in and a way out. But the uncertainty of the direction and our location are always existing.

These three pieces may response to a question asked by Susan Blackmore in the book “Consciousness:

A Very Short Introduction”. “Can states of consciousness be mapped? Moving from one state to another can feel like moving in a vast multidimensional space.” In filmmaking, there are two problems people have to deal with. How to create a sense of time as long as hundred years or five minutes within a certain timeline? How to create a sense of space as big as a universe or one piece of paper within one frame? My question is how do we connect all these elements of time and space together to make up a whole story in our brain? How do we read them and how do we experience them? Gilles Deleusze used the term crystalline narration to describe time and space in montage. “We no longer have an indirect image of time which derives from movement, but a direct time-image from which movement derives. We no longer have a chronological time which can be overturned by movement which are contingently abnormal; we have a chronic non-chronological which produces movement necessarily ‘abnormal’, essentially ‘false’. ” It proves that our sense of reality could be altered by reconstruction and deconstruction of time and space in relationship to our bodies, perceptions and memories.

The pieces being made after them involve a lot of images of water and fire. The way of my working

became more intuitive and impulsive without much interference from my analytical mind. Part of the reasons is water and fire happened a lot in my dreams during that period of time. Such as the sea of fire devoured the all Alfred. Or the city of my hometown was in flood. For me, they are not only symbols of disasters, but also means renascence. It contains people’s desire, rage, hope and desperation in reality. In the water piece, the girl is always surrounded by water and she is somehow trapped into this imaginative psychological space which possibly is her own reality she is trying to get out from. In the fire piece, all the footages I got were shot in Alfred. The video starts from normal shots and then starts to shake, smoke and burn. The houses, bedroom, streets, cars including people were all set on fire. However, it is not real fire, but digital simulated fire and smoke created by a software called Motion. So that things are still keep moving or being normal when they are burning. The sense of humor in it gives it another layer of the meaning of this “reality�. People might doubt if it is real or non-real. Are we confronting a true disaster or an imaginative situation? Should we being frightened or think it as a joke?

Bill viola is video artist also interested in water and fire. He said, “When you make a video of fire and of

water, the resonance is incredible…a flowing stream interpreted by flowing electrons. I was drawn to water intuitively, with a camera in my hand… drawn to its pulsing waves and undulating surfaces, and of course, optically to the images it reflected.” In most of his works, he explores the spiritual and perceptual side of human experience, focusing on themes of birth and death, the unfolding of consciousness. His work has a strong connection to tradition of religious iconography in metaphysical imagery. So where he looks at painting of scene, paintings of religious people and how they represented and becomes part of what he is tapping into.

Bill Viola -“Acceptance“ 2008, <>.

Slavoj Zizek is a Slovenian continental philosopher and critical theorist. His book “Enjoy Your Symptom!

Jacques Lacan in Hollywood and Out” talks about the films of Alfred Hitchcock. His works are very much about emotion, fantasy, desire, what can be said, what can’t be said. They are very psychological in many ways. If you look at all of Hitchcock’s work, you see there are certain motives, certain kind of images, gestures that happens across all of the films over and over again. Such as people looking at people, the staircases or falling down from staircases…in a sense, they are almost like dream imageries. The point is Hitchcock came up with stories that would allow him to represent the images. The images were the most important thing, the things of what that is, the thing about how do they look each other, the thing of staircases, that was really the imagery he was abscised by and he came up with narratives that allows him to use those imageries. So in a funny way there is a similar process to what I was talking about. To find a form for these obsessions and dreams images to live in.

“Meshes of the Afternoon” (1943) is short film directed by Maya Deren and her husband. The narra-

tion of this film is circular. It repeats a number of psychologically symbolic images including a flower on a long driveway, a key falling, a door unlocked, a knife in a loaf of bread, a mysterious figure with a mirror face... It is a surrealist film that depicts a world more and more difficult to catch reality.

All of the themes that are in my works could carry out in painting or in any medium. However, under-

standing of the potential of new medium language and working with new mediums of technology could bring me more experience. New media has affected the way of our thinking and the way of our behaviors in many ways. It is the media that created us and made our lives became different. It even formed the structure of our society. To me, it is something more related to our contemporary lives. Using new medium could be the best for me to deal with contemporary issues and problems. Marshall Mcluhan who is a Canadian philosopher and scholar, know for the expressions of “the medium is the message” and “global village”. He defined media as “the extensions of man”. Mcluhan proves that how media extends our perception and extends our awareness of reality. He wrote, “We shape our tools and afterwards our tools shape us.”

The other writer Siegfried Zielinski helped me a lot with understanding of media development. In the

book “Deep Time of the Media, Toward an Archaeology of Hearing and Seeing by Technical Means”, he argues that the history of the media does not proceed predictably from primitive tools to complex machinery. It is not liner but more like a genealogy. There are turning points of media history or many dead ends where things stop. Dynamic moments of intense activity in media design and construction that have been largely ignored–the hidden layers of media development.

In my research, I also tried to study the processes and techniques of analog and digital image pro-

cessing and to explore the possibilities of different ways of working. Such as manipulating video with a Frame Buffer in conjunction with Max/MSP/Jitter; screen capturing the realtime video streams run form the program; scanning the electronic signals and compositing the scanned images as animations with After Effect. Though working with the input and output of digital and analog signals, I found a way that provides an alterative view of reality. It helps me to explore the potential of creating visual images that helps me to understand reality and memory that is beyond or even more real than the way it is normally perceived. The recordings I did with The Dave Jones Digital Video Frame Buffer have a very strong landscape reference and the resulting sound defined by them. In fact, they are not images of landscape but abstracted pixelization that it is the digital process and the iconic aspect creating the illusion of landscape. They are something we feel we are familiar with but really not. They are imaginative and illusive imagery of something real. People might have references to the things they already know. They might think what they were and what did they become.

Google Earth, a space opened up in time

My homes from 1983-2010

Because of the continuous progress of media and the development of transportation, the world is

becoming smaller. People can more easily and freely travel between different cultural context and systems of thought, and swim between various languages and media. However, the confusions about time and space and the difficulty of locating oneself in the world have lead to the decreasing sense of belonging.

When Google Earth (GE) came into my sight, some thoughts in my heart start vibrating strongly. GE is a

program displays superimposed images provided by satellite, allowing users to visually see the earth resting in space and looking closely to the cities, houses at different angles. Through this access, users are able to gather mapping and geographic information of the address typed into the program. It utilized 3D triangulation photography and CAD (Computer-aided design) to created a computer-simulated environment which is a mirror world, a hybrid space, a representation of the earth we are living in. Alfred University in Google Earth

GE makes a person feel omnipotent in the world and able to navigate the

world at will. The user could unlimited access to this space, a space opened up in time. Anywhere in the world could be arrived at. In a way, the whole earth is possessed by individuals. Each of my previous homes could be easily found which would take me back to that specific period. And more unknown world could be learned beyond my knowledge and life scope. Although it is a simulation or image of the physical earth, it has its own order of time and space. In general, our behaviors are formed according to the environmental conditions. When people stand on the actual surface of the earth, we determine if a place is further or closer from the point where we are standing, based on the linear distance measured from point to point, then the time consumed would be calculated. Time itself does not exist, but series logical connections generated by the human brain, based on space and sequence events happening in spaces. In GE, the viewers do not have to consider how much time was consumed when passing through distances, and they are also not affected by the time lag. People are standing in a space outside the earth, in which there is no need to measure the distance between the starting point and end point, there is no up and down, there is no order either. All the information about time and space have been written into this massive image database, in which all the information stays in parallel position. The viewers could travel to anywhere in time and space by reading a certain information according to their own willingness. Human consciousness is holistic, fluid, and none-linear. The meaning of GE is far beyond a map, the way of seeing was completely changed. In this simulation of reality, the body is liberated, which provide people a kind of experience beyond time and space, beyond body and matter. The human being seems forgotten, the consciousness seems universal, and it could be infinitely small or infinitely big, from known to unknown, shuttled freely.

The first piece I did with GE â&#x20AC;&#x153;Mind Travelingâ&#x20AC;? was a 10h 20mins video. The whole recording was the

virtual vision of my eyes traveling from Beijing and going all the way back to where I started. The person who is actually seeing it becomes the one who is flying there. The geographical environment during the journey is gradually transforming through time. Also, we could see the division between really detailed imagery of the earth and the imagery roughly represented by 3D graphic based on the population of the region. Both of them are photographed in different time and composed together. They are relatively real or not real in comparison with the actual earth.

Google Sketchup, Montage of reality

Google Earth is a huge image database. The aim of creating this database is to simulate reality

as close as possible rather than creating reality. In GE viewers only have the ability to read without the opportunity to change the database. Compared with reality, GE is just an accessory, a tool for reality. GE functions as a “searching” engine, but not as something for creating. Then Google SketchUp appeared. SketchUp is a 3D modeling software designed for architects, game developers and related professions. It includes features to facilitate the placement of models in Google Earth. So that people can create their own models and then upload them to Google earth. Surprisingly, the 3D models can be accurately placed in any position of the digital globe, that is, GE has become an “space of ideas”, and is completely open. Cities, villages, mountains and ocean. . . These public spaces have become available as autonomous private space. Through people’s participation, GE, based on reality, has become one part of our reality. There are no walls and obstacles in this space which exist in reality. Then one question appeared: If you can own the entire planet, what would you do with it?

The other feature of SketchUp is the 3D Warehouse that lets users search for models made by others

or contribute their own models. I collect 3D models from Google 3D models Warehouse. Then I manipulated them and put different models together to create new scenes in Autodesk Maya. They are reconstructed realities usually contain pieces of land or other kind of spaces isolated from the rest of world. This body of works includes digital prints and animations based on the scenes I created in Maya. Print is another display technology. I chose Design Winder and Giclee drum printing to print my prints. They have a particular quality because of the way of the inks mixed and how the ink soak into paper is quite different than the Epson printers which have a more commercial contemporary style. So that in the same way, there is a differences between traditionally be on rice paper or other kind of paper. In tradition art making, we identify the significant differences.

3D modeling in Google SketchUp

Augmented Reality in ARSights and Linceo VR, an extension of the human senses

The black and white maker in ARSights

As I continue to become familiar with the program Sketchup, one of the plug-ins of the program

ARSights became more important. ARSights relies on the technology of Augmented Reality (AR). Augmented Reality is a term for a live direct or indirect view of a physical environment which augmented by virtual computer-generated imagery. Its research explores the application of computer-generated imagery in live-video streams as a way to expand the physical world. As a result, the technology functions by enhancing one’s current perception of reality. For example, people can see the 3D model in hand just like watching a physical object. It’s like a real time feedback loop seeing myself holding something that in fact I’m not holding. But as an image, I’m holding it.

Augmented Reality in ARSights

Virtual reality (VR) is a term we are more familiar with. It applies to computer-simulated environ-

ments that can simulate places in the physical world as well as in imaginary world. Users can interact with a virtual environment or a virtual artifact either through the use of standard input devices such as a keyboard and mouse, or through multimodal devices such as a wired glove. It has been commonly used in pilot training and VR games. Virtual reality technology always associated with terms such as simulation, interaction, artificiality, immersion, telepresence, full-body immersion, and network communication.

â&#x20AC;&#x153;As is the case for virtual reality (VR), several formal definitions and classifications for augmented real-

ity exist. Some define AR as a special case of VR; others argue that AR is more general concept and see VR as a special case of ARâ&#x20AC;ŚThe fact is that in contrast to traditional VR, in AR the real environment is not completely suppressed; instead it plays a dominant role. Rather than immersing a person into a completely synthetic world, AR attempts to embed synthetic supplements into the real environment (or into a live video of the physical environment).â&#x20AC;?

The commonly know examples of AR is, in a television broadcasting of swimming events, there is

always a virtual line which indicates the position of the current leading swimmer. The AR system can be used in many video games too. It simulates the viewpoint of the user and give visual directions to a location, mark the distance, give information about equipment, and display other images based on whatever the game designers intend. In some current applications such as the airplanes, there is usually a head-up display integrated into the windshield.

The main hardware components for augmented reality are: display, tracking system, input devices and


The three major display technologies for AR are: Head Mounted Displays, Handheld Displays and

Spatial Displays. Head Mounted Display (HMD) are usually optical see-through or video see-through in nature. Optical see-through allows the image of physical environment to pass through the lens and the registered virtual graphical objects or information to be projected into user’s eyes. This display technology is being used in medical examinations and car repair. For example, when you wear a Head Mounted Display, instruction texts will show up according to what you see in the imagery from the glasses. In this case, it is about additional information that is present while you are looking at. Questions about the text are answered by that additional augmented information. It is a direct instruction according to your vision. You don’t have to find the information but it’s already in front of your visions.

Handheld Displays employs a hand held display installed on a small computer device such as an

iphone, a camera cell phone product made by Apple Company. It applies video see-through technique to overlay the virtual graphic to the physical environment in the live video steams. Instead of wearing or carrying the display, Spatial Augmented Reality (SAR) projects the image of graphical object or information onto physical objects. A projector or multiple projectors can be used to expand the display area on any surfaces or indoor walls. The viewers are able to touch the object to interact with the projected imageries.

The tracking technologies include: digital cameras, optical sensors, accelerometers, GPS, RFID (Radio-

frequency identification), wireless sensors. Each of these tracking technologies has different levels of accuracy. They applied to track the viewer’s head position or hand(s)/hand input device.

For consistent merging 3D images and the image of the physical world in real time, the software for

Augmented Reality has to process some part of the camera images information into coordinate system data. So that the virtual objects or information can be merged into the image from the camera according to the 3D geometry data.

The software I used in my works is Linceo VR. The techniques I am practicing in Linceo VR includes:

single camera position/orientation tracking

tracking code using one or multiple black square patterns and image realtime raytracing

realtime rendering

realtime animations

realtime contextualization of the digital models in any surrounding environment

publish online realtime AR applications using user-interface

generate QuickTime 360° views of one’s model in varies scenes

By using many techniques above, I could create a more realistic 3D model to its surroundings and

adjust the lighting and texture in real time. The 3D models could even project their own shadows on the physical surface they are standing at. The realtime features could also allow the viewers’ body to actually interact with my models such as wearing a T-shirt with a black and white square maker printed on them. There are many solutions I make to build very site-specific makers for different circumstance in order to track the maker’s orientation and position better. I also could have Augmented &Virtual Reality applications published online so that I can have the application embedded in web and design my own interface for it in HTML/CSS or Flash.

By screen capturing the realtime video streams run form the program, I collect all the footage I need

and then bring them to Finalcut Pro to edit and process them with After Effect or Max/MSP/Jitter.

The different pieces I made using the AR system are not really as a type of an image. But as an episte-

mology, which is the name for the branch of philosophy that is about the nature and limitation of knowledge. It is about a kind of knowledge, information and it’s quiet unique. The months I spent on my AR research of learning the techniques include creating models, textures, reflections and the suitable lighting. They are similar problem while I was dealing with painting. How do I visually construct the imagery using the software to create a more combined vision experience? In the future I am also going to apply animations on the 3D models. So it’s not only about the repositioning but they could also move. It’s going to be the next step in terms of the research. On the level of learning the software, learning the language of those processes.

This technology allowed me to think one step further. I found a simple and intuitive ways of breaking

down the digital space and physical space limits. Like physical objects which interact with the surroundings, digital objects can also interact with people’s movements and viewing points. AR technology is not a simple magic to deceive people’s perception, but a new sensory system which is an extension of the human senses. Since we already know, physical space and digital space are not in binary opposition, but act as one as a unit, it has become an irreversible trend for human senses to be digitalized. AR technology should be used to an enhance and a strengthen reality. The definition of real has to be enlarged. If we continue to believe that digital simulations are untrue to our perception, then what we once believed to be true of any experience might also become simulations. Our life would be the biggest virtual reality; virtual reality would be our biggest life. But in fact, they are not separate.

With the development of media technology, people are so free to use any kind of form of media. People

usually maintain the impact of media on an unconscious level, like a fish unaware of the presence of water. But there are many things we can not ignore, the increasing desire of control, the enriching form and scope with people’s relationships, the storage of human memory became smaller and smaller, people become more and more procedural way of thinking like a machine, emotional sustenance in the use of the media. . . Although media technology can change our way of life, to help us better understand ourselves, to help us better understand the world, it has brought more new problems. Media literacy has to be used to help us critically analyze messages, broaden our experience of media and develop creative skills in making our own media messages.

The first piece I made with AR system is “My Baby”. I had a very limited control on the baby model. I just

deal with the techniques I had for that time. Limitations didn’t stop the idea but became part of the idea. Just like the analogue signals, they are not perfect imageries but giving people more space to think about it. I don’t mind those limitations but I don’t mind to go further either. That’s again talking about the process in the research that does have to do with the tools and technology and using them along the way as I become more and more proficient.

My concern is that virtuality is deeply embedded in our perception, memory, identity, communica-

tion, culture and symbols in our contemporary life. It includes the human pursuit of truth, doubt and confusion with the impact of digital media technology. Information culture dramatically changes people’s lives. I hope I can maintain a sensitive and witty mind to avoid the rashly following of any cultural phenomenon, to find some revealing places, something can awake people from immersing in a narcissistic state with clarification and understanding of media literacy.

Chapter 2 Being and surroundings, “My Baby”

About the story

A spirit from a virtual world was born in the form of a baby and then came into reality. He observes

peopleâ&#x20AC;&#x2122;s life from his inexperienced perspective. His intelligence is like a human so he can understand human life. But his fleshless body keeps him from truly becoming involved with his surroundings. His confusions about the world seems to be the same as the confusions of every baby facing life experience.

To immerse into a space, to experience a reality beyond our daily lives…It is always a question for me

to think how far virtual reality could go. From painting, theater, cinema to a computer-simulated environment includes additional sensory information. Virtual reality is more and more close to real to our perception. It connects the viewers and the environments with a spatial relationship which is something we created in our mind. But there is an interface actually sitting in between. It is a boundary we could hardly go across.

People’s desires and ambitions to experience an unusual life are endless. But it doesn’t mean our daily

lives are going to be abandoned. It’s hard to escape or runaround from what we have in our lives. Something we have to face each day are not easy to understand or easy to gain.

However, what about letting a virtual body enter the physical environment? Let a virtual body experi-

ence our lives with his/her perspective. So we could stand at his/her point of view, forgetting our own bodies, forgetting the way of our living, forgetting our knowledge, everything start from a zero point. What would that feeling be? I believed a newborn baby would be the best character to choose.

The baby has to learn everything and start to think everything from the very beginning in this brand-

new world just like a real baby. However, he has an appearance of human being but no sense organs. What he has to face will be more challenging, he cannot breathe as a human being, he can barely move but only float in the air. He cannot speak, but keep thinking vigorously, which based upon a situation of not. He wants to communicate with others, but nobody could know his existence. How could he become truly involved into here with such a virtual body? How could he survive in here?

This piece also represents my desire to be involve in different surroundings, from China to America, from

one language context to the other, from one life style to a completely different one. It also represents the struggling that I have to communicate and understand a new environment. Standing from a distance to see one culture through the other culture I came from. The strangeness of it is very similar to this virtual baby. Through the baby’s eyes, a different perspective was provided so that the viewers could keep a critical distance to review our lives again.

At the end of this piece, the baby said: “I wonder if people feel as I do about being alive?” He doubt peo-

ple have been through the same situation as him. They are living in this way and act as a human being because they want to feel they actually exist in this reality just like somebody else. Even though this is such a “childish” conclusion he indicates the question offensively to people’s values toward life and world views. Maybe everyone is like an alien to our reality just like this digital baby. No matter where are we from, we still believe we belong here because we are simply used to being here.

The digital baby also has another layer of meaning which is a seed planted into our reality. He looks

weak but has strong life force. He breaks through the defined boundary between digital environment and physical environment. The interface in virtual reality disappeared. Or the boundary never existed at all in no matter what kind of context. Dualism does not work in my art. It became a parochial way to classify the world just like how we consider ourselves belongs to this reality but not virtuality. Digital technology gives us a chance to see the other side of our physical world, to rethink our physical world, time, presence and the act of creation. The virtual and the material are parallel, intercrossed and revealing to each other.

Title: My baby 
 Date: 2009.10 
 Duration: 07:10 Display: Sony Vravia 40’’ LCD TV 


 I don’t know why I’m here 
 But still I’m here 
 I believe I’m here 
 I have a shape of the body, but no skin 
 I have no bones, but I can stand 
 I have no eyes, but I can see 
 I cannot speak, but I know how to express myself 
 I want to feel alive 
 Just like everybody else 
 I want to 
 Sleep in a bed 
 Eat at a table
 Go to the bathroom 
 Read a lot 
 Watch TV 
 Learn how to drive 
 Make friends 
 Find a job 

 I don’t know what tiredness feels like 
 I won’t be hungry 
 I even don’t know what cold feels like 
 People can’t see me 
 It’s like I’m the air 
 I don’t know how they communicate 
 I don’t know if what I am doing is meaningful 
 I think I am a joke 
 I feel like a fish out of water 
 I don’t know what people are doing 
 Why are they doing so? 
 Life is not easy here 
 It is full of blindness 
 I wonder if people feel as I do about being alive?

Chapter 3 Perception in communication, “My best friend”

About the story

A girl was living with a dog beside the lake. One day, she found a little man in a red box. He is very

special. He has no weight, the mirror cannot see him, he could float in the air, he could walk on water, and he could stand on the tree dancing with the wind. The girl did a portrait drawing in order to remember him, even her dog came to see him. The dialog happened between the two drove by their questions and curiosity to each other. However, the girl realized she couldnâ&#x20AC;&#x2122;t physically touch him while she decides to get closer to him. She can not stand with this situation anymore and cast away the red box where the little man came from. He lay in the grass and gradually disappeared with only his outline of body left.

The “friend piece” was a two channels projection in a corner of a space. The images are 90 degrees

from each other. The girl and the little man always exist in different frames. They can’t get across the gap between them. but their conversation from their internal make them linked together. The simple relationship between these two characters indicate an unsolvable contradiction. They are closed but can not touch each other; they know each other but can not truly communicate; they stay at parallel spaces but have no intersection of time.

The friend, if considered as the girl’s imagination, this girl and the world she lives in could be a

dream of this little man. He passed by this physical world, passed by this girl’s heart. However, he couldn’t break through the gap of their communication and totally involved in her life where he did not exist. Although the girl finally decided to break the illusion of her imagination, there is even bigger emptiness left in her “reality”.

This piece also discusses the question of communication. But it is not like daily communication. First-

ly, these two figures are not living in one time line, their dialogs are not happening in real time. The dialogs happened because of the editing, like a time montage, the viewers match the images and language in their brains. For the space, the situation of the two are quiet different too. The lake, where the girl lives is more like a closed mental space. The boy breaks into this space and exists in this space in virtue of a medium-a box. But his body could not truly involve into his surroundings. His existence is temporary, unsteady and passive; he couldn’t interact or make connections to the environment.

Title: My best friend 
 Date: 2009.10 
 Duration: 06:10 Display: 2 HD projectors Design Lab Synchronizer system, Dave Jones synchronized DVD players


This is where I live. One day, I found a box.
 Who are you? Who are you?
 Where are you from? Where are you from?
 Do you like here? Do you like here? Why don’t you come up?
 Don’t you be afraid of falling down? Why are you watching at me on the shore?
 Don’t you be afraid of getting wet? Why don’t you fly?
 Don’t you be afraid of cold? You are too heavy.
 Let’s play a game. I think you are different from me.
 I like the way of you look. How do I look in your eyes? I want to remember your face I want to get closer to you. What does it feel like to be touched? I couldn’t live in your illusion. I’m going on another trip.

I set up the context of the story to establish the existence of communication as abstracted from daily

life, such as BBS (Bulletin Board System) in the message, the video game or various forms of social network such as the dialogue in Facebook, Kaixin, skype, the time and space in them have very similar characteristics as this piece. The first is unreal-time, the second is people and their environment can not make a physical link.

For example, in the video game, we communicate with a number of digital simulation of the figures,

we play games or are even involved in a war in which we can choose our own appearances and roles, choose a place and time to start the game. We exist in the digital environment by using a network identity, as the little man breaks into this physical space. In the digital space, our lives are very temporary and unstable, you can easily “die”, or even self-delete. While we have our internal time, we also established another set of time system in video games, which is relatively continuous by their own, but for us, it is just a timeline being patched up from our time clips, this time is completely controlled by us. We can easily pause, continue, slow down or skip. In this work, I placed the state of such communication in the physical space, so that the viewers as the subject of the physical space could feel the experience of digital body in our lives, to feel their birth and death. Likewise, they have their own set of time, our time is just a fragment of their time, our space is a place they passed by, thus to question our so-called reality, our existence and perception and emotion. Our existence in “reality” is a virtual existence; our brain is the largest virtual machine; our ways of communication is a virtual communication.

Cao Fei-“i. Mirror” 2007 <>

Cao Fei is one of the most important Chinese contemporary artists in her generation. She describes one

of her pieces “i. mirror”, a three-part 28 minutes Second Life-machinima, as a “virtual documentary” of her life in alternate reality. She was documenting those digital figures like they were real persons. Not only records stories from her Second life, “i. Mirror” is also a search for an individual existence and spiritual sustenance somewhere between reality and virtual life. “I knew that this world sometimes mirrored the real world. We are missing a lot of things in real life, the same as in Second Life...Second Life makes me figure out how to face real life.” Cao Fei thinks Second Life as an extension of the self. In it she has created another self, and through this virtual self she observes and understands her own depth and complexity while, at the same time, broadening its scope.

Chapter 4 None-bodyment and bodily form, “The New Diamond Sutra”

About the story

The little Buddha enter the city to beg for food and then came back to do mediation everyday. He takes

the subway and bus shuttle to travel around the city. He observes contemporary society, helps to liberate the suffering of people from their contemporary life. He tried to use language to communicate with people around him, to promote Buddhism, to save the living souls from the sea of misery. But in the busy crowd, no one knew of his existence, no one can stop to listen and accept his teachings. The Eastern thought he represents is alien to the Western context full of ideas and symbols of contemporary culture. He exists in reality, but he stands in an opposite place to deny the reality he observed from everything. Eventually, he is blown down by a wind caused by a whizzing past train.

The Buddha is a rich semantic symbol contains historical facts and religious illusion. It has a dual mean-

ing of real and simulation. He was a distinguished man and worship idol. People could not achieve his spiritual level. However, in “The New Diamond Sutra”, the Buddha appeared in our physical world, a 21st century modern city, where he lives as an ordinary person with a humble identity to experience the modern way of life. His body is a digital simulation, the development of electronic technology enables him to be born again among us, and his soul is activated from then.

The core discourse of the Buddha’s speech and the inherent meaning in the form of the Buddha are

the same, People believe he is unreal because his body is non-physical, people can not see him. However, he is using the “unreal” body which created by this reality to deny the unreal reality as he acclaimed. In the same time, he is also using a language in a contemporary form of expression and concepts to deny the language and concepts formed by contemporary ideology. The biggest contradiction in the work is that the unknowable language and unknowable thought can not be conveyed. So what do we need to express? Why using the language? Why do we communicate?

The Diamond Sutra is the world’s earliest dated printed book in the year 868 AD. It is one of the most

important Buddhist text that has a long and profound history. Towards the end of the book, the Buddha calls it “the diamond of transcendent wisdom”, because its teaching will cut like a diamond blade through illusions of reality that surround them. In the sutra, there is a repetitive dialogue between the Buddha and his disciple Elder Subhuti regarding the nature of perception. The Buddha is trying to help him to unlearn his preconceived, limited notions of reality and enlightenment and help him to transform the way he perceive the world. The Buddha describes reality as “ A start at dawn, a bubble in a stream; A flash of lightning in a summer cloud; A flickering lamp, a phantom, and a dream.”

My piece “ The New Diamond Sutra” is a very simplified version of the text I edited according to my re-

search to Diamond Sutra. Its idea didn’t go beyond the text but very strictly followed the original. The text itself has nothing new in it. The new thing about it is the context. The words were spoken by the digital virtual Buddha in such a contemporary surroundings. The new meanings came from the imagery but not the text itself. When an idea, sprit, concept existing in different context, people understand them differently, they become different.

Nam June Paik “TV Buddha” <>

“TV-Buddha” is one of Nam June Paik’s most famous video installations. It has a live camera on the

Buddha’s sculpture and the Buddha is looking at his live image, so there is live feedback loop. It means the past and present are gazing upon each other. They encounter in this particular situation Paik created. It is an encounter between Oriental deity and Western media. The Buddha is in the state of meditation and self-reflection while he is sitting in front of his virtual image. It is also a self-examination of the technology, mass media, culture and reality.

In my piece “The New Diamond Sutra”, the video is presented across 16 Apple computer displays.

The original video frame was vertically split into two halves right in the middle in order to play on two different computers patched together. It is the live Jitter patch program plays the two halves in two computers. The parent Jitter patch in one of the computers is sending the messages to the other child Jitter patch in the other computer, what is the time code of the video it is playing right now in the video timeline, so that the other computers can follow and make these two separate videos to be synchronized. It takes all of these displays for the image to be legible. Still there are a lot of frame activities going on when all the displays’ color are slightly different and they are out of sync for not more a second sometimes. The technology I used with jitter is case of a different kind of synchronization but it’s not absolutely sync, there is a little bit drift as timing those monitors. However, they are very hard to be observed. Part of it is the result of in the era of the technology and part of it that is also what is that piece is. This piece is in some relationship to that form.

Title: The New Diamond Sutra 
 Date: 2009.10 
 Duration: 07:10 Display: 2 sets of the 8s (16 Apple Display)


I entered the city to beg for food then came back sitting with my legs crossed.

 Men and women,

 How to subdue your craving desires?
 How to develop the most awakened mind? Listen carefully with your full attention.

 It is an illusion of the mortal mind.
 Like a bubble floating in a stream. So I say to you,
 this is your conditioned existence in this fleeting world. You shall be perfectly content in seclusion. Wherever my words are honored, there are my most venerable disciples.

Chapter 5 Subjectivity and Selfhood, “Me and I”

In this video installation â&#x20AC;&#x153;me and Iâ&#x20AC;?, there are eight TV glass tube Sony Trinitron monitors horizontally

arranged from left to right, but the whole installation is a symmetrical mirror image relationship. From the left first monitor to the third are images of interactions between girls and their digital simulation of themselves. And the numbers from the right side of the first to third are the boys and their interactions with another digital version of themselves too. The middle two monitors are places where the boys and girls encounter. Starting from both ends of the array were boys or girls trying to express and communicate with people wearing a mask of themselves; from the ends of the second monitor is the boy or the girl facing themselves or touch and kiss themselves; from the two ends of the third monitor are the images of themselves continued pursuit or follow themselves, while one of the middle two monitors is the girl and the boy stands closely to each other, but their digital simulated body are facing back to each other. While the other image is the opposite way, the boy and the girl are standing on both sides of the screen, eyes looking at the outside of the frame, they are far away from each other, but their digital simulated body are looking at each other very closely.

This eight channels installation make a leap from other AR pieces. It has no speaking and no text. A

body performative gesture, a limited set of gestures and movements are being used instead of the language. There are no imageries of different situations. The background is very clean so that the gestures can stand out. The background which was the same background as in the monitorsâ&#x20AC;&#x2122; background, the image of the gallery wall. The installation was set up as a narrative form, from his/her own behalf, to their separations of himself/ herself, into himself/herself pursuit their own, or self-confrontation, and finally to their relations with others. Each screen, the characters, both physical and digital, are all the same as the phantom who disappeared into the white walls in the background. It is a story without a beginning and an end, but it is a constant cycle of appearance and disappearance relationships.

In this installation ideas are focused on the relationship with the self. The people behind their masks

are trying to be self-expressive, being covered by themselves and trying to escape from the shadow of selves. The action of facing themselves â&#x20AC;&#x153;eye to eyeâ&#x20AC;? implies self-examination, meditation, and also narcissistic and self- criticism; chasing self and being in self-pursuit. Desperately these figures want to capture the traces of their own, follow their own shadow, this shadow is implied in the pursuit of their ideal selves and finding selves or self-control, being freed hem from the themselves, and this digital self is actually determined by themselves. What they pursuit is the shadow of them. They are always there. We can not escape from them and they can not be conquered. They belong to them but we will never get hold of them. We are ambiguous in the relationship with others. When our physical bodies are far away from each other back to back, or our minds are closed. When the two physical bodies are facing each other, our self-images are separate. When we feel separate from ourselves. What our choice would be? What we will comply with?

Janine Antoni is a contemporary artist who focuses on the interrelationships of her body and different

parts of it. For the piece she did called “ Lick and Lather”, she said: “I wanted to work with the tradition of selfportraiture but also with the classical bust...I had the idea that I would make a replica of myself in chocolate and in soap, and I would feed myself with my self, and wash myself with my self. Both the licking and the bathing are quite gentle and loving acts, but what’s interesting is that I’m slowly erasing myself through the process. So for me it’s about that conflict, that love/hate relationship we have with our physical appearance, and the problem I have with looking in the mirror and thinking, ‘Is that who I am?’”

All of the AR pieces have a documentation quality to the image I collected. There is situation that I was

recording, that is a very complicated social, cultural, political situation. For myself, the AR system I used has such a revolutionary function to the process of my video works.

The future

On the technical levels, the possibilities of AR technology seems endless to me. All the AR pieces in my

thesis show were presented by videos or video installations. We were still outside of the video frames. There were no participations.

For the long term, as my research and techniques in AR technology are being more developed, I hope I

can let the viewers interact with my digital models or even modify the models by their own will. They can control a digital-simulated control panel or affect the models by their body movement instead of pressing buttons on a piece of physical equipment. After the AR display technology becomes more widely used and affordable, it is possible that my viewers can wear a head mounted display or hold a handhold display so that they can see my works directly from their point of view but not seeing through a static interface.

For the short term, I chose 3D human figures as my AR figures in the thesis show. Because I wanted to

give them spirit, give them lives and let them speak for themselves. They were strangers to us but they can feel the world as we do. Their viewpoint was provided to us so we can feel their feelings as well. In this way, a more intimate connection was created between the AR figures and the viewers. It is part of the reason why I used narration in the videos to present the ideas. The AR pieces I am going to do next after the thesis show will involve AR text, AR image or AR animated objects rather than using only AR human figures. They may not be presented in a narrative way or shot live. It could be more conceptual and more performative. There could be a lot of action in repetition, interaction between multi-AR objects or a lot of site-specific performance. The goal is to explore the meaning of virtuality in contemporary life in a bigger range and a deeper level. Virtuality exists in human perception, memory, identity, and communication. Virtuality exists in a word, a concept and a history of a culture.

List of pieces included

“Time Game” Duration: 06:29 Dimensions: 720×480 Date: 10.2008

“Cycle” Duration: 10:21 Dimensions: 720×480 Date: 10.2008

“The Doll house” 6 channels video installation Duration: 07:23 Dimensions: 853×480 Date: 04.2009

“The fire” Duration: 6:35 Dimensions: 1920×1080 Date: 02. 2009

“The water” Duration: 12:06 Dimensions: 1920×1080 Date: 02.2010

“Undertow” Duration: 08:55 Dimensions: 720×480 Date: 12. 2008

“XII-Is” Duration: 07:16 Dimensions: 960×600 Date: 12. 2008

“The Organ City” Duration: 09: 12 Dimensions: 1920×1080 Date: 03. 2009

“Mind Traveling” Duration: 10:21:15 Dimensions: 1440×900 Date: 04. 2009

“In the middle of nowhere” Digital Print Dimensions: 20.5 ×17 inches Date: 02.2009

“My Baby” Duration: 07:10 Dimensions: 720×480 Date: 12.2009

“My best Friend” 2 channels video installation Duration: 06:25 Dimensions: 720 480 Date: 03.2010

“The New Diamond Sutra” 16 channels video installation Duration: 06:18 Dimensions: 1920×1080 Date: 04.2010

“Me and I” 8 channels video installation Duration: in loop Dimensions: 720×480 Date: 06.2010


Susan Blackmore, Consciousness: A very short introduction (Oxford University Press, 2005), 109

Jean Baudrillard, Simulations (Semiotext, Inc., 1983)

Slavoj Zizek, Enjoy your symptom! Jacques Lacan in Hollywood and Out (Routledge, 2001)

Marshall McLuhan, Understanding media: the extension of man (The MIT Press, 1994)

Gilles Deleuze, Cinema 2: the time image (U of Minnesota Press, 1989)

Oliver Bimber, Ramesh Raskar, Spatial augmented reality: merging real and virtual worlds (A K Peters, Ltd., 2005)

Siegfried Zielinski, Deep time of the media: toward an archaeology of hearing and seeing by technical means (MIT Press, 2006)

Soeng Mu, Mu Soeng Diamond Sutra: transforming the way we perceive the world (Wisdom Publications, 2000)

Technical Experience

Video and Sound Software: Fianl Cut Pro, Live Type, Motion, Color, Compressor, DVD Studio Pro, Adobe After Effects, Soudntrack Pro, specialty in HD workflows and surround sound

Video and Sound Hardware: Jones Sequencer, Jones Frame Buffer, Sandin Image Processor, Doepfer Modular Audio Synthesizer, Analog nonliner editing.

Interactive Digital Media: Linceo VR, MAX/MSP and Jitter, Processing, Arduino, Adobe Flash, Adobe Director, specialty in Augmented Reality techniques

3D Modeling Software: Autodesk Maya, Google SketchUp (work with Google Earth)

Graphic Design Software: Photoshop, Adobe Indesign, Adobe Illustrator

Printmaking, Digital Print and Photography: Woodcut, etching, lithography, Giclee and Design Winder drum printing, Large format inkjet printing, Color calibration, 35mm B&W and Color Darkroom, Alternative photography (Cyanotype and Anthotype)

Mixed Reality_MFA Thesis  
Mixed Reality_MFA Thesis  

MFA thesis