A perfect party: Look around, It’s up to you
THE PERFECT PARTy An Interactive Cinematic Experience for Oculus Rift Goal of the project
This project investigates the possibilities for implementation of interactive cinematic narratives in the Oculus rift. Using live action 360 degree footage our team created an interactive narrative experience for the Oculus Rift to see how you can tell a story in virtual reality. During the experience, you’re not just passively watching a story, you’re a character that actively influences how it unfolds. To understand our project it’s essential to know how virtual reality and the Oculus Rift work. Result
We created a guideline book with tips & tricks for storytellers who want to create interactive cinematic experiences, using the insights we gained during our production. In A Perfect Party your character hosts a get-together as your best friend wants to propose to his girlfriend. Everything depends on you to make the party a success: potential disaster is everywhere. By looking around and interacting with the environment you try to prevent things from running out of control. It’s up to you to make this party perfect. What is the Oculus Rift?
The Oculus Rift is a virtual reality headset that transports the viewer into a different world. Picture a set of ski goggles in which a large cellphone screen replaces the glass. A set of lenses is placed on top of the screen, focusing and reshaping the picture for each eye, and creating a stereoscopic 3D image. It differs from a regular 3D movie because it blocks off everything in your line of sight except the screen. The sensors inside the headset allow you to literally ‘look around’ in the virtual space by rotating your head. This give a true feeling of space. This not only offers new possibilities in the field of games and movies, but also in areas like education and therapy. Because the movie is shot in 360 degrees there’s a wealth of things to see. By choosing where to look you become the director of your own story.
Immersion “The feeling of immersion, whether physical or psychological in nature, allows the sense of belief that the user as left the real world and is now “present” in the virtual environment” (Sadowski & Stanney, 2002).
In a virtual reality movie the level of immersion is enhanced. When wearing the oculus rift and putting headphones on, the user is completely cut off from the outside world. Your brain is flooded with audiovisual information that suggests a new reality, different from the outside world.
Conclusion So, how can one create an interactive cinematic narrative for Oculus rift? The technology poses a number of challenges. Looking at them as creative obstructions that ask for elegant solutions in the ideation and writing process turned out to be a big help. The guidelines and other posters discuss all tips & tricks in detail. But the most important insights and conclusions can be summarized as follows: • The camera = the user
The camera represents where the user will find himself in the space of the set.
Interactivity • No camera movement
To make the experience even more immersive, we chose to make our narrative interactive. Oculus experiences are always interactive as you can look around. However the user is often relegated to simply being a spectator. In this experience the user is actually able to influence the story by looking around. You can interact with the environment just by looking at certain objects. This gives the viewer a feeling of agency, which adds to the level of immersion. Presence “presence is perceived as a feeling of being transported into a synthetic place or a place created by a computer through the use of various systems” (Sheridan, 1996).
If you combine the increased immersion, with interactivity and the ability to look around freely, we hope that our viewers will experience this sense of presence. The full 360 degree 3D image in combination with the 3D sound, fool your brain into thinking that you are present in the virtual world. Things like using a keyboard to walk around in the virtual world can disrupt the feeling of presence. That’s why we chose not to incorporate any input devices but the headtracking of the Oculus Rift, severing as many ties to the real world as we can, and making it more accessible to be viewed by the general public.
Camera operations are hard to do (operator is visible at all times) This means that your user cannot move around in space, it asks for an elegant solution in the story. • Video stitching
Footage of multiple cameras is stitched together to create a 360 degree image. The place were the images of two lenses overlap is called the stitchline. Action around stitchlines is best avoided. • Fly on the wall vs. user as character
Decide whether your user is acknowledged by the actors in the story (he/she is one of the characters) or the user is invisible (fly on the wall). • Avatar vs. no avatar
decide whether or not you want to give the user presence in the space, give him/her a virtual body. Note that he can’t control this body and it might feel unnatural. • Everything and everyone on set is visible
All your crew, all lighting and all equipment is always on set and therefore in sight and should therefore have a function in the story.
How it was done
Our movie was shot with a special camera system made by WeMakeVR. The camera consists of twelve lenses, all pointing in a different direction. This way the camera can film 360 degrees, horizontally and vertically, so you can see everything around you. When you put on the Oculus you take in the camera’s viewpoint, and because the camera films 360 degrees, the user can look freely around in the environment. The stitching of the separate video files was done with special algorithms developed by WeMakeVR, and the experience was developed in the game engine Unity. There are three main concepts that are essential to understanding virtual reality: interactivity, immersion and presence.
“LOOK AROUND, IT’S UP TO YOU” an interactive cinematic experience for oculus rift Opening 23 Jan - 5PM VondelCS Amsterdam - Expo: 23 - 28 Jan
challenges, tips & tricks: writing and Pre-production
THE WRITING PROCESS
Combined narative model
PLANNING AND STORYBOARDING
Creating the right narrative and narrative structure There are multiple structures that are suited for an interactive virtual reality movie: backbone structure, string of pearls and branching structure as described by Verdugo et.al. (2011):
Our story is a combination of the detour structure and the branching structure, that we have called the combined narrative model. In the beginning of the story, there is a moment of interactivity that lets the user divert from the main story line, only to come back to it later. After this divergence however, the story branches into four different endings. What type of ending you get, depends on the choices you make.
Storyboard For a regular movie, the use of a storyboard is conventional, in which each shot is visualized with drawings. This is a widely known model in the film-making industry. For a 360 degree movie, however, creating a storyboard becomes a little more complicated. For a storyboard to work in 360 degrees, each shot needed to be cut up in four different pieces of 90 degrees. So instead of one drawn picture for every shot, four were needed. Also through the use of the map, the director was able to see where each of the actors and extras were supposed to stand during each scene.
Detour narrative The detour narrative is a narrative in which the main story is the same for everybody, but sometimes the user can divert from the main storyline, only to return to a common point later.
String of pearls The string of pearls narrative is often incorporated in games. It features multiple levels (pearls), that are freely explorable. However this type is also linear in a way, as the user can not progress until the level is completed following the requirements by the designers.
Branching structure With the branching structure the user is set with a choice or task. Depending on what the user does or chooses, the story takes a completely different route. This means there will be multiple endings.
Guiding the viewer In a regular movie the director chooses what the viewer sees. Through cinematic techniques like close ups the director shows what he thinks is most important to the story. In a virtual reality movie however, the viewer has the freedom to look wherever he or she wants. One of our challenges was to find out how to direct the viewer’s attention in a natural way, finding the balance between giving you the freedom to look wherever you want and not missing important information. Our primary solution was the use of narration and other sound cues such as sound effects to guide the viewer in the right direction. Inner monologue and Focalization In our story we made use of a voice-over in the form of an inner monologue to guide the viewer through the experience. To find out what type of perspective to use, we created three identical virtual reality scenes with three different voice-overs: a first person, second person, and third person perspective. User tests pointed out that the second person perspective was preferred. Our subjects reported that it felt a bit instructional, but they understood it was about them and still experienced it as a story, an adjustment was made to move away from the simplistic usage of perspective. Instead, the previously described term external focalization (Gérard 1980) was used to more accurately describe the type of narration we used. Users reacted well to the instructions received from this focalization. This was less the case in the traditional first and third person perspectives we used: users reported those scenes were unclear or they felt like it was not about them. The novelty of this hardware and thus the general public’s inexperience with its functions make external focalization an excellent option to guide the viewer more strongly. Feedback & AGENCY During initial tests, subjects pointed out that they sometimes did not realize they had triggered interactive elements. But the results were unclear as to whether this was a negative or positive result. Therefore, two versions of the movie were developed, one in which the user gets some feedback at a point of agency, and one where the user gets no feedback. This will make clear if the user experiences a higher level of agency if through the use of feedback he or she is reminded that he or she just influenced the story.
Shooting time To shoot a 5-7 minute experience, 2 full shooting days were needed. Everything was shot on day 1, but to make sure it was perfect, there was a total reshoot on day 2. The total length of all the scenes together is about 14 mins, but because the viewer will only see one ending, the length of the experience will be about 5-7 minutes for each viewer. This is one of the drawbacks of using the combined narative model: a lot extra material is needed, that the viewer will never see. This is countered however by the fact that because of the length and nature of the experience, the viewer has a higher tendency to replay the experience.
Challenges, Tips & Tricks: On-Set and Post-production
Directing Actors have to act towards the camera. This can feel weird as they can normally act towards each other. It is advisable to have a lamp or a similar object in the position of the camera during rehearsals so the actor can get used to talking to an inanimate object. Another way to help the actors get to terms with this weird concept, is first to let them try on the Oculus Rift themselves. This will give them a better idea of what a virtual reality movie entails. Finally, actors should keep at least 1 meter away from the camera, because otherwise it will look as if the actor is way too close to your face. Rehearsing During the rehearsal of the movie, there was a lot to take into account. The actors and extra’s needed to be made aware of the stitching lines. These are the lines where the footage of the different camera’s is stitched together to create one image. The lines are normally not noticeable when you are wearing the Oculus Rift, but if somebody is standing right on them, it will look as if somebody is cut in half. We used balloons taped to the ground to serve as a visualisation of the stitchlines for the actors. Where to hide the crew In a regular movie the essential crew like the director and the cameraman are standing right behind the camera. They can see everything that is happening and stay out of the shot. On a virtual reality shoot, this is not possible as the camera films 360 degrees. That meant that the cameraman and director had to be part of the movie to see what went on during the shoot. In A Perfect Party the director was calmly playing the guitar, observing the party without distracting the main actors.
Lighting the scene You can’t use regular film lights because they are visible on camera. It is best to use plenty of white light, and use daylight bulbs of minimum 70-100 Watt that are put into regular lighting armatures. The light used was very white, because that is the easiest to adjust in post-processing. Continuity: tips & tricks Continuity is even more important in VR shooting, even for small things like drinks. • When recording, make sure you have neutral scenes. This helps with the continuity and making shots shorter. • Take still pictures for every scene • Using props to mark stitching line is helpful • Fix all props and objects so they stay in place (eg. with tape) • Actors and extras need to remember their position exactly • Everyone needs to be reminded that they are being filmed continuously Stitching To create a 360 degree image that is also stereoscopic, you need a camera that has multiple lenses. The camera used consists looks like a cube with two lenses on each side. The footage off these twelve lenses is stitched together during post production to create a 360 degree image. The stitching was done by WeMakeVR. The place were the images of two lenses overlap is called the stitchline. Actors in the scene can cross the stitching lines during the scene, but cannot linger right on the line. These will be visable while wearing the Oculus.
Hotspots for interactivity BY looking Hotspots are areas in the scene/movie that allow for or trigger an interaction. The hotspots are important because they determine where activity happens in the scene. They can be programmed to trigger after the viewer has looked at it for a certain amount of time and can also be programmed to be as big or as small as you wish. Do you want the hotspot to trigger when the user looks at the entire wall, or when he’s nearing a certain portrait on the wall? A few example of hotspot triggers • A telephone which will stop ringing once the viewer looks at it. • A door opening after the viewer has looked at it. • A DJ starting to play music after you look at him. • A person starting a dialogue with you after you look at them. References Genette, Gérard ([1972] 1980). Narrative Discourse. An Essay in Method. Oxford: Blackwell. Sadowski, W., & Stanney, K.M. (2002). Presence in virtual environments. In K. M. Stanney (Eds.). Handbook of virtual environments : Design, implementation and applications (pp. 791-806). Mahwah : IEA. Sheridan, T.B. (1996). Further musings and on the psychophysics of presence. Presence : Teleoperators and Virtual Environments, 5(2), 241-246. Verdugo, R., Nussbaum, M., Corro, P., Nuñnez, P., Navarrete, P. 2011. Interactive films and co-construction. In Proc. TOMCCAP 7 4 (2011), ACM (2011) Student team: Leon van Oord .................................................... Director Nour Tanak ........................................................ Producer Nick Valk .......................................................... Developer Shenyu Zhang ........................................ Postproduction Sammie de Vries ..................................................... Script Avinash Changa .............................................WeMakeVR Loes Bogers .....................................................MediaLAB Mirjam Vosmeer ...........................Lectoraat Games & Play Jolanda Dekker...............................................AVROTROS
THANKS TO: CAST, CREW & hosts at vondelcS Dylan Smith, Lisa Smith, Eva Scheltema, Lisanne Smit, Orlando Cabanas, Liam Pieters, Lotte den Hartog, Radoslav Gulekov, Amber Baars, Monique van de Ven, Stephan Aerts, Karin Flierman, Sam de Bruin.
AMSTERDAM CREATIVE INDUSTRIES NETWORK