MediaLAB Amsterdam is now the Digital Society School! You are viewing an archive of MediaLAB projects. Content on this website may be outdated, and is without guarantee.

Virtual Cinema

Team

Peter McLaughlin petermirrorimageuk@gmail.com

Description

In this project we will investigate and innovate Virtual Experiences and Virtual reality. The next months we will work together with HVA researcher Mirjam Vosmeer, Avrotros and WemakeVR to create a deepening Virtual Reality Experience for the brand new tv-series Project Orpheus.

Join us on our Virtual Journey

Journey

Enter our world and learn about the constant flow of changes happening in the domain of VR.  Platforms, new ideas, VR filming, narrative advancements and contemplation on technological challenges are some of the issues, which will be touched upon. Follow us and stay up to date  with our own virtual reality project as we move towards the creation of a VR film. Let’s embrace and explore the endless possibilities of constructing and designing new virtual worlds. As Charles Stross said, ”It turns out that the killer application for virtual reality is other human beings. Build a world that people want to inhabit, and the inhabitants will come.”

Meeting the composer

Whilst finalizing the edit, we have had a meeting with the composer of the Project Orpheus series. Since the series is still in production, one of the difficulties of our project is to find the right soundscape that would both enhance our project and fits the feel of the series. Also showing the edit of our film to the composer gave us insight for some last alterations of our project. Working on a film for a few weeks compromises your perspective in 360, and at a point it becomes difficult to understand how a first-time viewer would behave. For instance, we anticipate on certain actions to happen in the film and we know where to look. A first time user would not be able to predict these actions and therefore misses the executed actions. Sound becomes an incredible important part of the VR experience to create forms of guidance for the user, hence where the user should look to improve the experience of the story.

VR creates the demand for audio and visual production departments to work increasingly simultaneous. In cinema and television the sound department usually starts working after the final edit, which is made possible because the editor is already able to construct what the viewer sees. The to be watched image is already framed. In VR the framing becomes a combination between the created 360 spaces and the choice of movement by the viewer. Therefore sound becomes increasingly important as a tool to guide this simultaneous framing by the spectator and the creator of the VR film.

Filming Days

12316448_10154341021033362_5246372750524115306_nThe weekend of the 28th and 29th of November was the peak moment of our entire creative since  those days were our shooting days. Everything done until that moment was to prepare the ground and be ready for any kind of  problems and changes. The pre- production phase from the long scripting sessions, the room 3D sketching, the VR research, the call sheets, the meetings and the endless emails about a variety of matters to the scenes’ simulation and props hunt, it was an amazing ride. However, the smoothness of the filming days showed that we were lucky because we had a combination of opportunity and preparation. Everything went even better than expected, the crew was in tune, the actors were in the right mood (especially our rising star Martjin) and the sets were so beautiful that helped us create the best atmosphere during shooting. People were moving up and down but a sense of harmony and coordination prevailed since everyone knew his/her duties and responsibilities.11202447_10154351182638362_2287403331340650845_n
Filming with a 360 camera is a different experience from the one we see behind the scenes in Hollywood films. People who stand behind the cameras and observe the scene’s progress is something impossible to see with in Virtual Reality shootings. When the director starts rolling, the crew along with himself should have already gone outside. Thus, the performers are left alone in the room until the director shouts “Cut!” and then everyone swarms back in order to proceed to the next scene. The risk of losing a scene or having an unsuccessful one is something possible but impossible to control when the camera starts rolling. Fortunately, most of the footage turn out to be efficient in terms of quantity and quality.

Overall, filming in VR was amazing that taught us a lot about the newly emerging medium but also gave us a good experience on how crew and cast can collaborate in a professional but also entertaining way.

The Art of Worldbuilding

worldbuildingA different kind of approach towards creating stories is the concept of world building. Stories are not stale representations of ideas on paper. Evidently, this is not enough anymore since so many scripts and stories have been written, rewritten and adopted by others as inspiration. There is no virgin birth in writing. However, a story that goes in parallel with the visual construction of the world that surrounds the story. From maps and backstories to sketching the sets, forming languages and dialects, the creative department should cooperate with the director and the writers into formulating a story that is organic and is tested long before it can be considered finished. In this way, the script goes through various changes since the different parties influence the course of the narrative by adding or removing things in the initial idea. A strong visual environment, a historical account of the place but also a universe around the main characters that enriches the story and makes it plausible and interesting help the audience immerse deeper in the storyworld. The audience has to decipher the messages and dive into the multiple layers of the narrative, either by retrieving information on the main characters or on other aspects of the universe.
World-map-concept-art-game-of-thrones-21953715-2048-1146The workshop that was offered to us( the Media Lab interns) by our mentor Joris and his collaborator Romke Faber which was focused on storytelling and world building was extremely useful. These two aspects of narrative construction have been closely related since creating a huge universe around a story implies multiple opportunities for spin-offs, character building, story development through games, comic books but also films and series. Thus, the wider the world, the higher the chances of giving life to a plant-like organism that can expand and offer new creative way outs.

The whole workshop was interesting since it was really satisfying to witness that coincidentally our Virtual Cinema team followed a similar plan of work. Our story evolved through creating the actual spaces in order to see what works and what’s not. Thus, the idea of world building  is valid and highly inspiring for constructing fictional universes in the contemporary world.

 

 

Stitching: Editing in VR

Virtual Reality differs in its core to the traditional cinema. However, one of the important parts of it is the process of stitching. In cinema, the film is edited and the scenes are cut and modified in order to create a unified and linear end product but each scene is shot by on specific camera providing one point of view. In the case of VR, before the editing stage, there is a new category of editors that could be called the “stitchers”.

The Falcon Camera by We Make VR

The Falcon Camera by We Make VR

In order to have the needed spheric environment for VR, you have to stitch together the amount of cameras rolling at the same time and recording the scene from different perspectives. The range of the cameras’ number can reach up to 25 lenses which overlap and thus should be organized accordingly to get the best image quality but also prevent VR sickness as it’s called. Thus, the stitching phase is similar to puzzle solving because you have to put all the pieces in the right place. This process gives the editors a better control of their own footage by closely observing every provided angle. However, the filming process has to face different obstacles such as a camera failing during shooting or starting later or earlier than the rest of them. The program that can perform this kind of complex stitching is called Auto Pano and it is a French software dominant in the market for the time being. However, as VR evolves ,there will hopefully be new  editions of the software or more companies creating updated programs.

Learning to use this software gave us a better understanding of how Virtual Reality works and made us more aware of the infinite possibilities that this new medium brings with it.

Art Direction in VR

Virtual Reality is not  just another extension of film or theatre. Researchers and content creators are still experimenting on the all possible ways of approaching their virtually created environment.  Constructing narratives is a difficult process due to the first person perspective and so the story should be viewed through the eyes of the user in order to prevent nausea , dead ends in term of the narrative’s meaning as well as failure of the experience’s message to cross over.

art directionAddressing VR from a creative perspective, the art direction in a virtual experience is a complex and interesting field to experiment on. Since everything revolves around the user, then  the orchestration of the props, clothes and generally the whole setting should be organised in order to surround the viewer. Virtual Reality is not a flat projection of a story , au contraire it is a sphere whose core is the viewer himself/herself. Therefore, the art director should build an environment that matches the story but because the user is immersed into it, the set plays an organic role to it maybe even more than in cinema. The props that are in the closest range to the user can be observed in detail and thus should be planned and imagined with caution. That is the reason why Virtual Reality works really well with games, the VR creators are the ones who construct a spheric world around a player who is no more a spectator/gamer but he is an organic part of the world. Thus, in order to assume his/her position as an integral piece of a world, the user should feel immersed and what better way to achieve that by forming a plausible and inviting environment.

In our case, since our VR experience is linked to a TV series, an environment that creates general or more direct connections to the content and atmosphere of the initial story is of high importance. The extra mile is when VR also creates an abstract layer in order for the majority of the users to come to their own results or have a unique after-death experience and that is an end we work towards.

Dutch VR Days: A Threshold To The Future

dutch vr days 3

Vondel CS: The Centre of VR


The best way to inspire a group of students that work on a Virtual Reality project is definitely to throw them in a pool of more than fifty VR demo experiences from various companies and research teams. VR is the future and the “Dutch VR Days” event hosted in Vondel CS surely was a master blaster for the audience but also for tech lovers, games and researchers as well. On Saturday the 31st, the admission was free and thus everyone could pass by Vondelpark and enter a world full of infinite possibilities. You could actually transport to different eras, countries, fly on a broomstick (not kidding), become part of Van Gogh’s Night Café but also be in the middle of a crazy artistic video clip. However, not all of the available experiences were that exciting or well-made but in a period that this domain is flourishing, the only way to form a new channel of communication is to include every fish in the sea both small and big. The event was a successful experiment because it wasn’t promoted as an Avant Garde, elitistic kind of technological symposium but it proved to be the threshold of a whole new era where people can experience multisensory journeys.

 

However, this blog post aims to value the best ones in terms of visual quality but also content wise. It makes sense that content in a developing entertainment field is not an absolute necessity since concretizing the basic functionalities of VR is a top priority nowadays. Understandably, there will be creators that manage to balance the gap between technological progress and storytelling possibilities. The winners of the first Dutch VR Days are by far, NFA’S   “Thousand Faces “ and Revolver Amsterdam’s “Amani-360VR for Terre des Hommes”. They innovate in both storytelling ways as well as interactive strategies with the viewer in a tactile and empathetic way respectively. But let’s break it down in order to appreciate to the fullest the experiences they offered to the public.

amani

The little protagonist of the VR experience

Amani narrates the story of young girl in Africa who is being subjected to a regime of child slavery, which entails verbal and corporeal abuse as well as sexual harassment. The short VR film aims to raise awareness about this cruel phenomenon to the Western societies but it does not negate the art of cinema. On the contrary, this film could pose as a bridge between VR and traditional cinema since it encompasses cinematic techniques that could work in virtual reality as well. For example, in the beginning of the VR experience, the viewer is shown related to child-slavery with a voice-over introducing the topic of the whole venture. After this cinematic commencement, the magical moment comes that the viewer enters into the girl’s reality by literally getting in the screen. From that point on, the viewer experiences everything around the girl’s everydayness even though VR offer a rather static subject position. However, the film stages the scenes so successfully that the headgear user feels for the girl and becomes witness of one of the darkest spots on modernity. Even though, there are no specific shots since VR offers the illusion of freedom to the users, the directors Eelko Ferwerda & Joris Weerts manage to use sound localization to direct the viewers’ attention. The viewers watch the girl’s actions and simultaneously monitor the environment around her. After reaching a shocking climax, the VR experience succeeds in making the viewer feel responsible for their ignorance and quiescence. This results in making them want to somehow contribute in any possible way to the eradication of these inhumane situations.

thousand face

The Promotional Poster for the Dutch VR Days

On the other hand, Thousand Face decided to innovative more on the part of the technology and the incorporation of the senses. The basic story revolves around a spaceship exploring a new planet where a unknown source of power exists. Getting into a pitch black room with the Oculus Rift headset is a mysterious enough beginning for the experience. If you combine this with a haptic technology glove, you have the future of Virtual Reality. Being in an environment where you can interact with your surroundings but in a meaningful way so that it adds to the story, is what will define how successfully haptic technology will be combined with virtual reality. Gathering energy power from objects in the room that are digitally recreated in Virtual Reality seems like a good start to explore the infinite possibilities of modern gaming and visual technology. It is a basic start in therms of narrative but it captures the viewers mind because they actually feel useful by contributing somehow to the virtual environment they find themselves. Being a kind of an astronaut who collects intel about an unknown civilization is a workable basis with which the user can relate to since the reality he/she is in is visible and tangible.

Conclusively, there will always be Virtual Reality experiences that do not take into consideration the user’s needs or standards. As a result, they cause a sense of nausea (high speed, sudden or odd change of camera angles) to the audience but at the same do not satisfy with their non-existent or basic narratives. However, the creators that centralize the users concentrate their creativity and technical skills on fulfilling their expectations, pleasantly surprising them and challenging their notions of gaming, cinema and visual arts in general.

 

 

 

 

 

 

Experimentation with a Virtual Environment

12167491_10206458275727821_1349313760_n

The Stitched Angles and the room’s floorplan

In order understand and familiarize with Virtual Reality, simulation is mandatory. Having to think that an immovable camera films all the different angles simultaneously, is something quite uncommon for the average mind that views filming through traditional cinematic lenses. Recreating a room in 3D was the first step to visually grasp how a camera with 16 go pro can be used in a specific environment. After the room’s formation and decoration, the camera was put in the middle and take pictures according to the angle of each pair of go pros. Think of the camera as a bizarre multifunctioning organism full of pair of eyes. 16 go pros make 8 pairs of eyes and they seems as if 8 people are looking  around the environment you are in, towards all possible directions as well as up and down in some VR cameras. Thus, the viewer wearing the Oculus Rift headset is witnessing what the cameras recorded for him/her but without restricting the possible  head movements. This kind of freedom implies the creation of a well-structured environment and a demanding post production processing for the end product to adequately resemble reality.

The experimentation with the different pictures takes from the 3d room helped us understand how all the partial angles match to each other and therefore can be stitched together since they are all pieces of the same puzzle. The process after this realization was to print the pictures of each angle and “stitch” them (glue them one next to the other) together in order to create the feeling of the 360 environment. This revealed to us how much of a delicate procedure is to position the camera in the right place and then stitch the cameras’ footages avoiding overlapping angles and creating an as much as possible realistic environment for the Virtual Reality users.

Meeting the production company: NL Film

To shape, polish and reinforce the script for our virtual reality film we went to NL Film to discuss ideas with their creative producer Kaja Wolffers. Located at the lovely area of the Museumplein, we grabbed a coffee and talked about it in a chilled setting.

As the production company of successful Dutch series as Penoza, Spangas, Bluf and many more, NL Film has established itself quite successful in the television business. Soon to be airing new project called Project Orpheus, which is the base of our project, was the main subject of our meeting. During this meeting we talked about different elements of the series for us to put into the VR film to create a fitting trans-media story. We discussed overall themes, personages of the series, musical composition and the design and feel of the sets to recreate the same character of the series within our VR experience.

Untitled-1

Moodboards

To do so we brought moodboards, showing the look and feel of the scenes we are having in mind, and mixed them with the vision of Kaja on the series that he is producing. The main challenge would be to make our VR experience both a self-explaining work for those not familiar with the series and an interesting extension for those who have watched the series. This lead to the insight to transform aspects of the series into little clues, or Easter eggs, to be put into the VR experience to be found by the attentive viewer of the series. Also we were looking for an actor of the series to act in our film, again making both sense for those who watch the series and whose presence wouldn’t distract the viewer. After brainstorming we managed to find a solution and we now have the appropriate actor for our production. The second part of the meeting consisted of the production values of the project. This includes hiring actors from the series, set-locations in Hilversum and different personas of the production team such as art-director, location manager etc.

The end of the meeting resulted in virtually a finalised script, choice of actors, usable elements of the series for our project and a whole new range insight.

Sleepy Hollow: A success VR story

A visually stunning film without a good scenario can be a flop or even a blockbuster success. However, things are quite different in virtual reality. The narrative should be the main focus of any Virtual Reality project because that is the only way that this new kind of virtual entertainment can survive and flourish. As James Milward, founder and executive producer of Secret Location and Emmy winner for the Sleepy Hollow VR experience, stated in an interview for BBC: “Bad content will ‘destroy’ the virtual reality industry”. This is a blunt but honest statement since VR will never evolve if it does not cease to advertise itself as a sidekick attention-grabber to films, series and games. To centralize itself in the entertainment sector, the production companies should invest in the narrative and should investigate the ways that the users’ are stimulated by the input that they are offered.

1

The graveyard built around the story

For the sake of mapping out the right processes to reach the ultimate immersion in VR, let’s take the most successful example and use it as a case study. Sleepy Hollow VR is not the average VR video since its narrative storyworld is already built thanks to the series (2013-present) broadcasted by Fox. Therefore, the VR experience aims to excite the existing viewers but simultaneously intrigue prospective fans.  In order to understand the successful recipe of Sleepy Hollow, we have to observe how our brain receives the stimuli and how the human psych is influenced. First of all, by wearing the headset, the user enters a different world and thus his/her brain and particularly his/her Neocortex activates the Hippocampus which is responsible for spatial mapping and memory creation. In other words, the brain tries to familiarize itself with the uncharted territory by creating a visual mindset.

3

Ichabod Crane guiding the user

 

Next, Ichabod Crane, the protagonist of the actual series, appears and begins structuring the narrative. By warning the user of the existence of the Headless Horseman, the character recognizes the user and incorporates him in the story. Therefore, the user forms his/her own social and self presence in the virtual world by recognizing himself/herself and the character he/she encounters in VR. After the mapping and presence formation, the deep immersion begins where the user interacts with the surroundings as of the context is real and thus feelings are triggered such as fear and anxiety. A huge role in the stimulation of these feelings has the sound localization such as the subtle vintage music, the horse galloping, the heavy footsteps and the bats sudden screeching.

 

2All the above well-structured elements achieve the highest level of immersion when the Headless Horseman appears and beheads the user. The story may not be real but the astonishment cause by his action proves that if our brain is convinced of the reality of a situation, then our reactions are stronger and with various implications on behaviour, memory and the experience on the whole. Therefore, if the memories stored in the Neocortex (after being formed in the Hippocampus) are realistic enough, then the brain can confuse the real and the virtual. In the long run, this can only mean that the users may treat virtual reality as another part of their lives and not as something artificial. However, this is kind of far-fetched but in the present, but a positive virtual reality experience for the users, can create a new norm that will seek for virtual thrills. And as always if there is demand, more supply will come as well.