ABOUT

‘Back to the Moon’ was the first-ever Virtual Reality (VR), Augmented Reality (AR), and 360-degree interactive Google Doodle.

A collaboration between Google Doodle, Google Spotlight Stories, Google Arts & Culture, Cinémathèque Française, Nexus Studios and Fonic, the Emmy-nominated film celebrates the life and work of trailblazer; illusionist and filmmaker Georges Méliès by transporting the viewer inside his magical world showcasing some of his best known techniques and characters. Méliès pioneered numerous technical and narrative film techniques, primarily in the use of special effects and the creation of some of the earliest films in the science fiction genre. ‘Back to the Moon’ pays tribute to these techniques by bringing them to life throughout the film. The film also features a Mathieu Alvado score recorded by the London Symphony Orchestra.

On 2nd May 2018, ‘Back to the Moon’ premiered on the Google Homepage for 48 hours and was viewed by billions of people around the world. Google, Nexus and Fonic undertook an ambitious project to deliverer the film across the following platforms:

⁃ VR GoVirtual Reality Interactive Experience
⁃ Non-Linear Timeline
⁃ VR Youtube360 (2018)
⁃ First-Order Ambisonics B-format
⁃ Linear Timeline
⁃ Theatrical Film 5.1 & Stereo 2.0 (2018)
⁃ Traditional Linear Film
⁃ AR Google Spotlight Stories (2020)
⁃ Google Spotlight Stories (GSS) (2018)
⁃ 6 DoF

The project followed two different timeline structures: the traditional linear film, and a non-linear interactive experience. The VR YouTube360, Cinematic Film and AR Google Spotlight Stories all followed a strict linear timeline. VR Google Spotlight Stories has a non-linear timeline structure where the viewer acts as the camera operator, editor and director of the, or their, film.

Our focus started with the Ambisonics VR YouTube360 version with its strict linear timeline. The project planning and architecture of the Pro Tools session was crucial. All the various audio formats across the many versions was a challenge to keep consistent, but our detailed planning achieved precise and seamless audio experiences. Pro Tools integrated two different types of VR software, Facebook’s ‘FB360 Spatial Workstation’ and AudioEase ‘360pan Suite’. After meticulous trials, we felt that the 360pan Suite offered a broader array of features in comparison to Facebook’s offering.

We had early production meetings with Google and Nexus to discuss the filmic intentions and overall feel of the project. We began by spotting the session within Pro Tools using Clip Groups which identified the areas for our sound design. The visual use of Clip Groups enabled us to pinpoint and organise our SFX throughout the whole project and was a vital indicator when delivering all the individual SFX for the VR Google Spotlight Stories.

We started off by breaking down the story into ‘Main Characters’, ‘Main Action SFX’ and ‘Atmospheres’ and these elements were assigned their own object auxiliary tracks. Each ‘character’ was built up with of foley and SFX tracks, and when required, dialogue, which enabled us to cluster their specific sounds inside its own object. The placement of each object within the virtual space was an arduous task but a very important one. Using the AudioEase 360pan suite, the 360pan plugin was inserted onto each object’s auxiliary track. The plugin created a puck icon inside the Pro Tools video window and this location corresponds to its ambisonics placement. Automation enables the puck icon to glide seamlessly throughout the immersive arena. We discovered by writing the automation (using the puck icon) in half speed playback mode it enabled us to track the objects more accurately.

An area in which the sound design enhanced the immersive feel was the underwater sequence, as a virtual platform gave a unique ability to push and magnify the three dimensional arena. We submerged the listener underwater by treating both the music and SFX together by constructing the underwater sound design, applying a low pass filter across the music score and adding a touch of 360reverb to bring it all together. To emphasise the water drainage transition, we used the puck to track the draining animation from the ambisonic ceiling to the floor, which felt incredibly immersive. By providing a completed and comprehensive SFX tracklay, which, in this instance, more coverage was better than less. This gave us the flexibility to deliver a rich and immersive sound bed for both the YouTube360 and Google Spotlight Stories.

Our focus then turned to the YouTube360 VR mix. With the linear timeline format, this was a tricky one to get right as we needed to guide the listener throughout Méliès’s journey. Visually the VR world was incredibly detailed, so we couldn’t reveal every sound aspect in this delivery. We broke down the story ‘stage’ into three zones; the ‘Main Action’ zone, which followed the story and Méliès’s journey throughout the film, the ‘Secondary Action’ zone, which focused on the outer ring, and the ‘Back Action’ zone which was everything behind the viewers front focus. Our first mix pass was constructed of the ‘Main Action’ elements and then with each pass thereafter we carefully introduced a zone, as we wanted to be mindful not to distract the listener and muddying the mix. The toy train is a great example, as this continuously tracks around the stage, but we only introduce/hear it once it has entered the ‘Main Action’ zone. All ‘Back Action’ zone SFX were disregarded.

The production then turned to assembling the Google Spotlight Story (GSS). Nexus developed an ambitious and complex structure for Back to the Moon. The GSS was based on a non-linear timeline, allowing the viewer to control and direct their own journey. Break points were inserted throughout the film enabling them to pause the main story. Once the viewer stopped following the ‘Main Action’ zone, these breakpoints were triggered and looped. Nexus delivered each separate break point as mov files, which allowed us to create new sound design for these sections. The music was carefully composed to adapt to these changeable break points, enabling ins and outs to seamlessly run throughout the music score. In addition to these break sections, the GSS created a more complex Ambient-Set SFX. These SFX were triggered once the viewer visually interacted with a specific objects. For example, the jack-in-the-box could be triggered at any point within the story. Two versions were sound designed to cover the changeable ambience for both the room and underwater sequences.

The score was recorded at AIR Studios London and the session was engineered by Geoff Foster. Geoff welcomed us to experiment with the use of the Sennheiser AMBEO 3D VR Microphone. The pseudo-alien microphone was placed just over the conductor’s head which captured the dynamic Lyndhurst Hall live room. We processed the recordings using the Sennheiser’s dedicated decoder plugin, AMBEO A-B converter, which converts the raw A-Format recordings into B-Format (FuMa or AmbiX). These recordings created a spatial distinction and sonic width to the orchestral bed.

Once the YouTube360 was signed off and delivered, our focus turned to the Six Degrees of Freedom (6DoF) Google Spotlight Stories. The 6DoF virtual reality interactive experience allows the listener to move in three rotational and three transitional degrees of freedom. The Google Spotlight Stories had their own bespoke ‘Story Editor’ SDK which created the six-degrees-of-freedom.

We compiled and delivered just shy of 200 individual audio clips. Each clip had its own time stamp which corresponded to its relevant timeline placement. To allocate this time stamp a continuous frame counter was burnt into each production video file, with the entire film running from 0 to 3,901 frames. Due to the multi-platform delivery across the linear and non-linear timelines, the use of timecode wasn’t a viable option within this production.

A shared ‘Google Sheets’ document created a seamless workflow between Nexus and Fonic for these deliverables. Nexus had the arduous task of implementing and assembling each individual audio file, mapping them to their object/ roles inside the Google Spotlight Stories. Throughout this process a few updates were needed but overall the ambisonic translations from Pro Tools to the ‘Story Editor’ SDK software was pretty much identical.

The theatrical film version was a traditional cut by the director FX Goby. For this traditional linear film we stepped away from our ambisonic Pro Tools session an we extracted and reassembled the ambisonic Pro Tools session into our surround sound session, additionally stripping out any unnecessary SFX that didn’t relate to the theatrical cut. As the theatrical release had a larger dynamic range, this allowed us to enhance and bolster some of the larger SFX. Once the panning content was reworked into the surround sound field, we completed and signed off the 5.1 mix and created a 2.0 stereo fold-down.

This groundbreaking production was acknowledged at many industry awards ceremonies, picking up a nomination at the Primetime Emmy Awards for Outstanding Original Interactive Program, winning a Clio Award for Best Animation and The Lovie Award for Best Internet Video – Animation.

Through this project, Fonic broke new ground in the field of multiple format audio production. The seamless integration between AVID Pro Tools and AudioEase ‘360pan Suite’ enabled us to realise our creative audio ambitions for ‘Back to the Moon’. Our dynamic partnership with Google and Nexus helped us accomplish and deliver a high quality production across the complex platforms.