BEHIND THE SCREENS
BEHIND THE WORLD’S FIRST XR JAZZ SHOW Animatrik Founder, Brett Ineson reveals how his team at Shocap Entertainment harness mocap and XR technology to produce a unique virtual performance.
Photos: Animatrik
Vancouver-based Animatrik is one of the largest dedicated performance capture and virtual production studios in the world – facilitating the creation of motion capture shoots for huge video games, such as Gears of War, and blockbuster movies, such as Avengers: End Game. Over the past year, the studio has pivoted towards virtual production and live virtual performances – with Founder, Brett Ineson co-creating Shocap Entertainment to produce original XR music shows, including the world’s first XR jazz show on BBC Click. Produced by Shocap Entertainment, the unique virtual performance saw famed jazz singer, Jill Barber perform a Christmas setlist in an entirely virtual Palomar Supper club with a digital band. The venue was digitally
014
resurrected in CG to house the band and facilitate a transatlantic interview between BBC presenter, Paul Carter and Barber. Ineson oversaw the deployment of mocap for the band – this involved placing motion sensors on instruments, reordering the band setup, synching motion data with audio recordings and CG, and finer details, such as capturing drums with cameras while avoiding the reflective sheen. This data then needed to be synched and presented live, ensuring the musicians were comfortable and able to perform properly. Animatrik’s Vancouver studio was set up with 70 Optitrack cameras by Natural Point to capture the musicians who were all wearing motion capture suits. Unreal Engine programming was harnessed to create
a queuing system, managed by Shocap Entertainment’s Athomas Goldberg. The show required full motion capture performers and simulcam XR as well as Ncam technology for the simulcam. “We used GIANT software to retarget the characters in real-time. There were also several static cameras recording the performance alongside one closeup camera using NCAM to track all of Jill’s movements,” Ineson said, explaining how accurate compositing was applied to synchronise Jill Barber’s image with the virtual plate at all times during moving camera shots. Because Barber was recorded live and the musicians were rendered as CG assets, the team could offset the musicians in the space and set things up in a way that would be counterintuitive to any natural live setup. “The musicians were placed on the other side of the room facing Jill, rather than behind, as their CG assets appear in the CG performance,” Ineson said, highlighting that this made it far easier for Jill to be keyed out against the green screen in the performance. “It had a unique effect on the dynamics of the performance – the band could take direct cues from her in ways that would have been impossible if the musicians were physically positioned behind.” The instruments also had to be motion tracked, with special considerations made when tracking the drum kit in particular. “The symbols, for instance, have a strong reflection, which can potentially disrupt the motion capture cameras’ ability to track the movement, so we needed to mask the particularly reflective parts,” Ineson said, referencing marker positioning as key so as