As the lead singer in the Smashing Pumpkins, Billy Corgan had always been intrigued by the visual side of music from album covers to iconography to stage sets and music videos. VR had sparked Corgan’s curiosity from the start. As he began work on his first solo album, we partnered with co-directors Rob Ruffler and David Liu and Viacom to begin to make Corgan’s vision a reality with “Aeronaut,” a room-scale experience.

The unique content was not only designed as a fully immersive and participatory VR experience, but a 2D video was filmed within the VR experience using Cinemachine, a new software tool from Unity.  The 2D video filmed inside the 3D VR world was imagined by artist Danny Bittman using Tilt Brush and Blocks from Google, and brought to life by the joint Viacom and Isobar team. By capturing Corgan’s performance in volumetric video at Microsoft’s Mixed Reality Capture Studios, we were able to place Corgan inside of Bittman’s world, and use Unity to take viewers inside that world as well.

We filmed the performance on a stage using 106 cameras, then used computer vision algorithms to create textured 3D surfaces of everything in view. We further processed the resulting holographic video to provide some consistency in the meshes over time as well as compressing it for easier transmission and viewing.  The team then worked inside of the VR experience by collaborating, reviewing designs, and discussing interactions. The world premiere of “Aeronaut VR” took place in March, at SXSW's Virtual Cinema and will be available to the public later this year. “Aeronaut VR” been described as the highest quality representation of volumetric video capture and playback in the marketplace and won the Grand Prix for Digital Craft at the Cannes Lions Festival for Creativity earlier this year.