When Generative AI Meets Virtual Production, What Does It Mean for Advertising?

It feels like only yesterday that virtual production was being talked about as the cool technology they used to make ‘The Mandalorian’. Advertising people watched longingly, waiting for the day when it would be commercially viable to use LED volumes to produce ads.

Now the virtual production complex has grown up with dozens of examples of the technology being used for commercial purposes. We have a whole channel on LBB about it. 

“Virtual production has rapidly transitioned from novelty to game-changer,” says Pablo Bertero, chief innovation officer at Wunderman Thompson. “Companies are embracing virtual production not only for its cost-saving potential and environmental benefits but also for its ability to enhance visualisation, simplify lighting, and create more realistic visual effects, ultimately delivering a superior storytelling experience. The era of the green screen is fading away.”

What’s on the LED volumes is changing too, thanks to a parallel technological revolution that’s been impossible to ignore – the rising prominence of generative AI

Wesley ter Haar, Media.Monks co-founder has immersed himself in the changes coming about in this area even more than the average AI-obsessed digital advertising practitioner. “I haven’t been this excited about a leap in innovation since the early days of the internet, and there are some great early examples of AI in practice,” he enthuses.

The imperative to keep track of these two technologies as they converge is something Sergio Lopez, global head of production at Publicis Groupe, is more than aware of. “Virtual production breathes life into worlds that are logistically challenging (imagine shooting in the ideal location, free from spatial, lighting or weather constraints) as well as provides unlimited creative possibilities (overcoming budget and logistical barriers). However, it’s more than just hardware,” he notes. “AI serves as a bridge, translating imaginative concepts into vibrant reality.”

There are three areas of production that Sergio says are “profoundly impacted” by AI: Ideation, production and versioning.

 

Everyone in advertising, no matter how tech savvy, has likely had a play with generative AI tools by now, even if only to write obscene limericks with ChatGPT. As a result it’s easy to see how concepting and creative development is already faster – staring at a completely blank page needn’t last hours ever again. 

“I think that initial concept art is the part of the production that currently benefits the most from AI,” considers Untold Studios real-time supervisor Simon Legrand. “That said, AI isn’t very good at changing minute details unless manually supervised. So I would always expect an artist to lock down the final design after the initial AI exploration.” 

AI is very sophisticated in copywriting, says Sergio. “And while I have not seen entirely original AI-scripts written from scratch, this is very useful for brands that have a very defined creative platform, tone of voice and recurring creative. Consider clients in the retail sector, for instance.”

When it comes to virtual production, it’s often been said that the post production needs to happen in pre production. “This significant shift in the production workflow has seen pre-production take on a new role as the driving force behind the final product,” says Pablo at Wunderman Thompson. “Filmmakers now strive to create meticulously detailed virtual environments well in advance of shooting, revolutionising the traditional post-production process.”

And much of that preparation is where generative AI is already demonstrating its power. “The most exciting aspects that I’ve witnessed with AI and virtual production are often concealed from plain sight, which is what truly amplifies its brilliance,” says Sergio. 

During creative development, it is helping creative teams and production designers explore creative options and pre-visualise ideas quicker and better, spanning storyboarding, mood boards, set design and location scouting – without the limitations of existing visuals or hand drawing. “The major brands are the ones pioneering this area with us, as we use AI-infused set design for Walmart and virtual production with Mondelez,” says Sergio.

Dan Samson, head of creative experience and innovation at Hogarth explains how teams there are using AI in pre-production to generate ideas, text prompting to create visuals. “We combine that with traditional techniques, such as 3D and digital painting. We also use it to take sketches and unlit 3D and ‘paint’ in details. This puts the artist at the centre,” he says.

Melania Kulczycka, client director at VUFINDER STUDIOS, a virtual production studio in Warsaw, Poland, is seeing more and more generative AI feeding the creative processes taking place there. “Currently, 2D image generators are excellent at inspiring creativity” she says. “These generators offer prompts that fuel innovative ideas and lead to fresh approaches in crafting ambience and mood, such as illuminating a scene. Combining human ingenuity with AI-generated outputs has the potential to expand our creative capabilities, pushing the boundaries of possibility even further.”

Director Kristyna Archer is the sort of filmmaker who’s predisposed to using tools like virtual production and generative AI. “My aesthetic of stylized realism lends itself to worldbuilding in ways that have a heightened sense of reality, and these new workflows unlock solutions for the type of work I do,” she says. She can already see AI optimising virtual production workflows for tight on-set shoot schedules, including generating elements in real-time versus loading individual elements from the Epic Games marketplace.

Kristyna, who’s represented by ArtClass, has been experimenting with tools like Cuebric or Dash, Unreal’s new generative AI 3D plugins and has been impressed. “You don’t need to worry about heavy 3D environments that burden the software and require a week of pre-production to optimise them for the wall. Instead, you could consider sourcing 2D stock assets as backgrounds and using AI to add motion to create a parallax in 2.5D.” Vū Studios in the US is building an entire workflow with its own custom AI and previz software that she calls “a gamechanger.” 

For a recent short film, Kristyna incorporated AI in combination with an LED volume to help complete the story she wanted to tell. “Because of my budget constraints it really pushed me to problem solve by embracing an AI workflow and finding my directorial voice in it,” she says. She generated AI environments, set extensions and bespoke footage for the intros and outros out of thin air, without adding time to the shoot day. “It’s another set of tools that can allow directors and their clients to dream and deliver in ways that were previously impossible,” she says.

With the aid of generative AI tools, VUFINDER STUDIOS is venturing into the terrain of crafting 3D environments and assets. “This intriguing advancement holds the potential to significantly reduce the time traditionally needed for constructing virtual worlds, adding another layer of efficiency to the virtual production process,” says Melania. “Although we're currently in the initial phases of generating 3D objects using this approach, refining and optimization are underway. 

“While early attempts have been made to generate 3D models and entire worlds, it's still a work in progress. Typically, these models need quite a bit of additional work to clean and optimise them for practical use.”

In Melania’s opinion, MetaHumans are the most exciting and breathtaking technological advancement. Epic Games recently announced the MetaHuman Animator tool, powered by AI, which can convert a real person into a virtual human within minutes. These virtual humans can be implemented in a 3D environment and become a part of the content displayed on the LED wall used in virtual production. “This tool is continuously boosting virtual production possibilities, allowing for numerous extras in the background with fully-programmed behaviour. For example, it could enable the recreation of a stadium full of people or crowded streets without hiring additional on-set extras. As these tools continue to evolve and mature, we anticipate quicker delivery of virtual environments to our clients.”

On WWF’s ‘Bring our World Back to Life’, Untold Studios used a combination of on-set motion-capture and generative AI to create visuals based on the performer's movement. Simon notes that this project was created before the current ‘AI boom’ and was therefore quite trailblazing. “It was difficult to visualise what the final result may look like whilst shooting the live action and motion-capture, simply because we did not have a lot of information on what results generative AI could yield. However we quickly realised that, if anything, it would create a very fresh new look that wasn’t the same old datamoshing. Using the mocap in conjunction allowed us to have finer control over the latent noise that initiates the generation of each frame, therefore creating a much more stable render than would otherwise have been possible.” 

 

            

While whole, photoreal environments for LED volumes aren’t quite ready to be generated by AI, many elements can be. “It also can be used to very quickly generate textures, skies and horizons that fit the required mood and lighting of a scene,” says Simon. 

Generative AI also comes in the form of automated layout and scene creation tools. “These tools ‘learn’ how to place objects in a scene,” says Simon, “and allow the user to automatically populate a forest with naturally scattered trees or books on a bookshelf without having to place each one by hand. 

“We’re now seeing the birth of Generative AI capable of generating 3D worlds from scratch without the use of existing assets. Whilst this is still in its infancy and not quite ready for primetime, I expect this to become usable soon. It will likely suffer from the same issues as simple image based gen AI, such as being good at initial fast iteration, but needing human intervention to take it to a perfect final result.”

Dan has to be circumspect about the virtual production applications of generative AI that Hogarth are exploring, but he reveals his team is working on “several very innovative pipelines and methods for some of the world's biggest clients – for campaigns, the creation of daily content or R&D.” Replacing tabletop photography for FMCG and cosmetics customers is the most common, he notes. “The most exciting is where we have been generating entire environments using a combination of 3D and generative AI alongside shooting real people in them.”

Most applications are more limited, such as creating textures and details for a low or medium polygon object and projecting them onto the geometry. “We can experiment with various looks and aesthetics much faster than with texture painting and sculpting, which used to take hours,” says Dan. 

Media.Monks has seen immense promise in gen AI when it comes to production efficiency. Wesley mentions how his team recently worked in partnership with HP on a series of films. “We built an AI-empowered production pipeline to level up HP's creative ambition in a series of short form films. Joining together DreamBooth, Stable Diffusion and ControlNet, the virtual production workstream enabled HP to overcome budget and timing constraints to deliver incredible variety in their creative.”

“While AI production holds immense allure, I believe it has a longer journey to mature. It is complex and full of nuisances,” says Sergio at Publicis Groupe. 

Where AI is reshaping production most radically is versioning, says Sergio. “Its ability to process data and modify visuals within minutes is changing the way we adapt, tweak and deliver work.” Publicis Groupe’s DCE tool, for instance, automatically crafts video versioning using insights and optimising the edit within 24 hours, leading to engagement boosts of 15-41% for clients.

“Image creation for social is being accelerated rapidly with AI,” he adds. “While it is going to take some time to have the functionality to create images from scratch at scale for brands, AI is helping us adapt, enhance and optimise images for heightened performance.”

Likewise for Wunderman Thompson, where Pablo is animated about the potential. “Generative AI integration in virtual production holds the key to unleashing true personalisation in content creation,” he says. “The dynamic capabilities introduced by AI, coupled with the ability to localise ads at the production stage, empower marketers to deliver highly engaging and relevant content. By responsibly capitalising on AI's potential, we can shape a future where ads become immersive and personalised experiences for viewers. It’s an unparalleled opportunity, the likes of which we don’t see often in our industry.”

Wesley at Media.Monks is sent into a reverie by the possibilities these converging technologies promise. “I think AI holds the key to really unlocking the original intent of digital advertising: hyper-personalization at scale,” he says. “In the AI-driven era of marketing, the ‘big idea’ can't compete with the relentless pace and variation at which adversarial AI churns out new, ever more optimised creative iterations that outrival industry legends. AI can scale creative ambition to new heights across more touchpoints and more targeted demographics.”

While we are already seeing ‘super shoots’ using virtual production studios to amass as much content in as cost-effective a manner as possible for brands, generative AI can turbocharge this process. This is part of the big-picture thinking driving Media.Monks’ vision of the future industry. “Accelerating content production is the low-hanging fruit, and we call this foundational solution Synthetic Production,” says Wesley. “Marrying the powers of AI and our tried-and-true virtual production expertise, synthetic production offers a full end-to-end pipeline in the cloud that eliminates the need for photoshoots, without sacrificing quality. Augmenting real human talent with AI and virtual locations can help brands deliver high-end craft at a massive scale.” 

At Publicis Groupe, a similar vision dominates the thinking in Sergio’s department. They call it ‘Synthetic Imaging’. “There are a suite of tools that, when combined, become very powerful to achieve more relevant content,” he says. This encompasses AI, CGI, gaming engines, AR/VR, synthetic voice, virtual studios and more. “Combining all or some of them is the only way to fulfil the brand’s needs for scaled, agile, relevant content.”

“At this point, AI is a runaway train. It’s heading towards its own near-limitless potential, with or without our intervention,” says Wesley. And the human resources a creative business like Media.Monks needs to use to harness that need to be flexible. His team includes ‘AI directors’, who he says are tasked with predicting how these technologies’ potential will play out, and harnessing it in inventive ways for the benefit of Media.Monks’ clients. “So their key skills are centred around technical prowess, strategic capabilities, data analysis, and cross-functional collaboration. We need them to be grounded dreamers, analytical yet creative thinkers, strategic but bold leaders, and a little bit psychic.” 

From Hogarth, Dan watches the emergence of this new role with curiosity. “As an ex-Monk and creative technologist, it's all about exploring the next possible,” he says. His team at Hogarth includes designers, 3D artists, "prompt engineers", programmers, writers, game designers, illustrators, product designers, and conceptual thinkers. “This is not your typical creative team,” he says. “Being interested, getting dug in, and exploring these technologies is key; I really believe all creatives should think like creative technologists. We turn up with prototypes to meetings, not just ideas, and we explore and experiment with new methods. Part thinker, part creator, production and thinking together, not silos. New work methods necessitate new teams.”

Working in an environment where these new disciplines are being put to work, Melania at VUFINDER STUDIOS sees the key to success in this context as “writing an effective prompt which is specific, lucid, and full of details. Having an experienced team skilled in composing prompts is pivotal for achieving outstanding outputs using AI tools.

“AI can certainly reshape certain roles, potentially reducing the demand for positions such as graphic design, since AI can now generate images. Yet, there's no need to fear artificial intelligence. It won't entirely replace humans but will rather complement and augment them. Embracing AI and understanding its proper utilisation is essential for improving our efficiency without undermining the human touch.”

Sergio doesn’t want AI expertise concentrated in any particular discipline within the production department at Publicis Groupe. “We believe AI will be used across everything we do, thus urging us to leverage its power across our entire team, not just a select few,” he says. “Some people in our team use AI to breathe life into visuals (directors, art buyers), others to better understand performance and insights (strategists and data translators) and others as an execution tool (retouchers, VFX artists).

 

“It’s much like using Google Docs or Office 365. And by the way, there used to be typists back in the day too.”

As is so often true of hyped technologies, it’s crucial to remember the fundamental principles that have guided the creative industries up to this point. That’s the lens that Pablo at Wunderman Thompson views this exciting time through: “As we embrace the potential of AI virtual production, it is crucial to strike a delicate balance between innovation and practicality. Meaningful applications that enhance the audience's experience should take precedence. We must ensure that generative AI technology amplifies storytelling rather than overshadowing the artistry and narrative substance that lie at the core of filmmaking.”