Beyond the Hype: How AI Is Really Being Used in Animation and Post

“Sometimes, it feels like you are wrestling a 1000lb gorilla, and you just want to give up. But we can’t give up.” 

Brian Sanford, director of post production at New York-based creative studio, Versus, says that to get what you want with generative AI-enabled workflows, “you need to really wrestle with the tech”. 

“Relinquishing control is not an option,” he says. “Control is our creative agency over our ideas and the essence of why we do what we do as artists. It baffles me that some of the same great creative minds that I have worked with who would lose their minds when their clients asked them to make the logo 2% bigger are now parroting the trends that AI is going to take over.”

Brian believes that AI can empower artists to reach their full potential, giving them more control, allowing them to find the path forward more quickly, and expanding how they communicate with one another. “For example, an editor will now deliver an amazingly crafted animatic back to VFX after using some of their style frames and generative AI. These kinds of network effects will only make the work better and never betray our values to the craft and our artists.”

But while he says the real-world creative solutions and efficiency gains are obvious, Brian notes that the technology’s downsides come from how the tools are being applied. “We’re seeing examples lately where the ‘how’ isn’t considered, and there’s too much emphasis on the novelty of the technology rather than the quality of the creative output,” he explains. “Even worse, there’s a tendency to use ‘oh, it’s AI’ as an excuse for bad work. AI is meant to supplement creativity and enhance the level of the creative output, not degrade it.”

Looking at the technologies Versus has had the most success with so far, Brian lists Midjourney, Runway and Adobe’s array of tools as some of the leaders of the ‘AI arms race’ that’s seeing myriad companies sprouting up each day to jockey for pole position. “Midjourney and Runway are an amazing one-two punch for image and video generation,” he says. “Adobe is leading the pack when it comes to ethical and sourceable outputs and native integration into its suite of products.” 

Brian and the team have been using Adobe’s generative machine learning model, Firefly, to generate both images and video for animations – a process he demonstrated recently for editing YouTuber, Premiere Gal. “Firefly, like any of these tools, works best when you stack it intelligently to get the results you’re after. Don’t let the system dictate your creative output, and definitely avoid the ‘one-prompt fallacy’.”

He continues, “When I’m using Firefly, I usually start with style and composition references for image generation. I try not to work off a single, straight prompt. Once I get an output that feels right, I bring it into Photoshop to make more detailed changes using Firefly’s tools in Photoshop. This gives me more control. After I finish those edits, I save the image and move on to a new idea. The real power of these tools is in how much I can ideate before committing to a single style.”

“After that, I take the image into Firefly Video and add the movement we want, using it as a reference image. Then I bring everything into the edit to see how it’s coming together. If I need new angles or feel the look isn’t working, I just start the process over to expand my options. These tools are great for quick ideation. I wouldn’t expect any of them to be part of the final output on their own. AI is here to boost our workflow, not define it.”

For Brian, the “perfect example” of how to use AI tools effectively for animation right now is when they’re used in tandem with compositing – either inserting AI elements into real footage or vice versa, compositing real elements into an AI video. And for this, an experienced and talented compositor is required to turn the generative AI pieces into great work.

“It’s one of those parts of VFX and filmmaking where, when it’s done perfectly, you don’t even notice it. On the flip side, when you can tell something used AI to generate the work, that’s when it has failed.”

One of the keys to this seamless use of generative AI is utilising good reference images, reducing the appearance of what Brian calls ‘hallucinations’ – unwanted visual anomalies in the programme’s output. And while these references can be sourced from anywhere – depending on your licensing needs – he says that Adobe is the leader in “ethical generative AI” thanks to its models trained on licensable content.

“One of my favourite tricks is using the generative extend tool in [Adobe] Premiere Pro to fix clips that were almost perfect,” he shares. “It’s an exceptionally well-done tool that lets you trim out hallucinations in generative AI renders and re-generate just those problematic parts of the clip. It’s incredibly versatile, and I’d recommend everyone check it out.” 

“That said, sometimes you just have bad luck, and the tools spit out some wild, LSD-inspired mess. When that happens, there’s not much you can do except change the reference image and keep prompting until you get the result you’re looking for… Again, the key is not to rely entirely on a prompt but to use the tools to guide and refine your results.”

For commercial projects at Versus, generative AI is mostly being used in the pitch phase. However, for several feature-length documentaries under development, the studio is also using these tools to ideate and experiment with archival footage and production planning. “So many times, you would just get a 25-minute edit with 10 minutes of title cards that said ‘GFX HERE’ or ‘PICK UP SHOOT HERE’ – now, the editors are using their decades of experience and generative AI to put together rough cuts with exceptional levels of craft that act as a blueprint for the next team that takes over.”

“This also saves us a lot of time and money that can be better spent on making a superior film, since we won't be round-tripping between edit and VFX very much,” he adds. “The editor is able to start with delicately defined rhythms and pacing for the animation or VFX team to work with.”

Versus also uses AI when creating storyboards, allowing the teams to ideate faster and bring more concepts to the stage when pitching. “So we can present ideas we know will get the client excited, while making it clear this is just the starting point of our collaboration.”

Above: Brian showing storyboards for an AI animation on Premiere Gal's YouTube Channel

“That said, let’s be honest – with AI, anyone can now create amazing boards for a pitch,” Brian continues. “That’s why it’s so important for brands, agencies and entertainment networks to be cautious about how studios promise to deliver on those pitches, especially in the animation and VFX space. There needs to be a strong creative leader behind the work, someone with an unwavering commitment to quality control and the craft of filmmaking. Otherwise, you risk ending up with something embarrassing for the brand because they handed the project to an ‘AI company’ instead of a creative studio that truly understands storytelling and VFX.”

As for the future of AI in post production workflows, Brian believes that the next step is all about improving integration. Describing the UX for many AI tools as “pretty brutal”, he would like to see generative AI built directly into the platforms the industry already uses, and at a more reasonable cost than the “downright ridiculous” current pricing structures.

“I think we’ll see these big software companies really nail integration,” he predicts. “Putting these tools right into your timeline or canvas, so to speak. That’s incredibly exciting because it means greater adoption. And with that, we’ll see artists pushing the boundaries of what this technology can do. Once again, it’ll be clear that great work comes from the artist using the tech – not the tech itself.”

In this same vein, a phrase which Brian has said before is that there is ‘no one-click solution for great creative work’. But with the technology improving by the day, could that eventually become a reality?

“To answer your question: no, I don’t think a one-click solution for great creative work will ever exist,” he says. “Without getting too philosophical, it might be possible to make a whole lot of mediocre work with a single click – but that’s a different story. The real danger isn’t the technology – it’s the acceptance of ‘good enough’ winning out in the minds of clients as opposed to the pursuit of ‘great’.” 

And Brian and the craftspeople at Versus believe in that pursuit – using generative AI as just a tool to enhance their work and with purpose – and, as he says, they’ll keep wrestling that “1000lb gorilla” to protect it.