Pixar Used AI to Stoke the Flames in ‘Elemental’

While some of those questions could be answered with good old-fashioned suspension of disbelief, Pixar’s animators thought the fire issue was a real conundrum, especially considering that one of their movie’s leads, Ember, was actually supposed to be made of the stuff. They had tools to make a flame effect from years of previous animations, but when you actually tried to shape it into a character, the results were pretty terrifying, a cross between Studio Ghibli’s Calcifer and Nicolas Cage’s Ghost Rider, but somehow harsher.

“Our fire fluid simulations are very naturalistic and they’re designed to mimic reality,” says VFX supervisor Sanjay Bakshi. With a character like Ember, Bakshi says, “it’s really important to concentrate on the performance of the face,” but the studio was having trouble balancing the dynamism of the fire with the character’s shape and sensibilities. Paul Kanyuk, a crowds technical supervisor at Pixar, says that at first crack, Ember looked like a ghost or even a demon. “It can look horrifying if it’s too realistic, like you actually have a human figure made of real pyro,” he explains.

Even if you can get the scary tamped down, Sohn says, you still have to craft something that’s recognizably fiery. “Fire naturally is so busy, but if you slow it down, it can turn into something that looks like a plasma,” he explains. “It was interesting to compare it to other anthropomorphized characters, because they’re all very fantastical and you can do anything with them. If you’re drawing an emotion, there is no one-to-one, but everyone knows what fire looks like.”

Basically, Sohn explains, to make Ember, every single shot of Elemental would need an effects pass, something that’s not only incredibly time-consuming, but also very expensive.

Fortunately, Kanyuk had an idea. He’d been working on crowd animation at Pixar since 2005, starting with Ratatouille, and always struggled with ways to make the clothes on big groups of people look right. While trying to solve the problem he’d gotten involved with the Association for Computing Machinery’s Siggraph, a community organization devoted to the advancement of computer graphics. Around 2016, he found some of the group’s research on using machine learning to hone cloth simulations and has been trying to master it ever since.

Elemental gave him an opportunity to apply what he learned.

Kanyuk enlisted the help of Disney Research Studios, who Pixar had worked with once before, on Toy Story 4. The lab, based in Zurich, specializes in researching how AI and machine learning can do things like make actors appear older or younger, or how to best recreate someone’s skin quality. “Many of us didn’t do machine learning until it started becoming prevalent recently, so we’ve kind of learned on the job,” says Kanyuk, “whereas the research coming out of the Disney lab—they live and breathe this stuff.”

He started meeting regularly with the Research Studios team and eventually they cracked the issue, recruiting a Pixar artist named Jonathan Hoffman to draw a set of swirly, pointy, and almost cartoonish flames the team dubbed the “fleur-de-lis.” The NST could combine them with the blobbier fire from the original simulation and—bam—you get the movement and intensity of fire tempered with just a bit of Pixar’s control and style.

“Once you apply a style transfer to naturalistic fire, you can actually start to direct its style and start to bring the artist’s hand into something that is otherwise not touchable,” says co-character supervisor Jeremie Talbot. “It was a real breakthrough to be able to say, ‘I see the size of her features and the shapes here, and I want to complement those shapes with my own style.’ It harmonized the look of Ember in a really unique way.”

“The requirements went up and up,” says Kanyuk, “so we ended up getting a 20X speed-up from when we started to when we ended up deploying it by tapping into the GPUs everyone at Pixar has on their computers. We figured out a way to virtualize the GPU and take half of it to use overnight, making the time to render a frame go from about five minutes to one second.”

It worked. Ultimately, Kanyuk and everyone else involved with Elemental was able to render the shots they needed. Pixar is still “scratching the surface” of what NST can do, he says, “but I’m very excited that we found a use case on Elemental that elevated the kind of imagery that we can create.”

For Sohn, it was an opportunity to make the movie look the way he wanted, while also making something that looked like nothing audiences had ever seen before. It symbolized, he says, one of the things he loves about Pixar: the meeting of art and technology, where the latter is a big part of the process, but only one element.

“It’s this coming together of left brain and right brain, and using technology as a tool to help express emotions,” Sohn says, “and in turn we can connect to tech, versus it feeling like some cold new thing.”