​U of T startup draws on AI, linguistics to power facial animation in video games

A University of Toronto startup is spurring the success of one of the best-selling video games of the past year – the dystopian action role-playing game Cyberpunk 2077.

Growing out of research in the department of computer science, JALI Research has developed a suite of tools that power the game’s hyper-realistic facial animations, allowing characters to convincingly deliver dialogue in different languages.

Multi-language animation capacity was touted as a novel feature of Cyberpunk 2077, which sold over 13.7 million copies globally in its first month, according to game developer CD Projekt Red.  

“Having brand new animation that responds to recorded dialogue from voiceover actors across 10 languages – and not just dubbing over English animation – means you’re watching animation that matches up to what you hear,” said Sarah Watling, the startup’s CEO.

“That was one of the things that CD Projekt Red believed would have a huge impact. They were going to be able to grow new retail markets by giving a localized experience to players.”

Founded in 2016, JALI Research uses AI to accurately map phonemes – the smallest elements of sound that can be isolated – to visemes, the corresponding mouth shapes that we see on a person’s face as they’re making those sounds.

Watling says JALI’s technology helps bridge the gap between actual gameplay and the more cinematic, pre-constructed scenes that are used to tell the game’s story. “The person who’s playing the game is spending most of their time in gameplay. So, the improvements we’re able to deliver improve that aspect of the game as well,” she says.

The company’s roots can be traced to research by Chief Technology Officer Pif Edwards while he was a PhD student in the department of computer science in the Faculty of Arts & Science. Edwards noticed that existing automated dialogue systems weren’t matching the nuance and complexity of human speech – so he set about developing an alternative.

Edwards was the lead author of a 2016 paper that introduced an “animator-centric viseme model for expressive lip synchronization.” The paper’s co-authors included Chris Landreth, an Academy Award-winning animator who has served as distinguished research artist-in-residence at U of T’s Dynamic Graphics Project; Eugene Fiume, professor of computer science and dean of the Faculty of Applied Sciences at Simon Fraser University, who earned his graduate and doctoral degrees at U of T; and Karan Singh, professor in U of T’s department of computer science.

The resulting software, which makes use of a facial animation rig to help map phonemes to visemes, was dubbed “JALI” – a combination of “jaw” and “lip,” the two anatomical features the paper said account for most variation in visual speech.

Watling joined the startup as CEO in 2020, the year the company announced that its software would be powering Cyberpunk 2077’s facial animations.

She says a key advantage of JALI’s technology is it enables studios to scale up facial animation at a relatively low cost. That’s in contrast to more expensive approaches such as keyframe and motion-capture animation, which are only financially viable for the biggest studios.

“Video-game worlds are becoming more and more expansive and immersive,” she says. “At the same time, gamers are pretty indiscriminate about the scale of the company. So, the expectation for greater immersion, more realism and higher production values still gets applied – regardless of whether you’re a two-person studio or a massive multinational studio.

“Our technology really provides the ability to capture ideal quality at scale so that the return on investment increases over time, enabling smaller studios to punch above their weight.”

She credits U of T’s expansive innovation ecosystem – particularly the UTEST program, which is currently taking applications for its next cohort – for helping the company take its research from lab to market.

“The UTEST program is a great hybrid of courses – some of them through MaRS and others through partnerships with various mentors who support businesses in the startup life cycle, including IP law firms,” she said. “You get to work closely with the leadership of UTEST on various aspects, whether it’s thinking through your problem space or value proposition, or improving your pitch.

“As well, the Innovations & Partnerships Office is so good at helping startups progress from that initial process of ideation and invention to a viable commercial path, navigating bureaucracy, legal obstacles and incorporation.

“We’ve found the experience has been incredibly supportive.”