Coming to a cinema near you: U of T computer science research

Comic book-like sketches of a film, known as storyboards, are common practice for Walt Disney Productions and were famously used by director, Alfred Hitchcock, to pinpoint suspenseful set details.

Storyboard artists are mainstays of pre-production, and getting the director’s ideas communicated on set, before filming takes over.

But how do you sketch a 3D film?

“Live action 2D films are actually 3D, but we think of it of as a flat image within a frame,” explains Rorik Henrikson, a PhD candidate in the department of computer science.

“With 3D, or what are technically stereoscopic 3D films, you’re looking through a window, into a 3D space. So it’s not only what you are seeing, but where things are placed. For artists and directors there is very little planning for stereoscopic 3D right now.” 

Enter Henrikson, who took his interest in early stage film and television planning to create Storeoboard, 3D sketching software for storyboarding. He brought his early research to Immersed, the Immersive Technology Alliance (ITA) conference for technology makers, in gaming, film, consulting and academia.

(Storeoboard screenshot image below courtesy of Rorik Henrikson)

Cue Canadian director and filmmaker, Dylan Pearce. 

“I was looking everywhere for 3D storyboarding software,” said Pearce, who met Henrikson at the conference and was able to use Storeoboard in the planning of 40 Below and Falling, the first 3D romantic comedy. “I wanted to use it to connect with the other departments working on the film, so they could understand how 3D was going to be used.

“We explored what would be in front of the screen, on the screen, and behind it, and how this would affect the story.” (Read more about 40 Below and Falling)

Much like the musical score, the audience’s emotional connection to the story can be increased with depth, known as the “depth score”. 

“It can take several seconds for your eyes to adjust to new depth,” says Henrikson. “So if you have something moving from in-front of the screen, to behind, and back to in-front, it can be uncomfortable. The director has to be careful, or your eyes are going to be fighting constantly. You’ll get headaches, and you’ll feel nauseated.” 

Storeoboard itself acts like a regular pen and paper note pad, no need to save, as the work is automatically kept in a series of sheets. 

“To keep the workflow, it had to be very natural. The current version has a slider on the side, so you can select depth and the sheet you want to draw on. You’re always drawing on the surface, so drawing stereoscopically feels comfortable, visually,” says Henrikson, who pairs the software with common red/cyan anaglyph 3D glasses, so the effect can be seen. 

In addition to storyboarding, filmmakers use previsualization, a form of rough 3D animation to prepare a scene, especially for high-cost shots, like the public closure of New York’s Times Square. 

“The difference between storyboarding and previsualization is that a storyboard is done by only one or two artists,” says Henrikson. 

“Storyboarding is very quick, iterative, usually done on a paper, or computer sketch pads. I was looking to bridge the gap – get the traditional artists involved in previsualization side, because it’s very quick and iterative, and there’s a lot of value there.” 

While it may seem that Hollywood is cashing in on the resurgence of 3D, the technology is now here to support it. 

“In post-production, we can reshape images so that they’re easier on the eyes. We know more about the planning and presentation, and the glasses used to view stereoscopic 3D film, are all designed better,” says Henrikson, who would like to see Storeoboard become a startup.  

“I feel 3D is no different than lighting or sound design – it's another tool used to tell your story,” says Pearce, who is working on two more 3D productions. “I think the industry will start to see a change in perception with 3D films in the next few years, with more directors using it as a part of their storytelling tools.”

Nina Haikara is a writer with U of T's department of computer science