WinTOR’s on its way: U of T partnership to train self-driving cars to handle tough winter conditions

Companies around the globe are racing to create fully autonomous vehicles that can handle anything the road throws at them. But, according to the University of Toronto’s Steven Waslander, there’s one scenario that hasn’t yet received nearly as much attention as it deserves: winter. 

“Winter conditions aggravate the remaining challenges in autonomous driving,” says Waslander, an associate professor at the U of T Institute for Aerospace Studies in the Faculty of Applied Science & Engineering.

“Reduced visibility limits perception performance, and slippery road surfaces are a big challenge for vehicle control.”

To drive safely in all conditions, including winter, Waslander says autonomous vehicles need to fully observe their surroundings despite limits to their sensor range to get advanced warning of challenging situations and to react quickly to changing conditions.

Along with fellow U of T Engineering researchers and members of the U of T Robotics Institute – including Timothy Barfoot, Jonathan Kelly and Angela Schoellig – Waslander is leading a new partnership that will address these challenges by bringing together the best minds from academia and industry. The project, called WinTOR: Autonomous Driving in Adverse Conditions, is a new collaboration that aims to transform Toronto into a global hub for research and development related to autonomous driving in winter. Corporate partners include leading companies in the autonomous vehicles sector such as General Motors Canada, LG Electronics, Applanix and Algolux.

It's one of six projects from U of T to receive support from the Ontario Research Fund. Five  of the projects are led by researchers from the Faculty of Applied Science & Engineering, while the sixth is led by a researcher at the Faculty of Dentistry. (See the full list below.)

“With this investment, the Ontario Research Fund is supporting important work of U of T researchers that will benefit all Canadians,” says Christine Allen, associate vice-president and vice-provost, strategic initiatives. “And it’s exciting to see that two of the initiatives to receive funding, focused on advanced robotics and innovative mobility, are among U of T’s key areas of strategic, interdisciplinary focus.

“Through these Institutional Strategic Initiatives we’re mobilizing interdisciplinary cross-divisional research and collaboration to address societal challenges. We’re committed to developing these initiatives to a scale that can attract support from government, philanthropic or industry sources – and WinTOR is a terrific example.”

The WinTOR team already has a track record of success. Last year, Waslander and his collaborators published the Canadian Adverse Driving Conditions (CADC) dataset. Created using the Autonomoose, a self-driving vehicle designed by Waslander and his team, the open-source data record real winter driving conditions on roads in southwestern Ontario.

The valuable dataset is already being used by researchers from around the world to train new AI software. It joins a long list of research accomplishments from the project’s research team, whose combined expertise covers the full extent of autonomous driving perception and planning domains, including object detection and tracking, robust state estimation and calibration, localization and mapping, prediction and planning and safety-critical learning control.

Read more about autonomous vehicle research at U of T Engineering

All four professors are advisers to the aUToronto team, a group of undergraduate and graduate students who have designed and built a self-driving electric vehicle called Zeus. The U of T Engineering team has placed first in the international AutoDrive Challenge in each of the four annual competitions to date.

The new partnership is supported by more than $12 million in funding from a variety of sources. In addition to the Ontario Research Fund, that includes funding from the Natural Sciences and Engineering Research Council of Canada, as well as direct and in-kind donations from the project partners. More than 20 individuals will work as part of the team, including graduate and undergraduate students, professors and engineers.

“We are excited to understand how we can apply the collaborative research under this program to real-world scenarios,” says Louis Nastro, director, land and autonomous vehicle strategy at Applanix. “It gives us an opportunity to attract highly talented individuals with the experience needed to join our team, and helps Canada establish itself as a leading provider of advancement in autonomy.”

“Algolux’s mission is to solve the issue of computer vision robustness in harsh driving conditions, a fundamental problem not effectively addressed by current approaches,” says Felix Heide, co-founder and chief technology officer of Algolux. “As a Canadian company, we are thrilled to bring our expertise to this project and continue to advance the state-of-the-art in perception technologies.”

“We are excited and proud to be a partner of this initiative,” says Kevin Ferreira, director of the LG Electronics Toronto AI Lab. “It is challenging to drive social impact with game changing technology in a fast-moving industry such as autonomous driving.

Partnerships and collaborations, such as this initiative, are an effective strategy to make contributions to the research community, and to deliver impactful applications that improve safety while driving.”

The WinTOR project is divided into three broad themes:

• Sensor filtering for object detection: New ways of analyzing the data from sensors such as visual cameras, radar and lidar will help to separate the signals that represent real objects from the noise caused by falling or blowing snow. Strategies will include both pre-processing techniques and improved artificial intelligence algorithms trained to be aware of the limits of their own performance.
• Sensor fusion, localization and tracking: While today’s self-driving cars can reliably determine where they are in relation to their surroundings, the techniques they use begin to break down under adverse driving conditions. The team will leverage new algorithmic strategies in vision and lidar registration, as well as new sensing options, such as ground-penetrating and automotive radar, to make localization algorithms more resilient in adverse conditions.
• Prediction, planning and control: self-driving cars of the future will need to change the way they drive in response to winter hazards. For example, they might take a slightly different path to avoid a snowdrift or slow down when driving over a section of road that their sensors have perceived as particularly slippery. They will learn the implications of adverse weather on the vehicles around them and be able to assess the increased uncertainty of outcomes, enabling them to plan actions that can be executed reliably in winter conditions.

As ambitious as the current plan is, the team is just getting started. They are leveraging their network – including the U of T Robotics Institute, one of more than a dozen institutional strategic initiatives at the university – to grow their team.

“We continue to seek additional faculty, partners and funding to grow the effort,” says Waslander. “We have many more ideas to work on, from multi-hypothesis prediction and interaction planning to attentive perception and explainable, efficient AI for autonomous driving.” 

Here is the full list of U of T researchers who are leading projects that received Ontario Research Fund support:

• Konstantinos N. Plataniotis, Faculty of Applied Science & Engineering, Transforming Pathology Using Artificial Intelligence to Improve Patient Outcome and Hospital Efficiency
• Philippe Lavoie, Faculty of Applied Science & Engineering, Characterization, Prediction and Control of Flap Noise Sources from Aircraft
• Eric Miller, Faculty of Applied Science & Engineering, iCity 2.0: Urban Data Science for Future Mobility
• Patrick Chang Dong Lee, Faculty of Applied Science & Engineering, Innovative and Cost-Effective Micro/Nanocellular Foaming Technology for Sustainable Lightweight Applications
• Steven Waslander, Faculty of Applied Science & Engineering, All-Weather Autonomy: Securing Ontario’s Leadership in the Self-Driving Revolution
• Christopher McCulloch, Faculty of Dentistry, The Fibrosis Repair Team