Robots in the wild: U of T's Florian Shkurti on overcoming 'edge cases' in machine learning

The technology behind self-driving cars has been racing ahead – and as long as they are cruising along familiar streets, seeing familiar sights, they do very well.

But the University of Toronto’s Florian Shkurti says that when driverless vehicles encounter something unexpected, all that progress can come screeching to a halt.

He offers the example of a self-driving car that is following a large truck on a winter road.

“There’s a wind gust – and now the snow is coming at you, so you can’t see anything,” says Shkurti, an assistant professor in the department of mathematical and computational sciences at U of T Mississauga who runs the Robot Vision and Learning (RVL) lab. “And suppose your LIDAR (light detection and ranging system) misperceives the snow as an array of objects, so it thinks there are a million small objects coming at the car.”

Shkurti’s research extends far beyond self-driving cars to autonomous systems in general. How do they learn? How we can make them learn better? How they can successfully navigate complex environments at the service of humans? That includes making sure that robots can handle so-called “edge cases,” like the snowy truck example – cases where the robot “comes across a rare scenario, for which there’s little or no training data.

“Then you have to either collect more data, or you have to accept that there will be these rare events that your perception system won’t recognize,” Shkurti says.

Simulation is an important training tool. Self-driving cars, for example, can be trained on simulated roads and highways before they’re let loose on actual city streets. But scalability remains a challenge. If an autonomous system has to be specially trained for every possible scenario it might encounter, progress will be haltingly slow; there will be no way to take what’s been learned from one scenario and scale it up so that the system can handle more general cases.

In an ideal world, Shkurti says, a robot could learn similar to the way a human would.

Take, for example, robots that help scientists collect data underwater – an effort Shkurti has been involved with for several years. A human diver “has to collect data manually, one data point at a time, one location at a time,” Shkurti says. “It’s painstaking work; it’s not scalable.”

An autonomous robot, on the other hand, could take over the data collection process if it’s capable of maneuvering underwater and equipped with a camera and other sensors. “If the robot could understand what it’s doing – if it has a model of what the scientist thinks is important to pay attention to in a particular environment – then the robot could collect data on behalf of the scientist.”

Such an approach has many benefits, according to Shkurti: It’s much cheaper to deploy additional robots than to train more scientists; and it frees up the scientist to look after higher-level tasks. “The scientist can give the robot some hints as to where to collect the data – but then the robot can take care of the rest,” he says.

Shkurti, who did his undergraduate studies at U of T before earning his PhD in computer science at McGill in computer science, was hired by U of T in 2018. He recently received a Connaught New Researcher Award for a project titled “Robotics and Machine Learning in the Wild: New Directions in Automated Environmental Monitoring.”

Hey says that while everything about computer science fascinates him, the field of robotics holds special appeal. 

“Robotics lets you play in different ‘playgrounds,’ like control, perception, and machine learning,” he says. “It allows you to examine these different fields, and I really valued that – and I still value it.”

As for the lofty philosophical questions that sometimes crop up when people talk about advanced computer systems – such as whether machines could learn to “think” – Shkurti prefers to stay focused on the science. Machines can reason, he says, and they can try to act optimally as they strive to achieve their goals.

“If that’s thinking, then they’re doing it,” he says. “But I don’t spend very much time worrying about ‘consciousness.’ I have enough other things to worry about.”