hitchBOT programmer uses speech recognition expertise to help people with dementia

North America’s romance with the open road has taken a lot of forms. From Jack Kerouac’s On the Road to Thelma and Louise, we’re familiar with the trope of the free-spirited traveller setting off in search of something.

We’re just not used to it being a robot.

A hitchhiking robot, to be more precise.

Meet hitchBOT, the creation of three Ontario researchers, including Frank Rudzicz of the University of Toronto’s Department of Computer Science and the Toronto Rehabilitation Institute of the University Health Network.

On July 27, hitchBOT’s creators dropped it off in Halifax, NS. (hitchBOT was purposefully not given a gender.) They programmed it to share its goal with its human helpers – getting to Victoria, BC – and three weeks later, the friendly robot made it. Along the way, it picked up tens of thousands of Twitter followers and became a media darling.

“It was a sociological experiment,” says Rudzicz, who explains that the robot was built on the cheap from supplies purchased at Home Depot – its arms are pool noodles and its torso is a plastic bin. Add some cute boots and the ability to converse, and hitchBOT was ready to go.

As an expert in creating speech recognition software, Rudzicz was responsible for giving hitchBOT a voice. Thanks to a GPS, hitchBOT knew where it was, so it could chat with its chauffeurs about any number of local topics. History of Manitoba? No problem. hitchBOT could also have free-flowing conversations, though sometimes a person’s accent or noise in the car would give the quirky traveller some trouble.

The idea took shape when hitchBOT’s creators met as part of the Graphics, Animation and New Media Networks of Centres of Excellence (NCE). The NCE program brings together multidisciplinary researchers to focus on issues that are critical to Canada and the world. Through NCE, Rudzicz met project leaders David Harris-Smith of the Department of Communication Studies and Multimedia at McMaster University and Frauke Zeller of the School of Professional Communication at Ryerson University.

Rudzciz (far left) and collaborators working on hitchBOT’s voice and facial expressions in a last minute “hack-a-thon” before setting the robot loose. (Photo courtesy of hitchBOT/Ryerson University.)

Though hitchBOT was conceived as a cross between an art project and a social experiment, a lot of what went into creating its language system feeds into Rudzicz’s ongoing research, which is about using speech recognition software to improve the lives of people with dementia.

He started out as a computer scientist interested in helping people with cerebral palsy (CP) communicate. “CP is a physical disorder,” he says. “Cognitively, people with CP are fine, but they have a hard time being understood because their tongues and mouths don’t obey their brain’s commands. We developed a recognition system for speech that was hard for humans to understand. The system knew how the physical aspects of speech go wrong for people with CP, so it did a good job understanding them.”

From there, he moved on to building robots to help people with cognitive disorders like Alzheimer’s disease (AD). “For people with AD,” he says, “it isn’t so much an issue of expressing themselves, like it is for people with CP.  People with dementia have trouble remembering words or forming grammatically rich sentences.”

Rudzicz has developed software that can pick out certain differences in the voices of older adults with cognitive difficulties.

“Currently, people with dementia go to their doctor every couple of months,” he says. “They do a couple tests of memory and cognition.”

A robot in the home, though, can get a much more accurate and nuanced sense of how a person is doing over time. Through conversation the person has with a robot, he says, “we can pick out slight differences in words that are being used, how complex the grammar is, tone of voice, and tell how the disorder is progressing.”

He is also working on developing robots that will remind people living with AD to do things like take medication or turn off an oven that’s been left on. The challenge here is adapting the robot to deal with the kind of speech challenges it will encounter when conversing with people with dementia. What happens when the patient gets confused? The robot has to adapt and respond – just like hitchBOT did when he encountered someone with an unfamiliar accent.

“All the same issues came up to some extent in the hitchBOT project. We were unconstrained – hitchBOT was out on the open road – so it was an early attempt at tackling these kinds of challenges.”

Rudzicz thinks part of the reason everyone was so charmed by hitchBOT is that we don’t have a lot of real world experience with robots. He hopes by changing that, he’ll make a difference in the lives of people with cognitive disorders.

“We know about robotics in machinery – big, massive robots constructing cars, for example. We know about robots from science fiction. But we don’t see them very much in our day-to-day lives. But this little robot having this dream of crossing the country? For some reason it touches people.”

Jenny Hall is a writer with the Office of the Vice-President, Research and Innovation at the University of Toronto.