A common test to evaluate people's implicit bias has been 'oversold,' U of T researcher says

For decades, the psychology community has been using the Implicit Association Test (IAT) to determine if someone harbours hidden prejudices.

The tool was developed at Harvard University and is supposed to reveal hidden truths about the test-takers, including whether they suffer from implicit bias that results in unintentional racism. Freely available to anyone through Project Implicit, the test has been used in countless academic studies and has been employed by companies and institutions in their implicit bias training sessions. 

But Ulrich Schimmack, a psychology professor at the University of Toronto Mississauga, says his research suggests the test can’t delve into our unconscious and reveal our hidden prejudices. 

“[It’s] basically a roulette wheel that comes up with a random number,” he says.

He adds the test became popular after it was published some 20 years ago because psychologists had been fascinated with the unconscious but had no scientific method to measure it. The promise of the IAT was that it would “finally provide a scientific window to the unconscious.”

Schimmack, who studies happiness and well-being, was curious if he could use the same method in his research. If he could, he would no longer have to rely solely on self-reporting, where it’s more difficult for people to admit their lives aren’t ideal. He and his team undertook a study to see if the measure was valid.

“We found our happiness IAT was giving us random results. It wasn’t useful at all to measure a person’s happiness,” Schimmack says. “Then you wonder, if it’s not working for this, when does it work and what does it really measure?”

He published those first findings 10 years ago and continued his research without the use of IAT. More recently, though, he noticed another decade had passed and the test was still being widely used and taken at face value, so he took a closer look at the tool.

Those who take an IAT test are required to categorize concepts as quickly as possible. In the racial bias test, participants are asked to link people of European descent with words related to “bad” concepts and people of African American descent with words related to “good” concepts, and vice versa.  Based on speed and accuracy, a score is provided and a person’s implicit bias is determined.

Schimmack says many things can influence how fast people can do the task, which introduces other sources of variation that have nothing to do with attitude. The scores can also be different from one’s actual behaviour.

“Twenty years of research produced very little evidence that the IAT test predicts any real-world behaviour,” Schimmack says. “On top of that, some of the articles that claim it does, on close inspection, fail to show that.”

For example, one study claimed the IAT could predict some people would not vote for former U.S. President Barack Obama due to anti-Black implicit bias. But on closer inspection, Schimmack says they found that was not true.

Schimmack says his colleague also found that, in simulations, IAT scores could not predict if a police officer was more likely to shoot a Black suspect or a white suspect.

Others have also criticized the authors of the IAT for making bold claims without demonstrating that their measurement predicts any real-world behaviour of discrimination, Schimmack notes.

Racial bias is a reality, Schimmack says, but the problem is too many discussions of the issue are based on research findings that rely on flawed IAT measures. For example, some argue implicit bias training is not useful because it doesn’t change IAT scores. But if IAT scores are not valid in the first place, they are not likely to be effective evaluation tools. 

“(IAT) shouldn’t be used as a criterion to evaluate whether implicit bias training was successful. It shouldn’t be used in a training context,” Schimmack concludes, adding that it’s questionable, if not unethical, for a test to claim to provide scientific feedback if that’s not the case.

“For the most part it has been oversold and it offers very little,” Schimmack says.