Pokémon GO craze shows that augmented reality is hitting its stride, U of T experts say

It's not even available in Canada yet, but Pokémon GO, the new augmented reality (AR) app from Niantic Labs, is dominating watercooler conversations, social media and headlines both here and abroad.

The GPS-enabled game for Android and Apple smartphones is based on the Nintendo Pokémon characters. Players wander around searching for Pokémon, which the game triggers at semi-random physical locations. When they find one, it activates the phone’s camera and shows them the Pokémon.

While Pokémon Go is a new phenomenon, U of T researchers have been working with augmented reality for years. U of T News spoke to computer science professor Eugene Fiume and electrical and computer engineering professor Steve Mann about Pokémon GO and the future of augmented reality.

Eugene Fiume

What is augmented reality (AR)?
As the name would suggest, augmented reality (AR) is a computer-aided way to augment the physical world by inserting computer generated information into the view of a real scene. Normally, this is done by streaming one channel or more of video of the world into a device and then inserting computer synthesized imagery into those channels just in time for interactive viewing.  For the most part, augmented reality is about continuous visual phenomena, though one could imagine augmenting sound, images, touch, and one day perhaps even taste.  Common uses for AR include museum exhibits, automatic sign translation, interactive games, mapping, defect analysis, artistic tools, see-through displays on cars, etc.  AR should be distinguished from virtual reality (VR), whose purpose is to provide an immersive synthetic approximation of reality whose use generally is isolated from "real" reality. AR blends the real and the virtual, while VR replaces the real by the virtual.  These distinctions will in time blur.

How does it work?
The main technology needed for AR is the ability to recognise some aspects of how a user is moving and the position and direction of the camera. If I know where a person is and how he or she is holding the camera, I can then approximate (with a little modelling) apparent features in the real scene, such as walls, roads, and posts. This kind of information comes from accelerometers and position sensors: the same things that our fitness applications use to track our distance and speed.  As well, some image processing, computer vision and AR is often used to recognise objects or just basic geometry such as a large flat region in the incoming video stream. Then computer graphics jumps in to generate synthetic images (signs, Pokémon characters, narratives, animation) and blends them into the video stream just before it appears on the viewer's screen.

Augmented reality apps have been around for several years now. Why has Pokémon GO suddenly become so popular?
AR (and its cousin, virtual reality) have been around for a long time. Steve Feiner from Columbia University did AR walking around with a huge backpack, doing a lot of the stuff that is becoming commonplace on a handset today. But AR is only now hitting its stride. Generally as a technology this has to do with Moore's Law combining with devices such as mobile handsets that integrate sensors and functions such as cameras, GPS, accelerometers, image processing, graphics, etc. So now we're starting to see AR become commonplace. But the problem is that the fundamental functionality that AR needs such as registration of live video to a synthetic character or sign is quite hard to accomplish for austere applications that require accuracy.  And now we come to Pokémon GO. Here we have a perfect combination of an application that combines nostalgic characters, easy graphics, a relatively low need for accuracy, mobility, and community. 

Will its popularity be good for augmented reality apps, or will it lead developers to focus on apps with a narrow commercial potential?
I think its popularity will be good for AR overall, as long as people understand that AR has a very broad scope, and that there are many unsolved problems with this technology (as well as with Virtual Reality).

What other AR apps would you recommend to people who’ve become interested because of Pokémon GO?
There are some terrific little applications that do things like sign translation (Google Translate), Crayola Colo[u]r, Wikitude. There are some beautiful star-chart semi-AR applications that allow you to explore the night sky.

Pokémon GO is an entertainment app. Can you give examples of serious uses for augmented reality?
AR is already part of the landscape in high-tech situations, for example, BMW's Mini augmented glasses, which you can see here. I particularly like "austere" AR applications that are being developed now to do defect analysis in cars or boats, or to do 3D/AR training. See, for example, www.ngrain.com.

What is happening at U of T in terms of AR research?
Karan Singh, in our Dynamic Graphics Project lab, is doing work on VR and has a startup called JanusVR. Steve Mann, in Electrical and Computer Engineering, has been doing various forms of ground-breaking AR for many years.

A lot of the media coverage of Pokémon GO​ is about dangerous consequences – players hurting themselves because they’re not paying attention; players being lured to remote locations where they’ve been robbed, etc. But does Augmented Reality pose any dangers?
Looking at the world through a narrow portal is potentially dangerous, but certainly AR is not the only possible concern. The problem here is one of attentional focus as well as "change blindness".  The former reflects ongoing research that humans are in fact terrible multi-taskers. The latter points to the fact that in many situations we do not detect changes in our visual field. There are some very graphic examples of this, but the nicest one I've seen is a commercial (!) for ŠKODA.  In fact, AR systems could well be better "augmented" to detect environmental changes.

Steve Mann