When those who are blind encounter text, an object, or a landmark, how do they determine what is in front of them? Personal assistants, seeing-eye dogs, and canes traditionally have served this purpose. Now, Microsoft’s new AI-based app called Seeing AI presents another aid to overcoming this obstacle for the visually impaired.
On Wednesday, Microsoft unveiled the app as part of its overall AI strategy—which also includes using AI to uncover bias and address global challenges. The iOS app, previously available only on smart glasses, is now available for free on other devices—so anyone with an iPhone or iPad can download it. Microsoft has not yet revealed whether the app will become available on Android.
The app uses a camera plus an AI-powered vision recognition system to look at an object and describe relevant information about it to a user. For example, a person can point their phone at a menu, and Seeing AI will describe the offerings. Seeing AI is not meant for navigation, but it can detect objects directly in front of it—such as determining the text on a paper envelope, or scanning a nearby face, which it can memorize and store in a database.
Kathy Szinnyey, a library assistant at the Louisville Free Public Library, has been blind from birth, and said she “loves technology of any kind that helps me access the world.” Currently, Szinnyey uses a screen-reader and braille display, as well as tablet-like devices for reading and accessing the internet. She also has an iPhone, “because Apple finally woke up one day and decided that accessibility to blind folk was actually a really important thing,” she said, and she uses many apps that have been built to assist those who are visually impaired.
“It could be a game changer,” said Szinnyey. But without testing it, she said, it’s hard to know. “Between my GPS babbling at me, all the ambient sounds around me, and then this new Seeing AI giving me information, I don’t know if that would be sensory overload or not.”
To understand where this app fits into the overall landscape of tech for the blind, I spoke to Larry Skutchan, the director of the technology product research department for the American Printing House for the Blind, to find out what he thinks of the product.
“We’re all in!” Skutchan told TechRepublic. “Deep learning is exciting to everybody, but its application toward being able to recognize and describe images is really useful to somebody who’s blind. Not just identifying products, but guessing how old someone is in a picture.”
Seeing AI isn’t the only tool on the market. Google’s AI-based “Show and Tell” can identify images, and caption them, with 94% accuracy. And AI Polyvision is what the American Printing House for the Blind previously relied on.
But Skutchan calls Seeing AI “the most advanced app of its kind. It blows everything out of the water.”
While the app holds promise for those wanting to more quickly interpret nearby text and objects, some experts are skeptical about how much insight this app can offer.
Meryl Alper, assistant professor of communication studies at Northeastern University and a faculty associate at the Berkman Klein Center for Internet and Society at Harvard University, said she believes the app could be useful, even for people who are not visually impaired. “What is obvious to one person can be invisible to another,” Alper told TechRepublic.
However, Alper does not necessarily see this kind of tool as something that can provide knowledge. “While I think that this technology can open up new opportunities for those with visual impairments to engage in social interaction and participation, we shouldn’t conflate visual perception with ‘knowing,'” she said.
To illustrate this point, Alper raised the issue of bias in machine learning algorithms. “While the app can identify faces and guess emotional states, most of the data that facial recognition algorithms are trained on comes from white people, inherently skewing the reality that the app presents,” she said.
And, importantly, while the app may be useful to many, there is a huge population of the visually impaired that cannot yet access it.
“While the app is initially available for Apple’s iOS, about 90% of the world’s visually impaired live in low-income settings, according to the World Health Organization,” said Alper, “with an iPhone likely financially out of reach.”
Skutchan acknowledges that the app has limitations. “This is not going to replace a dog or a cane. It’s not going to replace braille,” he said.
“But any aid we can get to identify objects is a good thing. If you’re in a big, wide open area, and you’re disoriented, you can point the phone around and have it tell you ‘building,’ ‘door’—still, it’s not going to replace time-tested tools we’re all using.”