A rare sight... app allows the blind to 'see' photos
Visually-impaired users can now 'look' at pictures after Facebook unveiled a new AI tool, following a project led by a blind software engineer. Katie Wright reports on how social media is becoming increasingly accessible
For more than 15 years, screen readers have used text-to-speech technology to allow people who are blind or visually impaired to hear text on their computer screens. But, until now, deciphering images has been much more difficult. But, until now, deciphering images has been much more difficult.
Emojis are accessible with screen readers, because they have a Unicode definition attached, but photos aren't so simple - the sometimes hilarious results you get when using captionbot.ai, the app that tries to guess what's happening in your photos, can attest to that - which is why Facebook's latest unveiling is so significant.
Previously, a screen reader would just announce the word "photo" and the name of the person who shared it, but now "automatic alternative text" can recognise things like babies and beards, sports, all kinds of foods, and whether a landscape is snowy, mountainous or a beach (so you can be sure exactly what kind of holiday your friend is bragging about). Scroll over a photo of a forest, for example, and you'll hear "this photo may contain: outdoor, cloud, foliage, plant, tree".
Ten months in the making, the tool has been made possible because of developments in the social network's object recognition technology and is the first big project engineer Matt King has been part of at Facebook.
King, who lost his sight as a result of a degenerative condition called retinitis pigmentosa, says: "On Facebook, a lot of what happens is extremely visual. And, as somebody who's blind, you can really feel like you're left out of the conversation, like you're on the outside."
For the 285 million people in the world who are blind, or have severe visual impairment, the feature offers a lot more inclusivity.
They'll have access to around two billion photos that are shared on Facebook, Instragram and WhatsApp every day.
Driven by Artificial Intelligence, the system has been "taught" using millions of example images and tested thoroughly, and it will continue to get better, providing richer and more accurate descriptions as it grows.
Available initially on iOS screen readers in English, software engineer Shaomei Wu says that automatic alt text will be rolled out to other languages and platforms soon.
"While this technology is still nascent, tapping its current capabilities to describe photos is an important step toward providing our visually impaired community the same benefits and enjoyment that everyone else gets from photos."