How AI is improving the internet for the visually impaired

There are almost 350 million people worldwide with blindness or some other form of visual impairment who need to use the internet and mobile apps just like everyone else. Yet they can only do this if websites and mobile apps are built with accessibility in mind — and not as an afterthought.

The problem

Consider these two sample buttons that you can find on a web page or mobile app. Each has a simple background, so they look similar.

In fact, they are a world apart when it comes to accessibility.

It’s a matter of contrast. The text on the light blue button is low contrast, so to someone with a visual impairment such as color blindness or Stargardt’s disease, the word “Hello” may be completely invisible. It turns out that there is a standard mathematical formula that defines the correct relationship between the color of text and the background. Good designers know about this and use online calculators to calculate those ratios for each element in a design.

So far, so good. But when it comes to text on a complex background like an image or a gradient, things start to get complicated and useful tools are rare. Until today, accessibility testers had to manually check these cases by sampling the background of the text at certain points and calculating the contrast ratio for each of the samples. Besides being cumbersome, the measurement is also inherently subjective, as different testers may sample different points within the same area and come up with different measurements. This problem – laborious, subjective measurements – has held back digital accessibility efforts for years.

Accessibility: AI to the rescue

It turns out that artificial intelligence algorithms can be trained to solve these kinds of problems and even automatically improve them as they are exposed to more data.

For example, AI can be trained to do text summaries, which is useful for users with cognitive impairments; or to do image and facial recognition, which helps people with visual impairments; or real-time closed captioning, which helps the hearing impaired. Apple’s VoiceOver integration on the iPhone, whose main use is to speak email or text messages, also uses AI to describe app icons and report battery levels.

Guiding Principles for Accessibility

Wise companies rush to comply with the Americans with Disabilities Act (ADA) and give everyone equal access to technology. In our experience, the right technology tools can make that much easier, even for today’s modern websites with their thousands of components. For example, the design of a site can be scanned and analyzed through machine learning. It can then improve accessibility through facial and speech recognition, keyboard navigation, audio translation of descriptions, and even dynamic adjustments of picture elements.

In our work, we have found three guiding principles that I believe are crucial for digital accessibility. I’ll illustrate them here by how our team, in an effort led by our data science team leader Asya Frumkin, solved the problem of text on complex backgrounds.

Break the big problem into smaller problems

Looking at the text in the image below, we see that there is some sort of readability problem, but it’s hard to quantify in general, just looking at the whole sentence. On the other hand, if our algorithm examines each of the letters in the sentence individually – say the “e” on the left and the “o” on the right – we can more easily see for each of them whether it is readable or not.

If our algorithm continues to run through all the characters in the text in this way, we can count the number of readable characters in the text and the total number of characters. In our case, there are a total of four legible characters out of eight. The resulting fraction, with the number of legible characters as the numerator, gives us a readability ratio for the total text. We can then use a pre-agreed threshold, for example 0.6, below which the text is considered illegible. But the point is, we got there by doing operations on each part of the text and count from there.

Reuse existing tools where possible

We all remember Optical Character Recognition (“OCR”) from the 1970s and 1980s. Those tools were promising, but eventually became too complex for their originally intended purpose.

But there was a part of those tools called The CRAFT (Character-Region Awareness For Text) model that showed great promise for AI and accessibility. CRAFT assigns each pixel in the image to its probability of being in the center of a letter. Based on this calculation, it is possible to create a heat map in which high-probability areas are painted in red and low-probability areas in blue. On this heatmap you can calculate the bounding boxes of the characters and cut them out of the image. Using this tool, we can extract individual characters from long text and run a binary classification model on each of them (as in #1 above).

Find the right balance in the dataset

The model of the problem classifies individual characters in a straight forward binary way – at least in theory. In practice, there will always be challenging real-world examples that are difficult to quantify. Complicating matters even more is the fact that each person, whether visually impaired or not, has a different perception of what is legible.

Here’s a solution (and the one we took) to enrich the dataset by adding objective tags to each element. For example, each image can be provided with a reference text on a fixed background prior to analysis. That way, when the algorithm is executed, it has an objective basis for comparison.

For the future, for the greater good

As the world continues to evolve, every website and mobile application must be built from scratch with accessibility in mind. AI for accessibility is a technological capability, an opportunity to get off the sidelines and participate, and an opportunity to build a world where people’s problems are understood and considered. We believe that the solution to inaccessible technology is simply better technology. That way, making websites and apps accessible is an essential part of building websites and apps that work, but this time for everyone.

Navin Thadani is co-founder and CEO of proven†