How AI is improving the internet for the visually impaired

We’re excited to bring Transform 2022 back in person on July 19 and pretty much July 20-28. Join AI and data leaders for insightful conversations and exciting networking opportunities. Register today!

There are almost 350 million people worldwide with blindness or some other form of visual impairment who need to use the internet and mobile apps just like everyone else. Yet they can only do this if websites and mobile apps are built with accessibility in mind — and not as an afterthought.

The problem

Consider these two sample buttons that you can find on a web page or mobile app. Each has a simple background, so they look similar.

In fact, they are a world apart when it comes to accessibility.

It’s a matter of contrast. The text on the light blue button is low contrast, so to someone with a visual impairment such as color blindness or Stargardt’s disease, the word “Hello” may be completely invisible. It turns out that there is a standard mathematical formula that defines the correct relationship between the color of text and the background. Good designers know about this and use online calculators to calculate those ratios for each element in a design.

So far, so good. But when it comes to text on a complex background like an image or a gradient, things start to get complicated and useful tools are rare. Until today, accessibility testers had to manually check these cases by sampling the background of the text at certain points and calculating the contrast ratio for each of the samples. Besides being cumbersome, the measurement is also inherently subjective, as different testers may sample different points within the same area and come up with different measurements. This problem – laborious, subjective measurements – has held back digital accessibility efforts for years.

Accessibility: AI to the rescue

It turns out that artificial intelligence algorithms can be trained to solve these kinds of problems and even automatically improve them as they are exposed to more data.

For example, AI can be trained to do text summaries, which is useful for users with cognitive impairments; or to do image and facial recognition, which helps people with visual impairments; or real-time closed captioning, which helps the hearing impaired. Apple’s VoiceOver integration on the iPhone, whose main use is to speak email or text messages, also uses AI to describe app icons and report battery levels.

Guiding Principles for Accessibility

Wise companies rush to comply with the Americans with Disabilities Act (ADA) and give everyone equal access to technology. In our experience, the right technology tools can make that much easier, even for today’s modern websites with their thousands of components. For example, the design of a site can be scanned and analyzed through machine learning. It can then improve accessibility through facial and speech recognition, keyboard navigation, audio translation of descriptions, and even dynamic adjustments of picture elements.

In our work, we have found three guiding principles that I believe are crucial for digital accessibility. I’ll illustrate them here by how our team, in an effort led by our data science team leader Asya Frumkin, solved the problem of text on complex backgrounds.

Example complex backgrounds. Image by author

Break the big problem into smaller problems

Looking at the text in the image below, we see that there is some sort of readability problem, but it’s hard to quantify in general, just looking at the whole sentence. On the other hand, if our algorithm examines each of the letters in the sentence individually – say the “e” on the left and the “o” on the right – we can more easily see for each of them whether it is readable or not.

If our algorithm continues to run through all the characters in the text in this way, we can count the number of readable characters in the text and the total number of characters. In our case, there are a total of four legible characters out of eight. The resulting fraction, with the number of legible characters as the numerator, gives us a readability ratio for the total text. We can then use a pre-agreed threshold, for example 0.6, below which the text is considered illegible. But the point is, we got there by doing operations on each part of the text and count from there.

Example of complex background solution. Image by author

Reuse existing tools where possible

We all remember Optical Character Recognition (“OCR”) from the 1970s and 1980s. Those tools were promising, but eventually became too complex for their originally intended purpose.

But there was a part of those tools called The CRAFT (Character-Region Awareness For Text) model that showed great promise for AI and accessibility. CRAFT assigns each pixel in the image to its probability of being in the center of a letter. Based on this calculation, it is possible to create a heat map in which high-probability areas are painted in red and low-probability areas in blue. On this heatmap you can calculate the bounding boxes of the characters and cut them out of the image. Using this tool, we can extract individual characters from long text and run a binary classification model on each of them (as in #1 above).

CRAFT example. Image by author

Find the right balance in the dataset

The model of the problem classifies individual characters in a straight forward binary way – at least in theory. In practice, there will always be challenging real-world examples that are difficult to quantify. Complicating matters even more is the fact that each person, whether visually impaired or not, has a different perception of what is legible.

Here’s a solution (and the one we took) to enrich the dataset by adding objective tags to each element. For example, each image can be provided with a reference text on a fixed background prior to analysis. That way, when the algorithm is executed, it has an objective basis for comparison.

For the future, for the greater good

As the world continues to evolve, every website and mobile application must be built from scratch with accessibility in mind. AI for accessibility is a technological capability, an opportunity to get off the sidelines and participate, and an opportunity to build a world where people’s problems are understood and considered. We believe that the solution to inaccessible technology is simply better technology. That way, making websites and apps accessible is an essential part of building websites and apps that work, but this time for everyone.

Navin Thadani is co-founder and CEO of proven

DataDecision makers

Welcome to the VentureBeat Community!

DataDecisionMakers is where experts, including the technical people who do data work, can share data-related insights and innovation.

If you want to read about the latest ideas and up-to-date information, best practices and the future of data and data technology, join us at DataDecisionMakers.

You might even consider contributing an article yourself!

Read more from DataDecisionMakers

Shreya Christina
Shreya has been with for 3 years, writing copy for client websites, blog posts, EDMs and other mediums to engage readers and encourage action. By collaborating with clients, our SEO manager and the wider team, Shreya seeks to understand an audience before creating memorable, persuasive copy.

More from author


Please enter your comment!
Please enter your name here

Related posts


Latest posts

3 Ways You Can Support Hispanic Businesses During Hispanic Heritage Month

National Hispanic Heritage Month is a time to celebrate the rich culture and immense contributions of Hispanic and Latino Americans. This year, as we...

The Nostalgia Factor: Why Retro Gaming Continues to Thrive

In the fast-paced world of modern video games with their stunning graphics, lifelike simulations, and complex narratives, there's a distinct charm in returning to...

The Rise of Cross-Platform App Development in Australia

In the ever-evolving landscape of app development, the need for efficiency, cost-effectiveness, and wider reach has given rise to a significant trend: cross-platform app...

Want to stay up to date with the latest news?

We would love to hear from you! Please fill in your details and we will stay in touch. It's that simple!