Apple adds live captions to iPhone and Mac, plus more accessibility upgrades –

Apple has released a slew of new accessibility features for iPhone, Apple Watch, and Mac, including a universal live captioning tool, improved visual and audio detection modes, and iOS access to WatchOS apps. The new capabilities will arrive “later this year” as updates roll out to various platforms.

The most widely applicable tool is probably live captioning, already very popular with tools like Ava, which recently raised $10 million to expand its repertoire.

Apple’s tool will perform a similar function to Ava’s, allowing all spoken content a user encounters to be captioned in real time, from videos and podcasts to FaceTime and other calls. FaceTime, in particular, will have a special interface with a speaker-specific scrolling transcript above the video windows.

The captions can be activated through the usual accessibility settings and quickly toggled on and off or the pane into which they expand or contract. And it’s all done using the device’s built-in ML acceleration hardware, so it works when you have spotty or no connectivity and no privacy issue.

This feature could very well cut off the wings of independent providers of similar services, as often happens when companies implement first-party versions of traditional third-party tools, but it could also increase quality and competition. Having a choice between different carriers isn’t a bad thing, and users can easily switch between them if, as seems likely to be the case, Apple’s solution is best for FaceTime, while another, like Ava, could make it excel in other situations. . For example, Ava lets you save transcripts of conversations for later viewing – not an option for the Apple subtitles, but certainly useful in work situations.

Screenshots of Apple Watch apps on an iPhone and a transcript of a multi-person video call.

Image Credits: Apple

Apple Watch apps are getting improved accessibility on two fronts. First, there are some added hand gestures for people, e.g. amputees, who struggle with the finer interactions on the small screen. A stack of new actions available through gestures like a ‘double pinch’, which let you pause a workout, take a photo, answer a phone call, and so on.

Second, WatchOS apps can now be mirrored to iPhone screens, where other accessibility tools can be used. This is also useful for anyone who likes the Apple Watch’s smartwatch-specific usage scenarios, but struggles to interact with the device on their own terms.

Existing Magnifier and Sound Recognition tools also get some new features. Magnifier’s “detection mode” normally lets the user know if a person or anything readable or writable is directly in front of them: “person, 1.5 meters ahead”. Now it has a special “door detection” mode that details those all-important functions in a building.

Picture of a phone with information about a door he sees: "Muffin to write home about bakery"

Image Credits: Apple

Door mode, which like the others can be automatically turned on or off at will, lets the user know whether the phone’s camera can see a door ahead, how far away it is, whether it is open or closed, and whether relevant information has been posted, such as a room number or address, whether the store is closed, or whether the entrance is accessible.

Sound recognition is a handy option for the hearing impaired who want to be warned when, for example, the doorbell rings or the oven beeps. While the feature previously had a library of sound types it worked with, users can now train the model locally to pick up sounds native to their household. Given the variety of alarms, buzzers and other sounds we all encounter on a regular basis, this should be very helpful.

Finally, there’s a thoughtful gaming feature called Buddy Controller. This allows two controllers to work as one, so one person can play a game using a second, should it be difficult or stressful doing it all by yourself. Given the complexity of some games, even on mobile, this can be very helpful. Sometimes I wish I had a gaming partner with a dedicated controller so I don’t have to deal with a game’s shaky camera.

There are a number of other minor updates, such as adjusting the time Siri waits before answering a question (great for people who speak slowly) and additional text customization option in Apple Books. And VoiceOver is coming to 20 new languages ​​and locations soon, too. We’ll know more about exact timing and availability as Apple makes more specific announcements over time.

Shreya Christina
Shreya has been with for 3 years, writing copy for client websites, blog posts, EDMs and other mediums to engage readers and encourage action. By collaborating with clients, our SEO manager and the wider team, Shreya seeks to understand an audience before creating memorable, persuasive copy.

More from author


Please enter your comment!
Please enter your name here

Related posts


Latest posts

The Nostalgia Factor: Why Retro Gaming Continues to Thrive

In the fast-paced world of modern video games with their stunning graphics, lifelike simulations, and complex narratives, there's a distinct charm in returning to...

The Rise of Cross-Platform App Development in Australia

In the ever-evolving landscape of app development, the need for efficiency, cost-effectiveness, and wider reach has given rise to a significant trend: cross-platform app...

Embracing Efficiency and Nature: Why Outdoor Pods Trump Typical Cubicles

In the ever-evolving world of work, the concept of the traditional office is undergoing a profound transformation. The limitations of the conventional cubic office...

Want to stay up to date with the latest news?

We would love to hear from you! Please fill in your details and we will stay in touch. It's that simple!