Apple has released a slew of new accessibility features for iPhone, Apple Watch, and Mac, including a universal live captioning tool, improved visual and audio detection modes, and iOS access to WatchOS apps. The new capabilities will arrive “later this year” as updates roll out to various platforms.
The most widely applicable tool is probably live captioning, already very popular with tools like Ava, which recently raised $10 million to expand its repertoire.
Apple’s tool will perform a similar function to Ava’s, allowing all spoken content a user encounters to be captioned in real time, from videos and podcasts to FaceTime and other calls. FaceTime, in particular, will have a special interface with a speaker-specific scrolling transcript above the video windows.
The captions can be activated through the usual accessibility settings and quickly toggled on and off or the pane into which they expand or contract. And it’s all done using the device’s built-in ML acceleration hardware, so it works when you have spotty or no connectivity and no privacy issue.
This feature could very well cut off the wings of independent providers of similar services, as often happens when companies implement first-party versions of traditional third-party tools, but it could also increase quality and competition. Having a choice between different carriers isn’t a bad thing, and users can easily switch between them if, as seems likely to be the case, Apple’s solution is best for FaceTime, while another, like Ava, could make it excel in other situations. . For example, Ava lets you save transcripts of conversations for later viewing – not an option for the Apple subtitles, but certainly useful in work situations.
Image Credits: Apple
Apple Watch apps are getting improved accessibility on two fronts. First, there are some added hand gestures for people, e.g. amputees, who struggle with the finer interactions on the small screen. A stack of new actions available through gestures like a ‘double pinch’, which let you pause a workout, take a photo, answer a phone call, and so on.
Second, WatchOS apps can now be mirrored to iPhone screens, where other accessibility tools can be used. This is also useful for anyone who likes the Apple Watch’s smartwatch-specific usage scenarios, but struggles to interact with the device on their own terms.
Existing Magnifier and Sound Recognition tools also get some new features. Magnifier’s “detection mode” normally lets the user know if a person or anything readable or writable is directly in front of them: “person, 1.5 meters ahead”. Now it has a special “door detection” mode that details those all-important functions in a building.
Image Credits: Apple
Door mode, which like the others can be automatically turned on or off at will, lets the user know whether the phone’s camera can see a door ahead, how far away it is, whether it is open or closed, and whether relevant information has been posted, such as a room number or address, whether the store is closed, or whether the entrance is accessible.
Sound recognition is a handy option for the hearing impaired who want to be warned when, for example, the doorbell rings or the oven beeps. While the feature previously had a library of sound types it worked with, users can now train the model locally to pick up sounds native to their household. Given the variety of alarms, buzzers and other sounds we all encounter on a regular basis, this should be very helpful.
Finally, there’s a thoughtful gaming feature called Buddy Controller. This allows two controllers to work as one, so one person can play a game using a second, should it be difficult or stressful doing it all by yourself. Given the complexity of some games, even on mobile, this can be very helpful. Sometimes I wish I had a gaming partner with a dedicated controller so I don’t have to deal with a game’s shaky camera.
There are a number of other minor updates, such as adjusting the time Siri waits before answering a question (great for people who speak slowly) and additional text customization option in Apple Books. And VoiceOver is coming to 20 new languages and locations soon, too. We’ll know more about exact timing and availability as Apple makes more specific announcements over time.
