Inclusivity of VR and AR Accessibility for the Visually and Hearing Impaired

VR and AR provide the amazing ability to explore new worlds, merge the digital world with the real world, and go on all sorts of adventures without even having to leave home. They also offer amazing opportunities for a multitude of different people with disabilities, and even quality of life benefits. People love the sense of exploration, entertainment, aid, and freedom that VR and AR offer, but it’s also a type of technology that relies on visual and audio cues. This brings new barriers to those who may have vision or hearing impairments, but also opens up innovative ways to leverage VR and AR to not only explore solutions in entertainment for all, but creative solutions to assist the visual or hearing impaired in new and improved ways.

Knowing these challenges, some companies have set out to explore alternative ways to offer VR and AR experiences in a way that’s accessible to those who are vision and hearing impaired. However, there have already been some extraordinary breakthroughs in an exciting new world that is still in its infancy.

VR and AR Accessibility for the Visually Impaired

Google Daydream is one major company testing the use of spatial audio cues for those with visual impairments. Using a prototype of a virtual room, Daydream Labs recorded and linked the name of every object in the room, and to enable navigation in a pitch-black room, created a 3D audio laser pointing system from an HTC Vive controller. This would then select and play the audio labels while leveraging audio location control. This allows the system to play a short impulse response tone at the location of the controller that quickly progresses to the actual distance to the virtual object.

Similarly, Microsoft created the Canetroller, which essentially takes the concept of a blind  or visually impaired individual’s white cane, which is a tool used to detect surrounding obstacles in their environment, and applies it to the world of virtual reality.

The Canetroller relies on haptic and audio feedback to simulate the world around the user in a virtual world. This is achieved by the five essential parts of the Canetroller itself. This is a braking mechanism, a controller, a slider that connects the braking mechanism to the cane, and a voice coil that’s mounted on the cane to provide the sensation of vibration to the user. The HTC Vive is actually mounted to the controller itself.

There are also even applications being developed to bring inclusivity to VR and AR technology. Near Sighted VR Augmented Aid is an Android mobile application that uses a user’s mobile device camera and displays the image to the phone screen in stereoscopic view. It is most beneficial for people who have visual impairments such as Stargardt’s disease, macular degeneration, optic atrophy, and more.

Additionally, companies like Aira, leverage augmented reality through a subscription based service that provides the ability to live stream video from smart glasses, such as Google Glass. Their focus is to assist visually-impaired people navigate their environments through a trained professional agent, at the touch of a button. Erich Manser, who is legally blind, was helped through the 26.2 mile Boston Marathon course using Aira’s assistive technology.

Experiencing the beauty of art has always relied on the ability to actually visually see these masterpieces of the world. Not anymore. From the creative collaboration team of Geometry Prague, NeuroDigital Technologies, and the Leontinka Foundation for the blind and visually impaired, art is now accessible to the blind through virtual reality technology. This real-life exhibit has been called Touching Masterpieces.

Users just put on a pair of NeuroDigital’s haptic gloves and an HMD, and they are free to explore recreated avatars of the most famous sculptures in the world, such as Michelangelo’s David, or the bust of Nefertiti in complete virtual reality. The original artworks were first mapped with laser scans, and added multiple layers of details that encompass the real-life statues. This results in a realistic tactile experience that allows users to feel these statues.

Augmented Reality for the Hearing Impaired

Augmented reality already bridges the gap between the digital and the physical, so it seems only logical that it should bridge the gap between the digital and accessibility for the hearing impaired.

One design company from Germany, Morphoria, created Deaf Magazine, which was in publication for a limited time, and focused on the culture of German Sign Language, that combined with their own augmented reality application. This let readers interact with the magazine’s pages by learning new words in sign, or even teach those who do not understand sign.

Also in Germany, Greta & Starks are iOS and Android applications that allow those with vision or hearing impairments to still enjoy the movie experience via smart glasses. Greta speaks descriptions and actions in time with the movie being played, and Starks displays captions in a variety of different languages.

London is following a similar feat with London’s National Theatre using Epson’s latest smart glasses which display subtitles in user’s field of vision. The National Theatre already had set up caption screens, and it reserves a set of seats specifically with a view of them. However, this requires theater goers to look away from the stage in order to read the text. With Epson’s smart glasses, the subtitles are always within the wearer’s field of vision as they continue to watch performances with sacrificing missing any of the action.

A Future of Innovation to Make VR and AR Experiences More Accessible

There are plenty of hardware and software solutions to get excited about in which companies are ensuring they are creating VR and AR experiences that are accessible, and also have a positive effect.

It’s very easy to design these experiences for a world we are accustomed to living in, in the way we are used to experiencing it. It takes great effort to go beyond that and move VR and AR technology towards mainstream adoption that’s inclusive of everyone. Luckily, we have seen some great steps forward in these innovations to embrace not only VR and AR experiences, but also the human experience.