AR glasses allowing deaf people to ‘see’ conversations adds new features

These aren’t your typical sunglasses.

Josh Feldman, hard of hearing since birth, is using them to read what people say to him.

The glasses feature an Augmented Reality (AR) display which can overlay any type of content onto what the user sees.

Now Feldman can read live transcriptions.

And for the first time, he’s able to talk with someone without looking at their lips.

“To have a conversation by not looking at the person you’re speaking to. Wow. Wow. And I think the people who are hard of hearing would understand the feeling I had when I had that conversation, which lasted about five minutes. And I didn’t look at the person to my right once and solely look ahead — It’s a life-changing moment,” says Feldman.

The company behind this solution is XRAI Glass. It was incorporated in May.

XRAI Glass used pre-existing technology — AR glasses and audio transcription — to offer a new product to deaf and hard-of-hearing people.

And it’s not limited to in-person conversations.

Thanks to the glasses, Feldman can read conversations he has on the phone too.

Smart assistants like Alexa or Siri can also be understood by the transcription service.

And on Thursday, the company announced the launch of its own personal assistant, “far more powerful & effective than any current voice assistant,” claims XRAI Glass.

The tech startup says it has the potential to change the life of millions of people with hearing loss.

New features also include the support of nine languages and live translation, including Mandarin, and the ability to summarise any conversion or replay it in full.

But the technology has some hitches.

The transcription, based on Amazon’s Alexa transcription service, struggles to understand group conversations when people speak over each other.

It also needs a fairly quiet environment for it to be as accurate as possible.

And users will also have to overcome the aesthetics of AR glasses.

“The idea that someone is wearing sunglasses in a meeting, is a bit of a weird idea right? I think, fundamentally, this is just the beginning,” says Feldman in July, as the company was just about to launch its product with 100 beta testers.

A free version of the app is now available, but users have to pay up to £49.99 ($59) a month for premium features like unlimited conversation history and speaker identification.

“We’re going with a minimum useful product that is our benchmark at the moment, and we don’t want to let perfection be the enemy of done. We believe that it’s at a point now where it’s actually useful to people. So that’s why we want to start to get it into the hands of people. It will get better and better over the coming weeks, in the coming months,” says Dan Scarfe, Founder and CEO, XRAI Glass.

The Nreal AR glasses can be purchased separately for £340.99 ($404) but are only available in a limited number of countries.

And even if you managed to get your hands on one of them, just 14 Android smartphone models sold in the U.K. — no iPhones— are compatible with them, according to the manufacturer.

That’s because they require a high-end processor and a USB-C port supporting DisplayPort, the technology that allows the glasses to be connected to smartphones as an external display.

“I think the real breakthrough will be when we get smart contact lenses. That will be the thing that actually revolutionises the space and that the very first prototypes are just coming out and they will go mainstream within the next two to three years. But then you will always be in an augmented reality world,” says Scarfe.

Some members of the Royal National Institute for Deaf People (RNID) have tested the technology.

The organization doesn’t endorse individual companies or products, but encourages technological advances such as this one.

“So if somebody is wearing a mask, it makes it really difficult for a deaf and hard-of-hearing person because they rely on lip-reading to a certain extent to hear what’s been said. So to have something like those glasses where it takes away the barrier of the mask that they can actually see what’s being said, and that’s a real effect and that would really make a difference to someone,” says Teri Devine, Associate Director of Inclusion, RNID.

People with hearing loss are already using apps on their phones offering live transcription in their daily lives.

XRAI Glass enables them to read them more freely, while their phone remains in their pocket.

The company is already working on the second version of its technology, which will pick up visual clues to better identify speakers and improve transcription thanks to automated lip-reading.

AP video shot by: Tristan Werkmeister