Huawei AI Is Helping the Blind ‘See’ Emotions Through the Power of Sound

A simple smile is something we all take for granted, but imagine a world without smiles? This is the world of the visually impaired. Unable to see faces and read emotions, they are often literally left in the dark. Now, HUAWEI’s Facing Emotions app can help change how blind people 'see'.

Saatchi & Saatchi Poland and HUAWEI have created an app that allows blind people to ‘see’ or experience emotions on the faces of people they are speaking to using the power of AI. The Facing Emotions app utilises the powerful camera and AI of the HUAWEI Mate 20 Pro, to translate seven universal human emotions, into seven unique sounds. Making it possible for the blind and visually impaired to have a richer experience and understanding when communicating with others. 

An AI powered app

Facing Emotions was created to push the possibilities of AI technology. By asking “how can AI help the blind and visually impaired to experience or ‘see’ more of the world?” Saatchi & Saatchi were able to find a unique new application for the power of AI. By teaching it to analyse facial expressions and emotions, and translate them into sounds, the agency created a new language for the blind to ‘see’ emotions like never before.

Using the back cameras, the application scans the face of the person with whom the blind person is talking, and identify elements of the face: like eyes, nose, brows and mouth, along with their position in relation to each other. AI then processes the identified emotion into a defined sound heard on the phone (or in the earphone the blind person is wearing). All of this happens in a real time and in offline mode. It is only possible thanks to a powerful processor and AI, which Mate 20 Pro is equipped with.

The project was an opportunity to use technology to try and help find a solution to a real world challenge experienced by the blind.

Bringing people and technology together

Facing Emotions was developed with the support of the blind community and charitable organisations. Huawei partnered with the Polish Blind Association and created a key group of blind testers which were involved in every key step of the app development process. 

The brand interviewed blind community representatives about their needs and based on those insights they developed solutions which were consulted with and tested by blind or visually impaired people. It was their input that helped shape the final user experience, as well as its functionalities, colours, and sounds.

“I was amazed by how quickly things could be implemented in the app. It is amazing, and the holder is a great success," said Monika Woznialis, blind test team Member.

Feeling inspired 

A key component of this process was the sounds themselves. They were created by blind composer Tomasz Bilecki - known among other things for the Invisible Exhibition - to capture each of the seven emotions. The sounds had to be short, concise, simple and non-invasive for the user and their environment. His unique perspective on the nature of sound allowed him to convey the emotions to other visually impaired people. Sound samples were then tested by the team of blind consultants, who rated them as easy to remember and understand. 

Said Tomasz Bilecki: “The sounds I wanted to create needed to be specific for each emotion, and not distract users from their conversations."

During the app development process, the team learned that visually impaired people would ideally love a hands free device solution so they have easy access to it with their hands often used for navigating or holding a cane. So as part of the campaign, award-winning designer Janek Kochanski developed a special 3D printed phone holder for the Mate 20 Pro that makes it easier for visually impaired people to use the app and their phone in general.

“For me, as a designer, function is foremost, and the form often results from that. First, the usage requirements have to be satisfied, only then can the aesthetic and visual values be met. The phone that the holder is designed for, the Huawei Mate 20 Pro, is very modern and stylish, but so is the phone case," said Janek Kochanski, phone holder designer.

These 3D holders are available for anyone to print on a click and deliver to your door website shapeways.com, as well as a certain amount available free from partnering Blind NGO’s in key countries from CEE&Nordics Region.

The Facing Emotions Short Documentary

To show how much difference the Facing Emotions app can make in people’s lives, a short documentary film was created. The film featured a key moment in the lives of three blind and visually impaired people, when understanding the emotions displayed on their loved one’s face were incredibly important for them. 

The people featured - Mirjana from Sweden, Robert from Poland, and Mustafa from Turkey, were found through a partnership with local blind NGO’s in several countries searching for real people who would like to use the app and be part of the project. Their stories and their reactions were incredibly moving, making this powerful short film something that will stay with viewers for a long time. 

“The Facing Emotions campaign embodies HUAWEI's mission to to bring new technology to everyone, especially those that need it most. Our new flagship HUAWEI Mate 20 Pro advanced artificial intelligence has given us a unique opportuniy to let us help blind people "see" emotions," said Mr. Wang, President, CEE&Nordic Region, Huawei Consumer Business Group.