Music: Not Impossible for the hard of hearing

Not Impossible Labs, the award-winning technology incubator and content studio has created Music: Not Impossible, a Vibro-textile wearable that creates an immersive experience of music for both deaf and hearing people.

Image shows people at a concert with amr sin the arm. A performer is just visible on stage.Humans love music. We hum, whistle, tap, and clap, and can turn an upturned pot into a drum or build huge, complex instruments of valves or circuits.

Music is tribal, communal, and visceral. 

For those who are deaf or have hearing loss, music largely becomes a one-dimensional experience of powerful low-frequency sounds hitting them, literally vibrating their bodies; something Not Impossible's CEO Mike Ebeling deemed 'absurd.'

His Not Impossible Lab specialises in using tech to solve absurd challenges and so turned its attention to this one.

AbilityNet's Hearing Loss and Computing Factsheet provides a useful introduction to some of the key assistive technology available.

Skin as an alternative to the eardrum?

We hear sounds when the vibrations of a speaker (or instrument, or voice) travel through the air to our eardrum. The vibrations transmit through tiny bones into the inner ear where the cochlear converts these vibrations into signals the brain can interpret. 

It's complex, and things can go wrong. 

To create a richer experience for people with hearing impairments, Ebeling decided to bypass the eardrum and look to an alternative receptor for these vibrations - human skin.

Read about award-winning innovations for disabled people from AbilityNet's Tech4Good Awards

The first prototype device was rough and ready. It was fabricated from wires, duct-tape and smartphone motors that cause a phone set to silent to buzz. Early users described the sensation as surreal and uncomfortable while chief designer and musician Daniel Belquer described it as "horrible" and said it "did not make sense." 

The team abandoned the idea of a vest turning its attentions to different vibrations on different parts of the body. The systems would send drums to the ankles, guitars to the wrists, vocals at the base of your ribcage and the base in the small of your back.

It was a breakthrough. The vibrations began to make sense. The team could recognise songs without hearing them.

 

However, the entire system was being designed and trialled by people who could hear. If Not Impossible was to realise its goal of "increasing audio inclusion" he would need deaf people involved in the project.

Are you designed websites for disabled people? Our accessibility experts can help you to get it right; first time, every time.

Testing with the hearing-impaired

Mick got in touch with Mandy Harvey, a musician who lost her hearing as a result of Ehlers-Danlos syndrome; a genetic disorder that affects the connective tissues in the body.  

Mandy's hearing loss occurred rapidly one month into starting a music major at University; a subject that was her passion.

Speaking on the BBC Ouch programme, Mandy told the story of being in a music dictation class (a piano is played and you have to chart the notes and timing onto paper) she explained that she was sitting patiently waiting for the test to start and noticed everyone around her scribbling away.

Her hearing aids weren't able to give her the information she needed and she was dropped from the music program.

"I really didn't know what to do, because my way of expressing myself ... was gone," she said.

Mandy's father did not want to see his daughter give up and encouraged her to pick up the guitar and play alongside him as they had often done; he encouraged her to learn a song; just one song that she figured out note by note, singing it into a tuner to gauge her pitch, watching for the green light and starting over if she saw a red one, feeling the vibrations of her own voice, the rise and fall of her larynx. Once she had the song, she pitched to the first note, closed her eyes and sang. 

Mandy is now an accomplished Jazz singer and songwriter.

She uses a tuner or app on her phone to pitch her starting note to, just as she had done with her first song, but then it's muscle-memory developed in the years of learning music prior to losing her hearing that kicks in and she sings.

Watch our webinar about how people with hearing loss can adapt their smartphones

She and her band have developed their own visual shorthand to give her cues, but she is also known for singing barefoot, explaining that the vibrations she feels through the floor allow her to keep time. She is not a 'deaf singer' she is a singer.

The system Mandy developed that allowed her to feel music was exactly what the Not Impossible team needed and almost immediately Mandy could see the potential of what Mick and his team were attempting. Mandy explained that although the team had divided the sound into different channels to be sent to different locations on the body, and this gave more tactile information, there was no difference in the intensity of the vibrations being sent.

Tech4Good Accessibility Award 2019 winner DigitMusic developed Control One in response to the need for a non-instrument alternative for severely disabled people to create music.

From prototype to production

Music Not Impossible harness (picture from audioexpress.com)Whilst having a design ethos "help one, help many", starting projects with the notion of solving one problem for one person, scaling a system as complex as the M: NI (Music: Not Impossible) system was not as simple as taking the prototype, making it look good and building more of them.

The wrist and ankle bands were wireless allowing freedom of movement, but each would require batteries to power LED lights, actuators and motors to produce the vibrations and all the wireless parts needed to communicate with the harness.

The harness itself needed to be lightweight and comfortable to move and dance in, it too incorporated lights and actuators and vibration modules as well as a subwoofer speaker.

All the separate parts needed to be synchronised and communicating with each other reliably, but without interfering with the main broadcast signal that had to be synchronised with the music being played.

All communication also had to take place within a 30-millisecond window or the sensation would be, as Avnet; the company charged with upscaling the product highlighted; "like watching television when the audio is out of sync".

First test event

The first test of the system was at the 'Life is Beautiful' festival in Las Vegas. You can watch the video, below.

It enabled an audience of hearing and non-hearing guests shared a musical experience; together; hearing was an additional element, but not a crucial one; the experience was not only inclusive but enhanced for everyone.

In an interview for audioXpress, Mick Ebeling explained; "When we beta tested this with the deaf community, they said, 'This is blowing our minds. We don't want to give this back!' Music: Not Impossible is a new gateway for how we'll sense music in the future. This is just as much a breakthrough in technology and music as it is for promoting human connection through shared experience.

The next event was different; this was more individual, more personal. Mandy was to perform her music accompanied by the Monroe Symphony Orchestra, something she would be able to do with her existing strategies, barefoot and with help from her band, but this time she was on her own with the M:NI giving her the feedback she needed; the sensation of the orchestra, the different sections; the violins, the oboes and clarinets, no longer needing visual cues from anyone.

"I don't sing for myself, I can't hear myself sing. It's not that interesting. The interesting part is how it affects other people and how it creates a community." - Mandy Harvey

Related content:

AbilityNet's Tech4Good Awards recognise organisations and individuals who use digital technology to improve the lives of others and make the world a better place.  

 

How AbilityNet can help?

AbilityNet provides a range of free services to help disabled people and older people.

  • Call our free Helpline. Our friendly, knowledgeable staff will discuss any kind of computer problem and do their best to come up with a solution. We’re open Monday to Friday from 9 am to 5 pm on 0800 269 545.
  • Arrange a home visit. We have a network of AbilityNet ITCanHelp volunteers who can help if you have technical issues with your computer systems. They can come to your home, or help you over the phone.
  • We have a range of factsheets that talk in detail about technology that might help you, which can be downloaded for free. You may find our factsheets about voice recognition and keyboard alternatives useful.
  • My Computer My Way. A free interactive guide to all the accessibility features built into current desktops, laptops, tablets, and smartphones.