There’s a great deal of activity in the fields of speech recognition and the “Internet of Things,” but one natural application of the two has gone relatively unpursued: helping the deaf and hard of hearing take part in everyday conversations. SpeakSee aims to do this (after crowdfunding, naturally) with a clever hardware design that minimizes setup friction and lets everyone communicate naturally.
It’s meant to be used in situations where someone hard of hearing needs to talk with a handful of others — a meeting, a chat at dinner, asking directions and so on. There are speech-to-text apps out there that can transcribe what someone is saying, but they’re not really suited to the purpose.
“Many deaf people experienced a huge barrier in asking people to download the app and hold the phone close to their mouth. These limitations in the interface meant no one kept using it,” explained SpeakSee CEO and co-founder Jari Hazelebach. “But because we designed our own hardware, we were able to customize it towards the situations it will be used in.”
SpeakSee is simple to use: A set of clip-on microphones live in a little charger case, and when the user wants to have a conversation, they hand those microphones out to whoever will be talking. The case acts as a wireless hub for the mics and relays the audio to the smartphone with which it’s paired. This audio is sent off, transcribed quickly somewhere in the cloud, and displayed on the deaf user’s phone.
Critically, though, each microphone also intelligently and locally accounts for its speaker and background noise.
“Naturally the microphones pick up speech from multiple people,” said Hazelebach. “So we included sensors that tell the microphone what direction the sound is coming from, and the microphones exchange these values. So we can determine which microphone should pick up which person’s speech.”
The result is quickly transcribed speech divided by speaker, delivered quickly and with decent accuracy (there’s always a trade-off between turnaround time and how the process is). And no one has to do anything but wear a mic. (They have a patent pending for this multi-microphone system.)
Hazelebach’s parents are deaf, and he grew up seeing how their ability to interact in ordinary circumstances was being limited.
The mics aren’t exactly small… but that’s how you know they’re real working hardware and not imaginary.
“As you can imagine my parents were the first to test this out,” he said. “At first we had a lot of issues but soon we started engaging with others. We wrote a post on a deaf blog and out of nowhere 200 people signed up. So we’ve been testing in the field with groups in the U.S., and also in the U.K. and the Netherlands.”
Right now English speech recognition is considerably ahead of Dutch and other languages, so the transcriptions will be better for the former, but even so the devices should work with any of 120 languages supported by the cloud service. Transcription is free for up to 5 hours of audio monthly, after which it’s a $10/month subscription. But if it works, it may be more than worth the money.
The team has a finished prototype but is seeking crowdfunding to get production off the ground. “We need to improve the electronics to meet specifications, battery life for example. We expect to ship in February of 2019,” Hazelebach said. Pre-orders are set at $350 for a dock and three mics.
The usual caveats (primarily “emptor”) apply when backing an Indiegogo type campaign — but at the very least, having spoken to the creator, I feel pretty sure this is a real, working product that just needs a boost to get to market.