New Glove system translates ASL

[Terralthra]

Since there are some strong opinions hereabouts regarding ASL, Deaf culture, etc., I figured this might be interesting to folks:

Sign language-to-speech translating gloves take out Microsoft Imagine Cup 2012
By Darren Quick
19:43 July 10, 2012

Since beginning in 2003, the Microsoft Imagine Cup has tasked students the world over with developing technology aimed at solving real-world problems. In this, its 10th year, students were asked to build their project around a specific Millennium Development Goal (MDG), with the finals held this month in Sydney, Australia. The winners have just been announced and beating out teams from 75 countries to claim first place (and US$25,000) in the Software Design category was the Ukraine’s quadSquad with their EnableTalk gloves that translate sign language into speech in real time.

Aiming to extend the communication capabilities of those with hearing and speech disabilities after interacting with hearing and speech impaired athletes at their school, the quadSquad team set out to develop a way for those who know sign language to more easily communicate with those who don’t. Their solution includes a hardware component – the gloves fitted with various sensors – and a software component – which translates the hand signals into speech in real time.

Although the software was developed under Windows Phone 7, the team was forced to turn to the older Windows Mobile platform for their entry because Windows Phone 7 doesn’t provide developers access to the Bluetooth stack, which is how the gloves communicate wirelessly with a mobile device running the translation software.

The gloves themselves are fitted with an accelerometer, gyroscope, compass, and 15 flex sensors along the fingers, thumb and palm that determine the position of the glove in space. Data from the sensors is relayed to a controller on the back of the glove that is powered by a rechargeable battery. The battery can be recharged via USB, while a small solar panel also helps extend the intervals between charging. Data from the glove is then transmitted via Bluetooth to a mobile device that translates the signs into speech using the Microsoft Speech API and Bing API.

The team anticipates selling each glove for around US$200 if they can get them to mass production. If they do manage to get them to market, they would sell the system with a library of standard gestures. However, with sign language varying greatly around the world, the team says users can teach the system new gestures and modify existing ones.

Congratulations to the quadSquad team as well as the more than 350 students from 75 countries who made it to the finals of the competition.

The team's EnableTalk promo video can be viewed below.

As for my opinion: neither myself (ASL fluent) nor my fiancée (native language ASL, child of Deaf parents) believes that these will replace a translator any time soon, since the speed it demonstrates even for fingerspelling is ridiculously slow by ASL conversation standards. For reference, the average ASL conversation can involve intelligible fingerspelling at 15-20+ characters per second.

However, Deaf people can't exactly go around with a translator all the time any way, and this device could fill a very useful gap for Deaf people who must interact with hearing folks without a translator. Better than reading lips and writing notes back and forth, anyway.

[madd0ct0r]

yeah - but's only one way. Unless the ASL user can lipread, the other guy will have to write notes anyway.

Friggen cool UI development though

[Losonti Tokash]

I would also imagine that continued work on the glove would increase its translation speed.

[Terralthra]

yeah - but's only one way. Unless the ASL user can lipread, the other guy will have to write notes anyway.

Friggen cool UI development though

I'd say the majority of ASL speakers can lipread to some extent. It's a fairly necessary skill in a hearing world.

[Alyrium Denryle]

yeah - but's only one way. Unless the ASL user can lipread, the other guy will have to write notes anyway.

Friggen cool UI development though

Still, it would come in handy. A friend of mine is a child of deaf parents. His dad has a cochlear implant and can hear somewhat, but his speech is not particularly intelligible, and with me not being able to sign, it is very difficult to communicate with his mom. Having a one way conversation with a notepad or whiteboard is much easier.

For some everyday things as well, such a thing would be very useful. It is very hard for a deaf person to even get decent help in a grocery store, because if they need help finding something, or want to order something from the deli, they have a hard time communicating with people who cant sign. Even when they can speak, it is sometimes very off, because depending on the degree of deafness, they cannot modulate their voice as well, it is like having a very extreme accent, they just dont have the ability to hear themselves, and thus cannot do some of the fine adjustments necessary for some sounds as well. Of course, that depends on how deaf they are etc etc. When I worked in a store as a teenager, I did decently because I carried around a notepad (I knew we had deaf customers) and was willing to try and be patient.

Being able to talk through signing and a synthetic voice would make a lot of lives much easier, as all a store clerk need do in order to indicate answers for the more common questions (like where to find product X or respond to a request for deli meats) is speak slowly (to facilitate lip reading), nod or shake their heads, and use unambiguous indicative gestures.

[Sarevok]

Amazing. Reminds me of that glove seen in the movie Congo that let Gorrilas speak to humans. It's astonoshing how science fiction sometimes so closely translates into reality.

[Isolder74]

Amazing. Reminds me of that glove seen in the movie Congo that let Gorrilas speak to humans. It's astonoshing how science fiction sometimes so closely translates into reality.

Well the ape in question, obviously a reference to Koko, already was speaking using ASL to begin with. The thing in the movie was this device only using a cruder system for the interpreting.

This gadget is amazing. It's only snag is it doesn't help a casual person speak back to the deaf user but it is a step in the right direction. Perhaps a two way mirror eyepiece to type out voice recognition as a person talks to the glove user?

[Irbis]

I would also imagine that continued work on the glove would increase its translation speed.

The problem here is not translation speed, it's the fact normal human can only move fingers so fast. At best, you can insert a few 'shortcuts' allowing user to not have to always finish words and improvements smoothing the voice reader, but more than that would require some fundamental technology breakthrough.

[Terralthra]

I would also imagine that continued work on the glove would increase its translation speed.

The problem here is not translation speed, it's the fact normal human can only move fingers so fast. At best, you can insert a few 'shortcuts' allowing user to not have to always finish words and improvements smoothing the voice reader, but more than that would require some fundamental technology breakthrough.

I think ASL speakers move their hands faster than you think they do. A better issue would be contextual inflection of words. Shorter, smoother, longer, faster, jerky, slower...ASL users inflect verbs directionally to indicate subject and object...any true translation apparatus would have to be able to understand all these inflections transparently. It gets back to the larger natural language processing problem, really.

[Purple]

Would it not be cheaper and easier to make a text to voice and voice to text app for a mobile phone or similar device? And it would be more useful too as I doubt that there is anyone any more under the age of 60 or something who does not have quick texting skills. And it would enable a two way communication.