For the deaf, new software brings sign language to cell phone video

In an effort to improve communication for the deaf, engineers at the University of Washington are developing software that transmits American Sign Language over cellular networks.

I spoke last week with project leader Eve Riskin about how the software works -- and about its implications for the deaf community.

How did you go about creating this software?

We started working on the project about five years ago. Our plan was to work on software to compress American Sign Language video. Our goal was to do it on PCs or work stations. We figured that would be challenging enough. But in the course of working on a project new ideas pop up. Four years ago, I got a new cell phone that happened to have a video camera on it. We had an 'ah-ha' moment that we could do this on a cell phone. We picked the HTC TyTN II. It was the first phone we found that was Windows Mobile and had the camera on the front.

How similar is your software to the iPhone4 Facetime feature?

That's exactly the idea. It's got snazzier software because they're Apple and we're just a research group, but the idea is there. We've had our program running for two and a half years now, so we've been able to go around and give talks about our project. [We'd] talk to kids in high school to get them interested in engineering and then show them [our] phones. They were flabbergasted because no one had ever seen it before.

Why is it important to be able to transmit sign language via video, instead of just using text?

[For] many people who speak sign language, [it] is their first language. You're texting in a foreign language. Some people who are deaf have rather limited English. Also, a lot of the emotion, expression, nuances are conveyed through the face.

Talk about the field testing you've done of the software.

It was a limited field test. It was nine students and two [teaching assistants] who are here in Seattle this summer. We'll have a larger field study in the winter where we'll investigate more of the technologies we're working on. The field study we had here was more about giving people the phones and seeing how they use them. In our next field study, we'll be able to test out our battery extension algorithms. We'll learn what people think when they're actually using the technology.

Once the field testing is complete, how will this technology be available to the public? Would it be through one phone only or through a phone application?

We are interested in porting it to other devices. We're very interested in trying it on the Android phone. Some of what we've done could mesh with Facetime if we had the person-power to do the extra software [development]. It could be an app for different phones. We just have to find the funding to do that.

The software looks like Facetime, but does it also include specific features geared toward the deaf audience?

• Our user interface was designed with input from a young woman who is a native [sign language] speaker. It's based on video phones. The user interface is very similar to what people who are deaf are used to using.
• We do some signal processing to make the quality better in the hands and face. Our goal wasn't to have beautiful video like Facetime. Our goal was to have intelligible sign language, so you could have a conversation.
• People who are deaf take turns just like people who are hearing. If you are signing and I'm not signing, then it's going to lower my quality. It saves battery lifetime because the cell phone is doing less work.
• We don't send audio, so that saves bandwidth and time. You don't need voice for our target audience.
• We have what we call a contact list. It's just like [the green dots on Skype and Google Chat that indicate when a person is available to talk]. It makes it easier than just calling somebody and not knowing they're there.

Click on the links below to watch videos of the software in action:

• UW working on app to help deaf, hard of hearing
• Deaf, hard-of-hearing UW students test sign language cell phone
• Hands-On: MobileASL- Video conferencing for the deaf community

Image: Eve Riskin

This post was originally published on Smartplanet.com