Open Captions

[Updated with the latest status of Open Captions project]

I started building Open Captions for the Hackday TV hackathon at General Assembly.

Open Captions gives the viewers, the ability to search and view closed captioned YouTube videos. While watching the video, the viewer can select individual words of the closed captions and then see an American Sign Language Representation of the word on the screen. The viewer can also jump back to the previous caption showed on the screen. Whenever, a word is selected, the video automatically pauses when the ASL is showed on the page. The working prototype can be seen on Open Captions

Currently the project is a work in progress, and is susceptible to fail at a few places. Please feel free to email feedback, ideas, suggestions about Open Captions.

The ASL representation is just one of the use cases of Open Captions. This can be linked with Google Translate to find out words in other languages (very useful for people for whom English is not the first language). This can also be linked with say Wikipedia and children can learn about a word or a place spoken about on TV.

This idea of selecting words from closed captions also works for videos from Universal Subtitles, an initiative which helps create closed captions for videos on the Internet. Open Captions example with Universal Subtitles

Reference
The ASL representations are being scraped from www.cats.gatech.edu, which has around 25000 words shown in ASL in 25000 web pages.

ASL Dictionary Greasemonkey script v0.1

Research says that around 90% of hearing impaired children are born to hearing parents. So if parents do not know Sign Language then it is difficult for the child to learn ASL quickly. Wouldn’t it be great if people could learn sign language on the fly when they are reading articles on the web? Maybe they are reading stories to their children from a website and they can have ready assistance to translate words in written language to Sign Language. So, I thought of creating a simple Greasemonkey script that would show the ASL equivalent of a selected word.
Center for Accessible Technology in Sign at GeorgiaTech has a huge database of words and their American Sign Language equivalent. I used their web pages that they had created for around 25000 words. Whenever a word is selected, I load the ASL equivalent in an iframe on the right top corner of the page.

To Install

Here are the steps to try it on Firefox

1. Open Firefox (version 3.0 – 4.0.*)
2. Install Greasemonkey if you are using Firefox
3. Click on this link to download the script
4. Let Greasemonkey install the user script.
5. Now open any webpage say – Wikipedia entry on ASL
6. Now select any word (not phrases/sentences) – for example – Sign, Language, also, common, English or languages

You will see the American Sign Language representation of the word on the top right.
If a word that does not exist in the database is selected then, it shows “Sorry word could not be found”.

Here are the steps to try it on Chrome

1. Open Chrome
2. Click on this link to download the script
3. Let Chrome install the user script.
4. Now open any webpage say – Wikipedia entry on ASL
5. Now select any word (not phrases/sentences) – for example – Sign, Language, also, common, English or languages

You will see the American Sign Language representation of the word on the top right.
If a word that does not exist in the database is selected then, it shows an error message.

An example, with the word Languages highlighted

Another example, with the word Sign highlighted

The representation automatically hides after 5 seconds. (So if you click on another word before 5 seconds, the results may be buggy)
Update

To Uninstall
To uninstall on Firefox, click on the Greasemonkey icon and disable the ASL Dictionary.
To uninstall, open chrome://extensions/ and disable the ASL Dictionary.

For future versions
1) If you select a phrase, it shows the sequence of signs
2) Fix “Word not found” error in Chrome
3) Ability to select the word, right click and have the option to play the video or the cartoon