Open Captions

[Updated with the latest status of Open Captions project]

I started building Open Captions for the Hackday TV hackathon at General Assembly.

Open Captions gives the viewers, the ability to search and view closed captioned YouTube videos. While watching the video, the viewer can select individual words of the closed captions and then see an American Sign Language Representation of the word on the screen. The viewer can also jump back to the previous caption showed on the screen. Whenever, a word is selected, the video automatically pauses when the ASL is showed on the page. The working prototype can be seen on Open Captions

Currently the project is a work in progress, and is susceptible to fail at a few places. Please feel free to email feedback, ideas, suggestions about Open Captions.

The ASL representation is just one of the use cases of Open Captions. This can be linked with Google Translate to find out words in other languages (very useful for people for whom English is not the first language). This can also be linked with say Wikipedia and children can learn about a word or a place spoken about on TV.

This idea of selecting words from closed captions also works for videos from Universal Subtitles, an initiative which helps create closed captions for videos on the Internet. Open Captions example with Universal Subtitles

Reference
The ASL representations are being scraped from www.cats.gatech.edu, which has around 25000 words shown in ASL in 25000 web pages.

My new friend Marvin

Given my anti-social tendencies (read not logging on to Facebook for 3 months at a stretch), I decided that I needed a new friend.

My Friend Marvin

Marvin (the paranoid android) is built out of waste thermocol (that came with the packing of my monitor). It has a small red breadboard attached as it’s heart. The light on the heart glows green if there are more tweets with positive sentiment going around in the (All izz well) and glows red if there are more tweets with negative sentiments going around in the virtual world. (I took about a handful of words like “environment”, “plastic”, “global warming” etc. as the baseline hashtags to be analyzed for sentiments) More on the implementation of the Arduino controlling the LEDs can be found in the Project Cycles post here

My last Labor Day weekend project was drawing this, I am happy that the trend of doing something I love during the long weekend has continued.

Project Cycles

The weekend of August 5th -7th 2011 was one of the most memorable weekends in recent times. I was participating at the Art Hack weekend organized by the Creators Project at Eyebeam. It was a  two-day, open-source hackathon (hacking+marathon) that celebrates new artistic experiences.

The goal: We were to design, code and prototype projects that re-imagine the way people create, consume, and interact with media.

tl;dr – Jump to straight to video
The participants (around 75 in number) met up on Friday at Eyebeam to socialize and form teams. Each of us introduced ourselves and talked about our skill sets (what we brought to the table) and also mentioned any ideas they had in mind for the weekend. It was very exciting for me as it was the first time I was participating in a hackathon that involved people other than just developers. It was a wonderful way to understand and see things from a different perspective. After the round of introductions we were “left loose” to mingle around and talk to people with whom we wanted to collaborate for the weekend. I had no particular idea in mind, but my criteria was 4-fold.

• Work on something interesting for the weekend
• Get out of my comfort zone
• Work on something that would increase social awareness and make people think
• Have :awesome: fun

One of the participants, Mark Stafford, a fine artist by profession, said that he wanted to work on an art installation made up of waste material. That triggered a bell for me as I am interested in waste management, recycling and all that good stuff. Soon, Mark, Meredith and me were sitting and threshing out our idea for the weekend. It took us to about 5-10 minutes to decide that the idea was something all of us wanted to invest our time and energies in.

So here is the idea and what we wanted to build -

We as a race are progressing at pretty good speed on the technology front. We keep building more and more devices, connect more and more in the virtual world, get more and more impacted by the technology around us. All this progress does leave something behind. Project Cycles, is an attempt to describe the life cycle of our digital, virtual and physical environments.The installation – a dying rabbit, signifying the flip side of technological progress, would be a living, breathing form that conveys this message. It carries the burden of the ill-affects of technological progress. We wanted to create an interactive art installation that would react to the environment around it as well as well to the virtual world beyond it. When people would come near it, it would start breathing heavily. The ears of the rabbit would twitch based on the sentiments of the tweets going around the virtual world about things like #recycling, #electronic waste, #environment.  The legs would move involuntarily would be depending on how the leading technology companies are doing on the stock market that day. The eyes would play a flashback of the dying rabbit’s life, when it used to live in a more greener and cleaner planet.

We started sketching some of our ideas on Friday night itself and Saturday morning we had started to put the pieces together. Mark and Saaid (he joined us on Satuday) started building the exoskeleton of the model. Meredith and me were handling the movements of the rabbit based on its sensing of the environment.

If the details below get too technical for you, you can jump to the end here.

The “technical” parts of building the model consisted of mainly 4 things -

- Getting the sentiment analysis of the tweets on relevant topics like environment, recycling etc., pushing those values to an Arduino controlled motors so as to make the ears twitch.

- Using a sonar sensor to sense proximity of the people around it. If someone is very close by, the rabbit would start breathing heavily.

- Connecting a 2nd generation iPod to show the “flashback” of the dying bunny’s life. The iPod would be playing scenes from Animal planet.

- Getting the live stock market feeds of leading technology companies and control the movement of the legs based on their fluctuations.

We were able to implement the first 2 ideas above in the given timeframe of 24 hours and left the other 2 for version 2.0. I was in charge of the getting the sentiments of the tweets and passing to an Arduino and this is how I built it -

Googling for “twitter sentiment analysis” gives out a lot of results and the one that I used was this. What this does is take the last 100 tweets and run a supervised learning algorithm on them to analyze the tweets. The tweets are marked as “positive”, “negative” or “neutral” and then the final sentiment index is calculated based on them.
They have a super simple API call and all I had to do was write a php script to call their API with the parameters. It returns a JSON object which has the analysis of the individual tweets:

{
    "sentiment_index": 22,
    "results": [
        {
            "created_at": "Mon, 15 Aug 2011 00:26:55 +0000",
            "from_user_id_str": "360406363",
            "profile_image_url": "imgUrl"
            "from_user": "alissawehingec",
            "id_str": "102899029364588544",
            "sentiment": -1,
            "metadata": {
                "result_type": "recent"
            },
            "text": "Rain gardens: A practical solution for water pollution"
            "to_user_id": null,
            "id": 102899029364588540,
            "from_user_id": 360406363,
            "geo": null,
            "iso_language_code": "fr",
            "source": "http://twitter.com",
            "to_user_id_str": null
        },...
        //99 more such elements
        "positive": 16,
        "negative": 72,
        "neutral": 12,
        "tweets_analyzed": 100

This is what my php script to make the call looked like

		$topics = array("recycle","plastic","ewaste", "environment","planet","global%20warming","prakriti", "risaikuru", "earth");
		$max = count($topics)-1;
		$index =  rand(0 , $max );
		$url = "http://data.tweetsentiments.com:8080/api/search.json?topic=".$topics[$index];
		$curl = curl_init();
		curl_setopt($curl, CURLOPT_URL, $url );
		curl_setopt( $curl, CURLOPT_RETURNTRANSFER, 1 );
		$result = curl_exec( $curl );
		curl_close($curl);
		$result = utf8_encode($result);
		$data = json_decode($result, true);
                // to get the actual values of the sentiments
                $sentiment_index = intval($data["sentiment_index"]);
		$negative = intval($data["negative"]);
		$positive = intval($data["positive"]);
		$neutral =  intval($data["neutral"]);

Once I had the values, I needed to push these values to the Arduino over the Serial port. I was able to use the guide here (I am running Ubuntu) to connect the Arduino and send data over serial using php.
It took me a long time to get this running. All I needed to do was to send 4 key value pairs (index,value). Finally the way I got it to work was to send data encoded in ASCII format (well now it sounds so obvious!). The Serial.read() command of Arduino reads the values that it gets as ASCII values. So for example, the character ‘a’ sent over serial would be read as 97 by Serial.read(). So I had to send numbers (the values) which I encoded in ASCII and sent. So if the value was 62, I sent the value as if it was the character ‘>’ (using the chr() method of php). Here is the code -

                // the protocol followed was s/p/g/n preceded the values
                // the characters s,p,g,n acted as flags
		$serial->sendMessage('s');
		$serial->sendMessage(chr($sentiment_index));
		$serial->sendMessage('p');
		$serial->sendMessage(chr($positive));
		$serial->sendMessage('g');
		$serial->sendMessage(chr($negative));
		$serial->sendMessage('n');
		$serial->sendMessage(chr($neutral));

Final php file can be downloaded here and the Serial class can be found here.
On the Arduino side, this code below reads the bytes sent over the Serial port and if it gets an ‘s’, it reads the next byte which is the value of the sentiment index, if it gets a ‘p’, then reads the next byte which is the value of the positive tweets,.. so and so forth

int neutral, positive, negative, sentiment;
void loop() {
   if(Serial.available() > 0) {
    number_in = Serial.read();
    if(number_in == 112){ //p
      // its the positive number of tweets
      positive = Serial.read();
      Serial.println(positive, DEC);
    }
    if(number_in == 115){ //s
      // its the sentiment index of tweets
      sentiment = Serial.read();
      Serial.println(sentiment, DEC);
    }
    if(number_in == 110){ //neutral - m
      // its the neutral number of tweets
      neutral = Serial.read();
      Serial.println(neutral);
    }
    if(number_in == 103){ // negative - g
      // its the negative number of tweets
      negative = Serial.read();
      Serial.println(negative);
    }
  }
  else{
      Serial.println("Still waiting for serial");
  }
    delay(300);
    number_in = 0;
}

The final Arduino file
Now that I have the values on the Arduino side, I can manipulate the rotation of the servo motors that control the movement of the ears.All we wanted to achieve was to move the left ear slowly (harmoniously) and the right ear violently. Due to lack of time I did not do any great mathematical manipulation, so it was just random servo movements in a loop.
PS -The final php and arduino files can be downloaded here. I wrote a similar version for sending data over serial in python (but it runs only once)
Meredith handled the part of the proximity sensing and making the dying rabbit breathe faster. Mark and Saaid were instrumental in building the model. It was fantastic work in limited amount of time. In the end, the 4 of us were happy to have been a part of it!

Here is the final video of Project Cycles that we presented

Yes, it was indeed one of the best weekends ever! I happened to meet a whole variety of people – artists, designers, developers and actually spent time building something tangible. There was this sense of elation even though we did not win. These were the other projects in the Art Hack.

ASL Dictionary Greasemonkey script v0.1

Research says that around 90% of hearing impaired children are born to hearing parents. So if parents do not know Sign Language then it is difficult for the child to learn ASL quickly. Wouldn’t it be great if people could learn sign language on the fly when they are reading articles on the web? Maybe they are reading stories to their children from a website and they can have ready assistance to translate words in written language to Sign Language. So, I thought of creating a simple Greasemonkey script that would show the ASL equivalent of a selected word.
Center for Accessible Technology in Sign at GeorgiaTech has a huge database of words and their American Sign Language equivalent. I used their web pages that they had created for around 25000 words. Whenever a word is selected, I load the ASL equivalent in an iframe on the right top corner of the page.

To Install

Here are the steps to try it on Firefox

1. Open Firefox (version 3.0 – 4.0.*)
2. Install Greasemonkey if you are using Firefox
3. Click on this link to download the script
4. Let Greasemonkey install the user script.
5. Now open any webpage say – Wikipedia entry on ASL
6. Now select any word (not phrases/sentences) – for example – Sign, Language, also, common, English or languages

You will see the American Sign Language representation of the word on the top right.
If a word that does not exist in the database is selected then, it shows “Sorry word could not be found”.

Here are the steps to try it on Chrome

1. Open Chrome
2. Click on this link to download the script
3. Let Chrome install the user script.
4. Now open any webpage say – Wikipedia entry on ASL
5. Now select any word (not phrases/sentences) – for example – Sign, Language, also, common, English or languages

You will see the American Sign Language representation of the word on the top right.
If a word that does not exist in the database is selected then, it shows an error message.

An example, with the word Languages highlighted

Another example, with the word Sign highlighted

The representation automatically hides after 5 seconds. (So if you click on another word before 5 seconds, the results may be buggy)
Update

To Uninstall
To uninstall on Firefox, click on the Greasemonkey icon and disable the ASL Dictionary.
To uninstall, open chrome://extensions/ and disable the ASL Dictionary.

For future versions
1) If you select a phrase, it shows the sequence of signs
2) Fix “Word not found” error in Chrome
3) Ability to select the word, right click and have the option to play the video or the cartoon