The power of Ubuntu – showing dynamic messages in your desktop background!

I worked on this cool hack to dynamically show Twitter messages embedded into the desktop background. The basic idea is to have some dynamic text (which could be fetched from the web) embedded in an SVG image, which is set as the desktop background. The SVG image contains the actual wallpaper that we intend to use.


[ad name=”blog-post-ad-wide”]

Here are the steps:

  1. We first start by creating an SVG template file called wall-tmpl.svg with the following contents and saving it in the Wallpapers directory (let’s say it is ~/Theme/Wallpapers):
    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd" [
    <!ENTITY ns_imrep "http://ns.adobe.com/ImageReplacement/1.0/">
    <!ENTITY ns_svg "http://www.w3.org/2000/svg">
    <!ENTITY ns_xlink "http://www.w3.org/1999/xlink">
    ]>
    <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="1280" height="1024" viewBox="0 0 1280 1024" overflow="visible" enable-backgroun
    d="new 0 0 132.72 127.219" xml:space="preserve">
    <image xlink:href="~/Theme/Wallpapers/-your-favorite-wall-paper-" x="0" y="0" width="1280" height="1024"/>
    <text x="100" y="200" fill="white" font-family="Nimbus Mono L" font-size="14" kerning="2">%text</text>
    </svg>
  2. Next we create a script to fetch the most recent Twitter message and then embedding it in the image. The script is called change-wallpaper and is placed in the bin directory. It has the following:
    text=`python -c "import urllib;print eval(urllib.urlopen('http://search.twitter.com/search.json?q=ubuntu&lang=en').read().replace('false', 'False').replace('true', 'True
    ').replace('null', 'None'))['results'][0]['text'].replace('\!','').replace('/','\/')"`
    cat ~/Theme/Wallpapers/wall-tmpl.svg | sed "s/%text/$text/g" > ~/Theme/Wallpapers/wall.svg
  3. We then add the following entry to crontab to fetch Twitter messages every minute:
    # m h dom mon dow command
    * * * * * ~/bin/change-wallpaper
  4. Run the script once, it will create a file called wall.svg in your Wallpapers directory. Set this as your desktop background and watch the background change every minute!

You could get very creative with this. You could have your calendar reminders embedded directly into your desktop background or you could have dynamically fetched background images with your own random fortune quote. The possibilities are enormous!

My weekend hack – Twitter Trending Topics

I got this idea of building an application which pulls all the pages mentioned in the trending topics on Twitter. Why would that be useful? Well, it’s the simplest replacement for Google News, but more real time and no tweet noise.



Here are the steps I followed to build this application:

  1. The first step was to use IPython and use the Twitter Search API to get the latest tweets.
  2. I then wrote the code to parse these tweets looking for URL’s in them.
  3. The next step was to get the content from these URL’s, and get the title of the pages.
  4. Next, I had to persist it in the store.
  5. Slap a front-end and allow navigation. At this point, the obvious choice for me was Google AppEngine, since it is the cheapest hosting alternative available. I had to make some changes to the application to accommodate it to Google AppEngine’s requirements, but they were mostly trivial.
  6. Build the styles, the icons, the pretty URL’s and you are done!

The initial setup of the application was done in less than 2 hours time!

You can access the application here: Twitter Trending Topics.

There are a few known bugs, but the overall results are impressive.

The Afterthoughts – Gmail forwarding and service interoperability – an interesting observation

“The Afterthoughts” is a series where I revisit some of my older blog entries and see how things have changed since the time I made the blog post and now.

The posts that I will choose initially will be from 2004 to 2006.

So here is the first one in the series:

Post: Gmail forwarding and service interoperability – an interesting observation
Originally posted on: 2005-11-21

The entry goes about explaining how when you connect various services together, you could end up with the same information multiple times.

This is increasingly becoming a problem these days. Services like Twitter and Friendfeed are not solving the problem elegantly, so you see more and more duplicates and links to the original post.

Here is a typical scenario today:
I make a blog entry. In order to ensure that my readers see my post immediately, I have a service that automatically posts a message in Twitter. This is like instantly messaging my friends (actually Twitter followers) telling them, “Look, I made a blog entry”.

Now, I use a lot of Web 2.0 services. So, in order to ensure that all my friends have a single feed to follow my activities, I use some aggregator like FriendFeed or Tumblr.

Some friend of yours (let's call him Bob) likes your blog entry and bookmarks it on del.icio.us. Another friend, Andrews bookmarks it in Magnolia.

Let us now say, there is another person Dave, who is a friend of you, Bob and Andrews. He is following all 3 of us in Friendfeed.

How many entries is Dave going to see of the original entry?
6 in total! 3 from you – 1 from your blog post directly, 1 from Twitter, 2 from Tumblr (1 via the blog post and 1 via Twitter), 1 from Bob via del.icio.us and 1 from Andrews via Magnolia.

The screenshot shows duplicate entries from mashable's blog feed and from Twitter:

Now this is real noise. And this is more true if Dave is not even interested in the blog post to begin with.

So the solution?
Friendfeed allows you to hide specific feeds from specific people. For example, Dave can hide all bookmarks from Bob or all Tumblr entries from me.

Now that is not a good solution because not all bookmarks from Bob are duplicates.

Tools like Feedblendr and Blogbridge have solved this problem for simple RSS aggregation. However things are different when it comes to social network and aggregation.

So right now there is no simple way of detecting duplicates and more and more people are complaining about this in the blogosphere explaining how Friendfeed is more noise than information and why the good old Google Reader is still relevant.

Here is one such discussion. As the discussion suggests, it is not just about eliminating duplicates; it also requires you to merge discussions/comments in each of these posts keeping in mind that not everyone is a friend of everyone else.

So what has changed over the last 2 years?
If anything, the problem has become a tougher one. I am sure the startup that does duplicate elimination and gives you a filtered feed taking your social networks into consideration is going to be the next hyped startup in the Web 2.0 world.

Privacy disasters with aggregation services

Imagine you have a host of aggregation services like Friendfeed, Tumblr, Suprglu, Lifestreams connected to each other, such that each one is reading from your various feeds and republishing the content.

Now imagine a disaster where one of these services, say Twitter, suddenly, because of some flaw, exposes your private messages.

It's like a Tsunami that cannot be controlled! Your private data would flow into various input streams in a matter of seconds and there is no turning back.

Things will only get worse with activity feeds and Beacon.

The bottom line is: Be careful about where your data is going and what data you put online.

Downloading your data using Greasemonkey

Whenever I use some service over the web, I look for several things. Ease of use and customisability are important factors.

However, the most important thing I consider is vendor lock-in (or rather the lack of it). Let's say I am using a particular mail service (ex, GMail). If someday, I find a better email service, would it be easy for me to switch to that service? How easy is it for me to transfer my data from my old service to my new service?

For services like Mail, there are standard protocols for data access. So this is not an issue. However for the more recent services, like blogging, micro-blogging etc, the most widely used data access methodology/format is 'HTTP' via 'RSS' or 'ATOM'.

However, it's not the case that all services provide data as RSS (or XML or in any other parseable form). For example, suppose I make a list of movies I have watched, in some Facebook application, or a list of restaurants I visited, how do I download this list? If I cannot download it, does it mean I am tied to this application provider forever? What if I have added 200 movies in my original service and I come across another service that has better interface and more features and I want to switch to this new service but not lose the data that I have invested time to enter in my original service?

In fact, recently when I tried to download all my Twitters, I realized that this feature has been disabled. You are not able to get your old Twitters in XML format.

So what do we do when a service does not provide data as XML and we need to somehow scrape that data and store it?

This is kind of related to my last blog entry.

So I started thinking of ways in which I could download my Twitters. The solution I thought of initially was using Rhino and John Resig's project (mentioned in my previous blog entry). However, I ran into parse issues like before. So I had to think of alternative ways.

Now I took advantage of the fact that Twitters are short (and not more than 140 characters).

The solution I came up with uses a combination of Greasemonkey and PHP on the server side:

Here is the GM script:
If you intend to use this, do remember to change the URL to post data to.

// @name           Twitter Downloader

// @namespace      http://buzypi.in/

// @author         Gautham Pai

// @include        http://www.twitter.com/*

// @description    Post Twitters to a remote site

// ==/UserScript==

function twitterLoader (){
	var timeLine = document.getElementById('timeline');
	var spans = timeLine.getElementsByTagName('span');
	var url = 'http://buzypi.in/twitter.php';
	var twitters = new Array();
	for(var i=0;i<spans.length;i++){
		if(spans[i].className != 'entry-title entry-content'){
			continue;
		}
		twitters.push(escape(spans[i].innerHTML));
	}

	for(var i=0;i<twitters.length;i++){
		var last = 'false';
		if(i == twitters.length - 1)
			last = 'true';
		var scriptElement = document.createElement('script');
		scriptElement.setAttribute('src',url+'?last='+last+'&data='+twitters[i]);
		scriptElement.setAttribute('type','text/javascript');
		document.getElementsByTagName('head')[0].appendChild(scriptElement);
	}
}

window.addEventListener('load',twitterLoader,true);

The server side PHP code is:

<?php

global $_REQUEST;
$data = $_REQUEST['data'];
//Store data in the DB, CouchDB (or some other location)
$last = $_REQUEST['last'];
if($last == 'true'){
	echo "
	var divs = document.getElementsByTagName('div');
	var j= 0;
	for(j=0;j<divs.length;j++){
		if(divs[j].className == 'pagination')
		break;
	}
	var sectionLinks = divs[j].getElementsByTagName('a');
	var href = '';
	if(sectionLinks.length == 2)
		href = sectionLinks[1].href;
	else
		href = sectionLinks[0].href;
	var presentPage = parseInt(document.location.href[document.location.href.indexOf('page')+'page'.length+1]);
	var nextPage = parseInt(href[href.indexOf('page')+'page'.length+1]);
	if(nextPage < presentPage)
		alert('No more pages to parse');
	else {
		alert('Changing document location');
		document.location.href = href;
	}
	";
} else {
	echo "
	var recorder = 'true';
	";
}

?>

The GM script scrapes the twitters from a page and posts it to the server using <script> includes. The server stores the twitters in some data store. The server also checks if the twitter posted was the last twitter in the page. If so, it sends back code to change to the next page.

Thus the script when installed, will post twitters from the most recent to the oldest.

Ok, now how would this work with other services?

The pattern seems to be:
* Get the data elements from the present page – data elements could be movie details, restaurant details etc.
* Post data elements to the server.
** The posting might require splitting the content if the length is more than the maximum length of the GET request URL.
* Identify how you can move to the next page and when to move to the next page. Use this to hint the server to change to the next page.
* Write the server side logic to store data elements.
* Use the hint from the client to change to the next page when required.

The biggest advantage of this method is we make use of the browser to do authentication with the remote service and also to do the parsing of the HTML (which, as I mentioned in my previous post, browsers are best at).

Microblogging experiences

I have been looking for the perfect micro-blogging service. In the last one week I tried 3 services: Twitter, Tumblr and Pownce.

Here is what I liked in each:

Twitter

The good

  • Simple and elegant. Gets the job done.
  • Has IM support.
  • Good Facebook integration.

The bad

  • The 140 character limit is more limiting than SMS.
  • All twitters are public.
  • No option to comment on twitters. The @ replies are confusing.

Tumblr

The good

  • Really cool interface. Ability to post text, photos, links, audio and video in one place.
  • Import entries from other services like del.icio.us, flickr, twitter etc.
  • Re-blogging.
  • Easy sharing of all my entries with people in my social-network.
  • Ability to comment on entries.
  • Entries can be either for self or public.
  • Good archive view.

The bad

  • No Facebook integration yet.
  • The timestamp on imported entries is wrong (possibly because of timezone differences).
  • Some of the entries are not imported – looks like a bug.

Pownce

The good

  • Simple interface.
  • Extensive privacy options for entries.
  • Looks like there is Facebook integration (not tried this yet).

The bad

  • There is no ability to import entries from other services.

All 3 services have API support.

I am still on the look out for a good microbloging service. I need some service that offers the simplicity of Twitter and the features of Tumblr. I will stick to Twitter until I see one service that helps me with all my requirements.