One of Google’s most used underacknowledged features is the I’m Feeling Lucky query. On the surface, it seems silly and beside the point, but many people have grown to depend on it. What is it good for? Link resolving. You could go to Youtube.com and type “Rickroll” into the search box, or you could type “rickroll video” into your Firefox address bar (effectively doing an I’m Feeling Lucky search) and jump straight to where you want to go.

I no longer type “wikipedia.org” when I can type “wikipedia“. Why type “mail.google.com” when I can just type “gmail“? To me, Google has become a kind of DNS — resolving keywords into a complete URL. At that point, it’s hardly even search — it’s just a convenient side effect.

Providing a transparent link resolving service seems like a great selfless gesture by our unevil Google overlords. Rescuing the lazy with transparency and anonymity… Well, not quite. It turns out that the I’m Feeling Lucky feature only really works if you run it from the Google homepage, and approximately half-works from anywhere else.

Getting technical…

Anyone can turn a normal Google query into an I’m Feeling Lucky query, just append btnI=1 to the end of the URL (contrary to popular belief, btnI can take on any value, even empty string).

Let’s examine the problem. Consider this query:
workerpool site:shazow.net

Try opening that link, and then to compare, try the same query from google.com by hitting I’m Feeling Lucky.

Notice how clicking the link brings you to the Google results page, but the same query from the Google homepage does what it’s supposed to — takes you to the first result. It’s starting to become evident what’s going on here, but let’s beat this dead horse into the ground by hand crafting some HTTP requests, for great justice.

$ nc google.com 80
GET /search?q=workerpool+site%3Ashazow.net&btnI=1 HTTP/1.1
User-Agent: Mozilla/5.0 (Oh hai, I can haz redirect?)
 
HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Date: Thu, 09 Oct 2008 01:07:47 GMT
Expires: -1
Content-Type: text/html; charset=UTF-8
...
 
$ nc google.com 80
GET /search?q=workerpool+site%3Ashazow.net&btnI=1 HTTP/1.1
User-Agent: Mozilla/5.0 (Oh hai, I can haz redirect?)
Referer: http://www.google.com/webhp?hl=en
 
HTTP/1.1 302 Found
Location: http://shazow.net/blog/2008/04/02/gentoo-rockband-code-and-music/
...

I used netcat to send a raw socket request (the first two lines) to google.com on port 80, and then Google responded (lines 4 and onwards). There are two requests, one is a vanilla request and the second includes a fake Referer header claiming that I was coming from the Google homepage, which yielded the 302 redirect we want.

Sometimes external I’m Feeling Lucky links work, other times they don’t. It appears that Google has some secret threshold to decide when to get in the way of your destination like an angry ceiling cat catapulting itself onto your face. It seems to happen when there’s a small number of results, or the magic search juice score of the top score is relatively low.

Sadface, but what can we do?

Google may not be completely selfless, but they’re still pretty good compared to some jerkwads. Thankfully, Google provides a no-strings-attached search API to their wonderful magic search juice. It’s not very well advertised (or clearly documented), but once you hunt it down, it’s simpler than sketching a wrinkly gourd. Observe: Same fail query as above in JSON.

Writing your own I’m Feeling Lucky redirector is a piece of cake. I took the liberty of making one on the Google App Engine with a few bonus features:
http://302found.appspot.com/ (source code @ github)

I had a dream last week about elevators that go through each other. After a bit of research, I found a few multi-car elevator shaft schematics and patents but they all seem to use a cyclic method of elevators cycling from one shaft and into another, instead of passing through each other.

Here’s my idea:

At least two shafts are required. One shaft is dedicated for people going up, the other is dedicated for people going down. Each shaft will contain many elevator cars, proportional to the number of floors. For an average residential building, 3 cars per shaft should be sufficient. Huge office buildings can have 5-10 cars per shaft.

Considering a single shaft (for people going up), it will work as follows:

People enter an “active” elevator car, the car goes up. Once the last person leaves the car, the elevator car transforms into “inactive” mode: The doors close and the floor collapses outwards to be on the same plane as each side wall. Then, each wall extends outwards one level, effectively allowing “active” elevators below it to pass through it. Once an elevator is in the “inactive” mode, it will make its way down to the bottom floor and wait until the last active elevator departed the bottom floor, at which point the walls will contract, the floor hatch will close, and the door will open waiting for new passengers.

Considerations

• Advantage: Elevator shafts are expensive. Consider a large office building with six to ten elevator shafts, imagine how much money would be saved if there were only two shafts with the same (or better) efficiency.
• Rush hour: This scheme is great for repetitive bursts of people. There should always be a steady stream of elevators, unlike current elevator systems where the shafts tend to accidentally get in sync and you end up waiting while all 4 elevators make their way down from the 40th floor.
• Danger: Collapsing floor instantly screams danger, yes. But with some thorough planning, it should be alright. The elevators cars can use sensitive weight sensors to insure that nothing remains in the elevator when the floor opens. If something refuses to leave the elevator, then the car will proceed to a special extra floor at the top (or bottom) of the building where further effort can be taken without interrupting the rest of the cars.
• Movement mechanism: The classic pulley setup wouldn’t work here. The columns on each corner of the shaft will need some kind of rails which each car can use to traverse and power itself.

In all practicality, it wont happen. But it’s still a cool idea. Who wants to write a simulator?

One of my favourite things about Python is how I can package up my module into an egg and easy_install it from a tarball, source export, or even an http link. All you need is setuptools and you’re off to the races, but there’s a few things to watch out for:

• There’s no easy_uninstall. To remove an egg, you would need to go into /usr/lib/python2.5/site-packages/easy-install.pth, manually prune out the offending line, and finally rm -rf the directory container of that egg. Oh, and carefully pick out the various binary scripts the egg installed into /usr/bin. Good luck.
• What if you’re working on two projects that require different versions of a specific egg? Having to re-easy_install a different version of an egg every time you switch contexts is no fun.
• Didn’t your mother teach you that it’s always better to not do things as root?

The solution is virtualenv.

Virtualenv creates a local copy of everything you need for maintaining your eggs. The prescribed way to use it is to virtualenv ~/python1 and then execute ~/python1/bin/activate to enter into the environment of that local installation. From then on, anything you easy_install will go into your local installation instead of your root install. If zombies attack and you need to purge your site-packages of any disease, you can scrap the whole thing and start a-new — try to do that with your root install without losing hair.

The shazow method

The default use of virtualenv is great, but I like to take things a little further. I setup a permanent local install as soon as I log into a fresh Linux user. This way, I never have to install any eggs as root.

sudo easy_install virtualenv
virtualenv ~/local

And add these two lines to your ~/.bashrc or equivalent:

export PYTHONPATH=~/local/lib/python2.5:$PYTHONPATH
export PATH=~/local/bin:$PATH

Now I never have to activate anything, I just happily use my local install without worrying about making a mess in my root. If I need to have different versions of things, I create a separate virtualenv install and activate that whenever I need it as mentioned before.

It’s nice to have your very own contained egg basket. This is especially useful for deploying production code on servers. Production environments often require very specific versions of packages, so deploying multiple Python apps on the same server is made much easier with virtualenv.

Update: Another good idea is to use --no-site-packages with virtualenv, this way the root PYTHONPATH can be omitted and all of your package dependencies will live entirely in the virtual environment. (Thanks, Chris! Check out his Virtualenv and PasteScript screencast for more on the topic.)

IPython caveat!

For those of you who have discovered the incredible life-changing tool of never-ending code-gasms that is IPython, there’s one extra step you’ll need to perform: It turns out that IPython saves some hard-coded paths when it’s installed, so you’ll need to re-install it with your new virtualenv active before it will work properly.

Went to Cuba a few weeks ago and I brought back this timelapse video for you.

Highlights from the trip:

• A female Cuban customs agent with a distinguished Stalin-like bushy mustache.
• Bottomless quantities of mediocre food where buffet leftovers aren’t discarded but re-served the next morning.
• Perfect ocean beaches with silky-smooth sand.
• Snorkeling with picturesque waterlife and enormous sea sponges slowly strafing their way across the waterbed.
• Sleeping.

There was a lot of sleeping.

Great experience. Recommended to anyone with a dire need of a real vacation or Cuban cigars. 3 comments

One of the perks of being sick and working from home is getting to answer all those fun telemarketer calls.

Telemarketer: Would you like a free Toronto Sun newspaper subscription for six weeks?
Me: No.
T: How about just a weekend?
M: No.
T: Do you read any other papers?
M: No…
T: Well then, how do you get your news, sir?
M: Online.
T: Do you realize you’re not getting the in-depth coverage?
M: Yes I am.
T: Have a good day sir. *Hangs up*

“News: If you’re not getting the in-depth coverage from the Toronto Sun, you’re doing it wrong.”

Rare as my posts are, Twitter has been siphoning whatever exhibitionist energy I have. You can follow me on Twitter if you’re interested in my day-to-day comments.

Why Twitter? Few years ago, I had an idea for a “one-liner blog” where one would post short one-line thoughts multiple times per day. Few months later, Twitter came along. Although it’s not exactly how I envisioned such a creation, it is useful nonetheless. In day-to-day usage, I get about 30% of my interesting news bits from various people I follow on Twitter. It’s great for getting a live real-life feed of what’s going on with the tubes, conferences, weather, sports, politics, or whatever you happen to be interested in — so long as you follow people who have similar interests as yourself.

Twitter has a lot of unspoken etiquette (tweetiquette) and it’s not for everyone, but I like it. It has a very small commitment threshold (unlike blog posts) and you feel more like part of a community.

I’ll still keep blogging, but I just wanted you to know where I’ve been.

You’ve heard of memcached, haven’t you?

Made by Livejournal to sustain millions of angst-ridden teen diaries, now used by web giants like Slashdot, Digg, Wikipedia, Facebook, Youtube, and countless more. Even this very Wordpress blog supports it.

The same misconception seems to haunt all those unfamiliar with the memcached: it’s some sort of magical layer that sits between your database and your application layer, and somehow it makes things faster. The magic is scary. How would something know what it needs to speed up from the database? It must require all sorts of nasty configuration, all sorts of nasty corner cases to cover?

Admit it. The thought of using memcached terrifies you. Well, I have good news for you: That’s not what memcached is. Are you read for it?

Memcached is a hash table.

No, seriously. It’s that simple. It’s a hash table that sits in memory, that can scale between multiple servers. All you do is launch the memcached daemon on your server with memcachedand you can use it!

“Well obviously that’s not all there is…” you say. It is! As far as the server is concerned, that’s all the configuration it needs.

Now, the client side…

How does the hash table work? It’s just like a dictionary. To save data: you send it a key/value pair, and it saves it. To retrieve data: you send it a key, and it gives you the value. Explicit. Simple. Efficient. It supports a handful of simple operations like set, get, and replace.

Here’s how to use 3 distributed memcached servers to cache a huge query in Python:

import memcache
...
cache = memcache.Client(['192.168.1.100:11211',
                         '192.168.1.101:11211',
                         '192.168.1.102:11211'])
...
def get_post(post_id):
    "Return data associated with post with given post_id"
    result = cache.get(post_id)
    if not result:
        # No cache, perform potentially costly SQL query
        result = model.Post.get_post(post_id)
        # Save result in cache for next time
        cache.set(post_id, result)
    return result

Do you see what they did there?

Yes, it’s not magic. Yes, you have to cache things explicitly. Yes, this is incredibly powerful. Now go. Go make a Slashdotting-proof web service, that’ll show them.

Today, you’ll get to hear about what I’ve been up to! This wont be on the exam.

• After a year and a half stint with Ubuntu, I’m back to my true love Gentoo. That is, with a shiny matte new quad core beast of a machine.
• Bought Rockband for my PS3 couple of weeks ago. Drums are ridiculously hard. Been rocking out. Good fun is being had by all those who dare rock. Can’t wait for Still Alive to be released for free. It shall be a triumph!
• What started out as pretty code is now a bonafide open source Python module: workerpool. People are using it. No, really.
• muxtape.com: A super simple music sharing web app launched last week. Its been enriching my life — doing what Pandora once did. Here’s my muxtape. Be right back, there’s someone at the door.

• Sony’s Blu-ray beat out Microsoft’s HD-DVD in the hi-def wars. Sony stock is going up. Microsoft stock not so much, especially with the recent bid at Yahoo.
• Scientology is now censoring eBay, as its has been permitted to arbitrarily delete auctions. Boycott time?
• J.J. Abrams, an ex-Scientologist and producer of Lost and Cloverfield, announced a Cloverfield sequal.
• In other Abrams news, Fringe (X-Files on drugs?) is being filmed in Toronto (UofT’s Bahen building, to be precise) as I write this. I dropped by today and saw an enormous FBI seal plastered on the lobby floor. Also, Great Hall has been revamped into an FBI HQ with suited agents running around. Very exciting.
• Bonus fun item: I learned that Naveen Andrews (Sayid from Lost) has a fetish for older women. He fathered a child with his math teacher at the age of 16, and is currently dating Barbara Hershey who is a solid 21 years older.