The cake is a lie

Don’t put all your Python eggs in one basket

One of my favourite things about Python is how I can package up my module into an egg and easy_install it from a tarball, source export, or even an http link. All you need is setuptools and you’re off to the races, but there’s a few things to watch out for:

  • There’s no easy_uninstall. To remove an egg, you would need to go into /usr/lib/python2.5/site-packages/easy-install.pth, manually prune out the offending line, and finally rm -rf the directory container of that egg. Oh, and carefully pick out the various binary scripts the egg installed into /usr/bin. Good luck.
  • What if you’re working on two projects that require different versions of a specific egg? Having to re-easy_install a different version of an egg every time you switch contexts is no fun.
  • Didn’t your mother teach you that it’s always better to not do things as root?

The solution is virtualenv.

Virtualenv creates a local copy of everything you need for maintaining your eggs. The prescribed way to use it is to virtualenv ~/python1 and then execute ~/python1/bin/activate to enter into the environment of that local installation. From then on, anything you easy_install will go into your local installation instead of your root install. If zombies attack and you need to purge your site-packages of any disease, you can scrap the whole thing and start a-new — try to do that with your root install without losing hair.

The shazow method

The default use of virtualenv is great, but I like to take things a little further. I setup a permanent local install as soon as I log into a fresh Linux user. This way, I never have to install any eggs as root.

sudo easy_install virtualenv
virtualenv ~/local

And add these two lines to your ~/.bashrc or equivalent:

export PYTHONPATH=~/local/lib/python2.5:$PYTHONPATH
export PATH=~/local/bin:$PATH

Now I never have to activate anything, I just happily use my local install without worrying about making a mess in my root. If I need to have different versions of things, I create a separate virtualenv install and activate that whenever I need it as mentioned before.

It’s nice to have your very own contained egg basket. This is especially useful for deploying production code on servers. Production environments often require very specific versions of packages, so deploying multiple Python apps on the same server is made much easier with virtualenv.

Update: Another good idea is to use --no-site-packages with virtualenv, this way the root PYTHONPATH can be omitted and all of your package dependencies will live entirely in the virtual environment. (Thanks, Chris! Check out his Virtualenv and PasteScript screencast for more on the topic.)

IPython caveat!

For those of you who have discovered the incredible life-changing tool of never-ending code-gasms that is IPython, there’s one extra step you’ll need to perform: It turns out that IPython saves some hard-coded paths when it’s installed, so you’ll need to re-install it with your new virtualenv active before it will work properly.

4 Comments so far

  1. Phillip J. Eby September 2nd, 2008 8:47 am

    Actually, you use “easy_install -m packagename” to remove it from easy-install.pth… and that will also dump out for you what scripts need to be removed.

    Second, if you need multiple versions of an egg installed, that’s what “easy_install -m” is for. You only need to re-run easy_install to change the *default* version - the one that comes up when the Python interpreter is started, not what comes up when you run a given project’s scripts.

    Not that virtualenvs aren’t great, but those two things don’t require them.

  2. shazow September 2nd, 2008 11:14 am

    @Phillip Ah, thanks for the tip! Unfortunately, easy_install -m doesn’t seem to cover the entire scenario. It does remove the entry from easy-install.pth, and it does print out what scripts it put into your bin directory, but it doesn’t remove the egg from your site-packages, nor does it prevent overwriting of other binaries with the same name.

    As for using the multi-version mode, it doesn’t really help you as much if the code in question does not use pkg_resources, but it is a step in the right direction.

    Overall, it’s still better to have a separate user-local Python site-packages, but I will keep -m in mind.

  3. Chris Perkins September 9th, 2008 2:47 pm

    I am surprised you do not use –no-site-packages. I find this indispensable. I have done a short v-cast on how to create a virtualenv and use it effectively:

    http://showmedo.com/videos/video?name=2910000&fromSeriesID=291

  4. shazow September 9th, 2008 2:53 pm

    @Chris you’re right! –no-site-packages is a great idea, I just use this setup on my own single-user machines so it’s not as much of a factor. It’s definitely recommended for multi-version situations and production environments. I’ll edit the post accordingly.

    Thanks!

Leave a reply

This is a captcha-picture. It is used to prevent mass-access by robots. (see: www.captcha.net)

You must read and type the 5 chars within 0..9 and A..F, and submit the form.

  

Oh no, I cannot read this. Please, generate a