My python environment somehow got messed up, I suspect due to the fact that I had at least three different python installations (system, official installer, homebrew) and lots of modules installed into the system site-packages using random commands (pip install, sudo pip install, easy_install, ...).

So I've resolved to be more disciplined about my python environment. To document the cleanup and setup of a new python environment, as well as my set of best practices (henceforth known as "ben practices") I decided to write this blog post.

Most of the hygine related setup is stolen from hackercodex.com.

Cleanup

The first order of business is to remove all the (non-system) python installations that I've got floating around.

  1. homebrew python is easy to uninstall, but you have to make sure you nuke the site-packages directory its Setuptools/Distribute uses. This should be:

    $ brew uninstall python --force
    $ brew unlink python
    $ rm -rf `brew --prefix`/lib/python2.7/site-packages
    
  2. Python from the official bindist doesn't (!) come with an uninstaller. You have to get medieval on it:

    $ rm -rf /Library/Frameworks/Python.framework
    $ rm -rf /Applications/Python*
    $ find /usr/local/bin -type l -lname '*Python.framework*' -delete
    
  3. other site-package directories where things have ended up over the last year:

    $ rm -rf /Library/Python/2.7/site-packages
    

Install new Python

I'm going to use the homebrew python since it is easy to manage (e.g. uninstall) and doesn't require admin priviledges to install. I'm also installing the version whick links to homebrew's openssl (not that I've ever used openssl for anything myself, but this was recommended.

$ brew install openssl
$ brew install python --with-brewed-openssl
$ brew link python

(this last step turned up errors relating to strange links in my /usr/local/bin. The linking can be forced using --overwrite.)

The key piece of my new rigorous Python hygine is virtualenv, so install that:

$ pip install virtualenv

This failed in a spectacular way the first time I tried, turns out the failure is due to distribute being upgraded, so I did that (removing some stale links by hand along the way):

$ pip install distribute --upgrade

(fails because of stale links in /usr/local/bin/)

$ rm /usr/local/bin/easy_install
$ rm /usr/local/bin/easy_install-2.7

$ pip install distribute --upgrade

Now let's try virtualenv and make sure it runs and is in my path (because on some previous attempts it was not installed into my path):

$ which virtualenv
/usr/local/bin/virtualenv

$ virtualenv
You must provide a DEST_DIR
Usage: virtualenv [OPTIONS] DEST_DIR
...

New Python base setup -- check!

Put a padlock on the fridge

Now I have to keep my global Python environment clean, this means installing dependency modules for different projects into sandboxes (using virtualenv) and disallowing myself from installing modules globally in a moment of weakness or desperation. The following environment variables acheive this:

export PIP_REQUIRE_VIRTUALENV=true
export PIP_VIRTUALENV_BASE=$HOME/Virtualenvs
syspip(){ PIP_REQUIRE_VIRTUALENV="" pip "$@" }

Some other useful settings:

export VIRTUALENV_DISTRIBUTE=true
export PIP_DOWNLOAD_CACHE=$HOME/.pip/cache

Let's see if it works:

$ pip list
Could not find an activated virtualenv (required).

$ syspip list
closure-linter (2.3.8)
distribute (0.7.3)
flake8 (2.0)
ipython (0.13)
ipython-doctester (0.2.2)
ipython-extensions (0.2)
matplotlib (1.2.1)
virtualenv (1.9.1)
wsgiref (0.1.2)
wxPython (2.8.12.1)
wxPython-common (2.8.12.1)

Yes, it worked! but damn! I've still got a bunch of random modules in my site-packages despite trying to wipe them all out earlier. These are nuke proof modules, like cockroaches of the python world..

$ syspip show wxPython
---
Name: wxPython
Version: 2.8.12.1
Location:
/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7/site-packages/wx-2.8-mac-unicode

This one comes from having a .pth file in /Library/Python/2.7/site-packages which is in my PYTHONPATH for some reason. Anyway, I nuked all those and basically redid all the steps here starting from Clean and now the package list according to pip looks good!

$ syspip list
distribute (0.7.3)
virtualenv (1.9.1)
wsgiref (0.1.2)

Not sure why wsgiref is there, but it probably comes along with the homebrew python bindings for openssl (my guess).

Using the new setup

So now I'm left to actually work in this clean environment. I'll start by installing what I need to power this blog.

$ cd ~/Virtualenvs
$ virtualenv bfj7.com
New python executable in bfj7.com/bin/python
Installing
distribute.....................................................done.
Installing pip................done.
$ . bfj7.com/bin/activate
(bfj7.com) $ cd ~/git/bfj7.com
(bfj7.com) $ pip install markdown pelican

Comments

comments powered by Disqus