UPDATE:
This documentation below has been superseeded by a much simpler, generalized and automated alternative: VirtualEnv-burrito.
UPDATE 2:
Strike last update, seems that pyenv is now the canonical way to have a virtual python environment across python 2 and python 3…
Why bother ?
Both virtualenv and virtualenvwrapper ease the hassle of managing python modules when one does not have root access on a system. In addition, no more “–prefix” flags are needed when installing modules. Or maybe better explained, from the official docs:
The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python2.7/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.
Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application.
Also, what if you can’t install packages into the global site-packages directory? For instance, on a shared host.
In all these cases, virtualenv can help you. It creates an environment that has its own installation directories, that doesn’t share libraries with other virtualenv environments (and optionally doesn’t access the globally installed libraries either).
After this howto you’ll be able to create an isolated clean python environment where you can install as many python modules as you want and where your PYTHONPATH, PYTHONHOME and friends are not tainted… unless there’s a module system in the way, oh, my !
We’ll see how to tame that beast too. Keep reading.
Go ahead
First we’ll edit our .bashrc. The “~/opt/mypython” directory is needed in order to bootstrap the virtualenvs:
`
User specific aliases and functions
export PATH=$PATH:~/opt/mypython/bin
export PYTHONPATH=~/opt/mypython/lib/python2.6/site-packages
source ~/opt/mypython/bin/virtualenvwrapper.sh
export WORKON_HOME=~/.virtualenvs
`
Then, we install virtualenv(wrapper) by running 3 commands:
<br />
$ source ~/.bashrc >& /dev/null && mkdir -p $HOME/opt/mypython/lib/python2.6/site-packages<br />
$ easy_install --prefix=~/opt/mypython pip<br />
$ pip install virtualenvwrapper --install-option="--prefix=~/opt/mypython" && source ~/.bashrc<br />
Once we have that, we create a virtual environment called “devel”, or whatever name you prefer. That will ignore whatever is installed on the cluster (note the –no-site-packages flag):
<br />
$ mkvirtualenv --python=python2.6 --no-site-packages devel<br />
If you need another python version (such as 2.7) you first have to deactivate the module system and install a new clean environment:
<br />
module unload python && mkvirtualenv --no-site-packages --python=/sw/comp/python/2.7_kalkyl/bin/python py2.7<br />
Module system fixes
Finally, if you want this virtual environment to go along well with the module system in UPPMAX you should define the following code in ~/.virtualenvs/devel/bin/postactivate:
`
#!/bin/bash
This hook is run after this virtualenv is activated.
source ~/bin/reload_uppmax_modules.sh
# We don’t want UPPMAX’s custom python
PATH=“${PATH/‘/sw/comp/python/2.6.6_kalkyl/bin’}”
# We unset PYTHONHOME set by the module system,
otherwise the system will not use the python
of virtualenv
unset PYTHONHOME
`
You can have a look at how I load my modules, or skip reload_uppmax_modules entirely if you use another approach.
Installing my crazy module
Installing new modules is as simple as running “pip”, no more 90′s .tar.bz2 and environment variables manual munging needed:
<br />
(devel)$ pip search cutadapt</p>
<p>cutadapt - trim adapters from high-throughput sequencing reads</p>
<p>(devel)$ pip install cutadapt</p>
<p>Downloading/unpacking cutadapt<br />
Downloading cutadapt-0.9.4.tar.gz (46Kb): 46Kb downloaded<br />
Running setup.py egg_info for package cutadapt<br />
Installing collected packages: cutadapt<br />
Running setup.py install for cutadapt<br />
building 'cutadapt.calign' extension<br />
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c lib/cutadapt/calignmodule.c -o build/temp.linux-x86_64-2.6/lib/cutadapt/calignmodule.o<br />
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.6/lib/cutadapt/calignmodule.o -o build/lib.linux-x86_64-2.6/cutadapt/calign.so<br />
changing mode of build/scripts-2.6/cutadapt from 644 to 755<br />
changing mode of /home/romanvg/.virtualenvs/devel/bin/cutadapt to 755<br />
Successfully installed cutadapt<br />
Cleaning up...<br />
See your “(devel)” prefix on your prompt ? Remember that you can create as many virtual environments as you want with the mkvirtualenv command shown above… And switch between those environments with the “workon” command:
<br />
$ cutadapt<br />
-bash: cutadapt: command not found<br />
$ workon devel<br />
(devel)$ cutadapt --version<br />
0.9.4<br />
For the rubyists out there, there’s a similar tool called RVM and for the camels, there’s local::lib.
Choose your weapon and enjoy your new userland freedom :)