How to install python modules with VirtualEnv... on UPPMAX

Roman Valls Guimerà bio photo By Roman Valls Guimerà Comment

UPDATE:

This documentation below has been superseeded by a much simpler, generalized and automated alternative: VirtualEnv-burrito.

UPDATE 2:

Strike last update, seems that [pyenv][1] is now the canonical way to have a virtual python environment across python 2 and python 3…

Why bother ?

Both [virtualenv][2] and [virtualenvwrapper][3] ease the hassle of managing python modules when one does not have root access on a system. In addition, no more “–prefix” flags are needed when installing modules. Or maybe better explained, from the official docs:

The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python2.7/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.

Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application.

Also, what if you can’t install packages into the global site-packages directory? For instance, on a shared host.

In all these cases, virtualenv can help you. It creates an environment that has its own installation directories, that doesn’t share libraries with other virtualenv environments (and optionally doesn’t access the globally installed libraries either).

After this howto you’ll be able to create an isolated clean python environment where you can install as many python modules as you want and where your PYTHONPATH, PYTHONHOME and friends are not tainted… unless there’s a [module system][4] in the way, oh, my !

We’ll see how to tame that beast too. Keep reading.

Go ahead

First we’ll edit our .bashrc. The “~/opt/mypython” directory is needed in order to bootstrap the virtualenvs:

`

User specific aliases and functions</p>

export PATH=$PATH:~/opt/mypython/bin
export PYTHONPATH=~/opt/mypython/lib/python2.6/site-packages
source ~/opt/mypython/bin/virtualenvwrapper.sh
export WORKON_HOME=~/.virtualenvs
` Then, we install virtualenv(wrapper) by running 3 commands: `
$ source ~/.bashrc >& /dev/null && mkdir -p $HOME/opt/mypython/lib/python2.6/site-packages
$ easy_install --prefix=~/opt/mypython pip
$ pip install virtualenvwrapper --install-option="--prefix=~/opt/mypython" && source ~/.bashrc
` Once we have that, we create a virtual environment called "devel", or whatever name you prefer. That will ignore whatever is installed on the cluster (note the –no-site-packages flag): `
$ mkvirtualenv --python=python2.6 --no-site-packages devel
` If you need another python version (such as 2.7) you first have to deactivate the [module system][5] and install a new clean environment: `
module unload python && mkvirtualenv --no-site-packages --python=/sw/comp/python/2.7_kalkyl/bin/python py2.7
` ## Module system fixes Finally, if you want this virtual environment to go along well with [the module system][4] in [UPPMAX][6] you should define the following code in **~/.virtualenvs/devel/bin/postactivate**: `
#!/bin/bash
# This hook is run after this virtualenv is activated.

source ~/bin/reload_uppmax_modules.sh

# We don't want UPPMAX's custom python
PATH="${PATH/'/sw/comp/python/2.6.6_kalkyl/bin'}"

# We unset PYTHONHOME set by the module system,
# otherwise the system will not use the python
# of virtualenv
unset PYTHONHOME
` You can have a look [at how I load my modules][7], or skip reload\_uppmax\_modules entirely if you use another approach. ## Installing my crazy module Installing new modules is as simple as running "pip", no more 90′s .tar.bz2 and environment variables manual munging needed: `
(devel)$ pip search cutadapt

cutadapt - trim adapters from high-throughput sequencing reads

(devel)$ pip install cutadapt

Downloading/unpacking cutadapt
Downloading cutadapt-0.9.4.tar.gz (46Kb): 46Kb downloaded
Running setup.py egg_info for package cutadapt
Installing collected packages: cutadapt
Running setup.py install for cutadapt
building 'cutadapt.calign' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c lib/cutadapt/calignmodule.c -o build/temp.linux-x86_64-2.6/lib/cutadapt/calignmodule.o
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.6/lib/cutadapt/calignmodule.o -o build/lib.linux-x86_64-2.6/cutadapt/calign.so
changing mode of build/scripts-2.6/cutadapt from 644 to 755
changing mode of /home/romanvg/.virtualenvs/devel/bin/cutadapt to 755
Successfully installed cutadapt
Cleaning up...
` See your "(devel)" prefix on your prompt ? Remember that you can create as many virtual environments as you want with the **mkvirtualenv** command shown above... And switch between those environments with the "workon" command: `
$ cutadapt
-bash: cutadapt: command not found
$ workon devel
(devel)$ cutadapt --version
0.9.4
` For the [rubyists][8] out there, there's a similar tool called [RVM][9] and for the [camels][10], there's [local::lib][11]. Choose your weapon and enjoy your new [userland][12] freedom ;) [1]: http://www.uppmax.uu.se/how-to-install-your-own-python-modules-or-specific-python-version [2]: http://pypi.python.org/pypi/virtualenv [3]: http://www.doughellmann.com/projects/virtualenvwrapper/ [4]: http://modules.sourceforge.net/ [5]: http://blogs.nopcode.org/brainstorm/2011/11/23/module-system-bad-and-ugly/ [6]: http://www.uppmax.uu.se/ [7]: https://raw.github.com/brainstorm/scilifelab/master/scripts/reload_uppmax_modules.sh [8]: http://www.ruby-lang.org/en/ [9]: https://rvm.beginrescueend.com/ [10]: http://www.perl.org/ [11]: http://terrarum.net/development/perl-virtual-environments.html#locallib [12]: http://en.wikipedia.org/wiki/User_space

comments powered by Disqus