In my experience on both sides of the iron, as a sysadmin and developer, none of them work as one would like to.

But first, let’s explore what several HPC centers have adopted as a solution and why… and most importantly, how to fix it eventually.

The good

Widely used in different research facilities, the module system allows users to choose different versions of several software. The approach is simple, just type “module load program/1.0″ and off you go.

On the sysadmin side, it’s the same old familiar spell “tar xvfz && make && make install”, and a “vim program” to define the module script that will set PATH, LD_LIBRARY or other variables and whatnot.

Consequently, the time required to wrap a software is minimal, conferring sysadmins with speedy quick hack superpowers. After all, velocity in research does matter, and getting things done to let research continue its way is mandatory.

Moreover, user-coded modules can be shared easily within the same cluster by simply tweaking MODULEPATH variable. What’s the catch ? Technical debt and most importantly, lack of automation.

The bad

Software packaging is a time consuming task that shouldn’t be kept inside institutional cluster firewalls, but openly published. Indeed, a single program could have been re-packaged a number of times on each academic cluster for each university department that has HPC resources. When new versions come up for each package the sysadmin has to take care of bumping it by creating directories and additional recipes. How does one justify this time investment ? It just doesn’t scale. Skip to “solutions?” section for some relief.

From a technical perspective, using package systems that are not shipped with the operating system introduces an extra layer of complexity. More often than not, updates on the base distribution will break compiled programs that rely on old libraries. Stacking package managers should be considered harmful.

Ruby, python and perl have their own mature way to install packages for most UNIXes, stacking package managers by rpm-packaging python or ruby modules, has several bad consequences. Granted, there are some concerns on uniformity, updates and security, but those again can be solved by the individual package managers.

But getting back to the module system, how well does it play with cloud computing ?

It doesn’t, thankfully !

One would have to install all the modules, and re-package the software for the virtual instances. In contrast, existing package systems, be it rpm, deb, pip, gem or lein solved that by themselves. On top of that, the module system will tweak crucial system variables such as $PATH or $LD_LIBRARY_PATH with bad side effects for python virtual environments or any other user-defined PATHs.

Lastly, from a human resources perspective, writing modules does not add value or expertise to your IT toolbox. On the other hand, search engines have something to say when looking up search terms such as job experience packaging software rpm deb. Actually, being involved in open source communities via packaging can give you some very valuable insights on how open source projects work.

The ugly

With aging and relatively unmantained software, some bugs arise. Under some circumstances here’s what occurs:

$ modulecmd bash purge
*** glibc detected *** modulecmd: free(): invalid next size (fast): 0x0000000001b88050 ***
======= Backtrace: =========
(...)

$ export MODULEPATH=AAAAAAAAAAAAAAAAAAAAAAAAA:AAAAAAAAAAAAAAAAAAAAA:/bin/bash && modulecmd bash purge
    *** glibc detected *** modulecmd: corrupted double-linked list: 0x00000000009c4600 ***

I would like to light a candle for those who dare running modulecmd with suid bit. I can only think of one sysadmin that could do that while being totally self-confident.

Solutions?

Here’s some brainstorming that might help in the long run:

1. Instead of complicating infrastructure, just state the software versions you are running in your publications. In python, pip freeze helps. Want an older version ? DIY.
2. Use CDE, rbenv and virtualenv before and after publishing if concerned about platform updates during your research.
3. Use virtual machine images to reproduce experiments.
4. If you really need the module system, at least publish the modules somewhere for people to reuse them.
5. If you are a sysadmin, get started with FPM as a first approach with the world of package management for your distribution.
6. Try to get those packages accepted upstream (best!) and/or create your own rpm/deb repo.
7. Learn puppet and/or chef.

Please report any issues via comments !

1. Firsly, follow my earlier post on how to setup your own python virtual environment on UPPMAX.
2. Once you have a prompt similar to: (devel) hostname ~$, you can continue, else, jump to 1.
3. pip install drmaa Mercurial PyYAML
4. Add the following env variables to your .bashrc:

   export DRMAA_LIBRARY_PATH=/bubo/sw/apps/build/slurm-drmaa/lib/libdrmaa.so
   export DRMAA_PATH=$DRMAA_LIBRARY_PATH
   

5. Create a file ~/.slurm_drmaa.conf with the contents:

   job_categories: {
         default: "-A <your project_account> -p devel"
   }
   

6. hg clone http://bitbucket.org/brainstorm/galaxy-central
7. Edit universe_wsgi.ini from the provided sample so that it contains:

   admin_users = <your_admin_user>@example.com
   enable_api = True
   start_job_runners = drmaa
   default_cluster_job_runner = drmaa://-A <your project_account> -p devel
   

8. On your local machine: ssh -f <your_user>@<uppmax> -L 8080:localhost:8080 -N
9. On your local machine: Fire up your browser and connect to http://localhost:8080

As a betatester you may expect some issues when running galaxy in that way. Firstly, keep in mind that it’ll not perform as fast as a production-quality setup, it’s just a developer instance. Furthermore the node you’re in might have time limit restrictions, meaning that your instance will be killed in 30 minutes if you don’t reserve a slot beforehand as Martin recommended on the section “Run galaxy on a node”.

A few words about Galaxy

With a 15+ core team and a very active contributor base, Galaxy is trying hard to provide a fix for the biomedical Babel in which life scientists work nowadays.

From its modest origin as a single perl script, later on morphing into a python web framework, Galaxy evolved rapidly. In short, Galaxy can be thought as the glue code that wraps and uniformizes a considerable amount of bioinformatics programs into a more consistent web interface.

But there’s much more under the hood: cluster job management, data conversion, dataset access controls, security, web services, etc… to name a few components and features.

“Everything is possible in Galaxy, As long as you can run it on the command line, you can incorporate it into Galaxy.”
– Hans-Rudolf Hotz, Friedrich Miescher Institute for Biomedical Research

But not everything shines in the galaxy since NGS tool inclusion hogged its main site at some point. This fact only proves the point that single sites like Galaxy main, handling 130.000 cluster jobs/month and 1TiB uploads per week, face sustainability issues on the big datasets era we’re living in. As a result, other than imposing reasonable cluster quotas, interesting scaling strategies are being tested on real research projects. Therefore, federation and cloud computing are the next steps on this particular quest to the bio-universe.

One interesting realization on the conference is that not only labs are rolling their own Galaxy instances, there was a big sequencing industry player showing some interest on it too:

“Galaxy is an attractive workflow engine candidate”
– Kirt Haden, Illumina Inc

Common concerns

There are some FAQ I’ve been asked by colleagues and that came up on the past conference too. Therefore, I would like to keep them here for future reference, feedback and further questions are very welcome via the comments… and their support options.

Our compute cluster is not used due to IT policy restrictions, what should I do ?

You might want to run it as a single user, as Martin Dahlö describes, setting up a SSH tunnel. Obviously, this option has many problems when it’s aimed at non-developer scenarios: it does not scale. Make sure you explain the big picture and aim to reach consensus with your IT department. Again, some basic key insights and common sense might help here.

Are software versions kept in the history when running a workflow ?

During the conference Kanwei Li came up with some patch to keep track of the software versions by appending a <version> tag on a particular tool’s xml. This tag just runs a “–version” when the tool is used and appends this information in the history. You might want to ask him through galaxy-dev mailing list if you’re interested in this feature.

How’s Galaxy’s sample tracking moving forward ?

There are currently two big sample tracking systems present on the galaxy sphere: the one already present on main and some nextgen patches by Brad Chapman. Try them out, or better yet, join the upcoming BOSC2011 Codefest and improve or merge the current systems.

What is the API status ?

In short: it is growing as needed.
Shorter: @web.expose_api decorator.
Better explained: Read up by slide 26.

My cluster uses a custom job scheduler, will galaxy work with it ?

If your batch system supports DRMAA, there’s a better chance to get it rolling. Check out the recent progress on SLURM system for instance.

How does galaxy splitting of datasets for embarassingly parallel tools ?

There’s a ticket for that.

Next stop: BOSC2011 and ISMB

I have just covered a few topics shown in the conference, but you can take the time to explore it further, both by videos and slides. Obviously the galaxy evolution does not end up here, there’s still more to come in a few days in Vienna.

Both virtualenv and virtualenvwrapper ease the hassle of managing python modules when one does not have root access on a system. In addition, no more “–prefix” flags are needed when installing modules. Or maybe better explained, from the official docs:

The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python2.7/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.

Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application.

Also, what if you can’t install packages into the global site-packages directory? For instance, on a shared host.

In all these cases, virtualenv can help you. It creates an environment that has its own installation directories, that doesn’t share libraries with other virtualenv environments (and optionally doesn’t access the globally installed libraries either).

After this howto you’ll be able to create an isolated clean python environment where you can install as many python modules as you want and where your PYTHONPATH, PYTHONHOME and friends are not tainted… unless there’s a module system in the way, oh, my !

We’ll see how to tame that beast too. Keep reading.

Go ahead

First we’ll edit our .bashrc. The “~/opt/mypython” directory is needed in order to bootstrap the virtualenvs:


# User specific aliases and functions

export PATH=$PATH:~/opt/mypython/bin
export PYTHONPATH=~/opt/mypython/lib/python2.6/site-packages
source ~/opt/mypython/bin/virtualenvwrapper.sh
export WORKON_HOME=~/.virtualenvs


Then, we install virtualenv(wrapper) by running 3 commands:


$ source ~/.bashrc >& /dev/null && mkdir -p $HOME/opt/mypython/lib/python2.6/site-packages
$ easy_install --prefix=~/opt/mypython pip
$ pip install virtualenvwrapper --install-option="--prefix=~/opt/mypython" && source ~/.bashrc


Once we have that, we create a virtual environment called “devel”, or whatever name you prefer. That will ignore whatever is installed on the cluster (note the –no-site-packages flag):


$ mkvirtualenv --python=python2.6 --no-site-packages devel


Module system fixes

Finally, if you want this virtual environment to go along well with the module system in UPPMAX you should define the following code in ~/.virtualenvs/devel/bin/postactivate:


#!/bin/bash
# This hook is run after this virtualenv is activated.

# We unset PYTHONHOME set by the module system,
# otherwise the system will not use the python
# of virtualenv
unset PYTHONHOME


You can have a look at how I load my modules, or skip reload_uppmax_modules entirely if you use another approach.

Installing my crazy module

Installing new modules is as simple as running “pip”, no more 90′s .tar.bz2 and environment variables manual munging needed:


(devel)$ pip search cutadapt

Downloading/unpacking cutadapt
Downloading cutadapt-0.9.4.tar.gz (46Kb): 46Kb downloaded
Running setup.py egg_info for package cutadapt
Installing collected packages: cutadapt
Running setup.py install for cutadapt
building 'cutadapt.calign' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c lib/cutadapt/calignmodule.c -o build/temp.linux-x86_64-2.6/lib/cutadapt/calignmodule.o
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.6/lib/cutadapt/calignmodule.o -o build/lib.linux-x86_64-2.6/cutadapt/calign.so
changing mode of build/scripts-2.6/cutadapt from 644 to 755
changing mode of /home/romanvg/.virtualenvs/devel/bin/cutadapt to 755
Successfully installed cutadapt
Cleaning up...


See your “(devel)” prefix on your prompt ? Remember that you can create as many virtual environments as you want with the mkvirtualenv command shown above… And switch between those environments with the “workon” command:


$ cutadapt
-bash: cutadapt: command not found
$ workon devel
(devel)$ cutadapt --version
0.9.4


For the rubyists out there, there’s a similar tool called RVM and for the camels, there’s local::lib.

Choose your weapon and enjoy your new userland freedom

When one is developing a daemonized service, it’s rather usual to encounter minor errors that require no further attention than just restarting the daemon. That could be like not being able to connect to a remote machine for some time:


Traceback (most recent call last):
(...)
File "python2.6/urllib2.py", line 1170, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "python2.6/urllib2.py", line 1145, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 111] Connection refused>


Granted, we want to fix this on the code so that the daemon does not die, but meanwhile it’s good to have a safety net that we can rely on. That’s were supervisord comes in handy. Let’s see how it’s done.

Its documentation is comprehensive and well written, but one may need a “simplest” example out of the full-featured output from “echo_supervisord_conf”. The absolute barebones to get up and running is what I wanted to share with you:

; Ideally to be put under /etc/supervisord.conf
; you can refer to it via "supervisord -c"
; if the fileconf lays somewhere else
[supervisord]
nodaemon=true

[program:name_the_daemon_to_monitor]
command=/path/to/your/daemon

IMPORTANT: Since supervisord is meant to be the parent of all your monitored processes, you should not have instances of your program running before supervisord. Instead, supervisord will take care of (re)spawning them for you.

Now, we launch supervisord:

$ supervisord
2011-04-21 13:01:34,159 INFO supervisord started with pid 14806
2011-04-21 13:01:35,161 INFO spawned: 'analyze_sequences' with pid 14807
2011-04-21 13:01:36,162 INFO success: analyze_sequences entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Now, let’s do a reality check, kill the program it’s monitoring. In my case, “analyze_sequences”:

$ kill 14807

And supervisord should gracefully detect that its child dies and respawn it shortly after:

(...)
2011-04-21 14:09:12,764 INFO exited: analyze_sequences (exit status 143; not expected)
2011-04-21 14:09:13,768 INFO spawned: 'analyze_sequences' with pid 18380
2011-04-21 14:09:14,769 INFO success: analyze_sequences entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

So supervisord got that the exit status was unexpected (143 instead of 0 or 2) and put the program back into RUNNING state, sweet !

After this check, one would want to get rid of the “nodaemon” directive so that supervisord runs on background and revise echo_supervisord_conf command to include the fancier features it offers. But after what I’ve shown, the default settings seem very reasonable to me.

Det var lätt som en plett, eller ?