So, in addition to having some blueprints such as those proposed by the Semantic Versioning guidelines, one needs specifics on how to integrate those practices in our day to day work with version control systems.

Setuptools saves the day by introducing versioning via git tags. In a post by Douglas Creager a strategy to use setuptools with git tags is devised. The workflow for tagging a new version results in:

1. Tag your release via git tag if the changes are significant.
2. Run python setup.py install, to bump the version on the filesystem.
3. git push.

The following code makes it happen:

# Fetch version from git tags, and write to version.py.
# Also, when git is not available (PyPi package), use stored version.py.
version_py = os.path.join(os.path.dirname(__file__), 'version.py')

try:
    version_git = subprocess.check_output(["git", "describe"]).rstrip()
except:
    with open(version_py, 'r') as fh:
        version_git = open(version_py).read().strip().split('=')[-1].replace('"','')

version_msg = "# Do not edit this file, pipeline versioning is governed by git tags"
with open(version_py, 'w') as fh:
    fh.write(version_msg + os.linesep + "__version__=" + version_git)

setup(name="yourpythonpackage",
      version="{ver}".format(ver=version_git),

As an addition to the git tags workflow proposed by Douglas, the ‘__version__’ attribute will be stored in version.py file. This allows the versions to be tracked even when our git repository is not available (i.e, via PyPi package installation), or when such a version needs to be queried from inside your own package.

Thanks Guillermo and Brad for the feedback and suggestions on this strategy.

So there’s a day, that day when an organization realizes that there’s a real need to have a solid cloud platform as an official infrastructure offering. Admit it, we all have some idle cycles we could make better use of.

Why the Rest of Us Need Virtualization Even If Facebook Doesn’t feedproxy.google.com/~r/PuppetLabs/…

— Roman Valls (@braincode) April 27, 2013

A bad cloud

Then someone owning some computer resources types some commands frenetically in a console and voilà, a beta cloud service is born, a fiction dialog between user and cloud provider follows:

- This is great! I want to have an account on your new service, where can I get it?
- Well, you have to come at our offices, we will scan your passport, have a 1 hour long meeting and then give you an account.
- Ermm, ok, I just want to use that service…

In the meeting, there’s an introduction on how to use the service, many people did not prepare their machines before the session and they get stuck by the overcomplicated client installation instructions, which involve installing pre-compiled binaries and config files inside a .tar.bz (ABI issues galore!). Next, you are given a password via SMS, “abc123″, which you cannot change (nor are encouraged to). You should explicitly ask the admins to change it for you.

Dirty secret: if they are not forced to change it, nobody ever does.

After editing some cloud templates text files, your first instance is up and running. Time to clone your CloudBioLinux copy and get some bioinformatics software installed in it… Unfortunately, it does not take very long to discover that the base distribution is more than 3 releases old. The user emails the imaginary beta cloud support and says:

- Hi cloud-support! Is it possible to have the newest Ubuntu release as an image?
- No it’s not at the moment.
- Emm, ok, I tried to apt-get dist-upgrade but it just runs out of space, can I get more space in the VM to do that upgrade myself then? One cannot do much with 4GB of disk these days, you know.
- No, it is not possible, you can use the 1TB NFS-mounted scratch space instead.
- Why isn’t it possible? Anyway, I see no straightforward way to use that scratch as an extension of the OS, while the VM is running, and I cannot access the filesystem offline and move, say, /usr away without doing some hackish stuff involving squashfs, ramdisks, etc… this is actually giving me more headaches than is worth.
- I’m sorry, we cannot bundle another distribution for you.

So what can we learn from that experience? What can a bad cloud do to become a better cloud?

1. Distributing a readily installable and tested client CLI package for the most popular platforms instead of a precompiled .tar.gz would have cut down that 1 hour long meeting to nil. Documentation should never be a substitute nor shortcut for a tested, directly installable package.
2. Distributing passwords, even via SMS, should adhere to basic good password policies at all times, even in beta services. Go double factor authentication if you fancy it.
3. All services should be auto-provisioned. Asking sysadmins to perform routine operations like changing passwords should be off the table.
4. Dimensioning a cloud (disk, memory, network interfaces) is not an easy task if the users have wildly different needs, but at least, it should be possible to easily increase VM image space, within reasonable limits. Other metrics such as RAM, network interfaces, DNS records, mountpoints should be directly accessible to the user, auto-provisioned.
5. Creating new cloud images from vanilla OS’s automatically should be in place somehow and before launching the beta.

One year passes, some more console typing and moving to new hardware resources should get the service a new face, ready for a second try.

The same issues arise, instead of automating the deployment of the whole cloud to other machines, it just has been moved to the new hardware. There is no evidence of automation being done since last year.

A better cloud

Automated testing is about software, since clouds are software, why not automate bits and pieces of the deployment until it becomes fully automatic? It is easier said than done, it takes a great deal of patience to go and:

1. Build and test a cloud component.
2. Automate its deployment, testing it elsewhere.
3. Take the whole cloud stack down, recreating it again from scratch.
4. Automate basic user-side (stress)-testing: create instance, record a DNS change, attach new volumes, destroy instance, etc…

Automation and testing are hard, it takes time to get them right and not overfit your immediate environment. But look, those guys over there seem to have gotten it right:

- So you only need my public SSH key? That’s all? No meetings nor passport, fingerprints, blood samples or photos?
- That’s exactly right, just login as root, break as much as you want in your own cloud, we can wipe your whole stuff out in less than 20 minutes. We’ll of course be gathering metrics from outside, just in case we detect something bad coming out your instance(s). We don’t want to get in your way.
- Nice! What about having the latest Ubuntu release…
- We just provisioned it as we speak (true story).
- I’m launching some hadoop jobs right now. It took me a few minutes to provision the nodes. thank you guys, you’re awesome!

Look around, good (and some free beer) clouds!

Now, while our inner clouds get better, how can I get my idea without spending much money on it. What should we tell our users? They want some cloud action and they want it now. While OpenStacks get stacked properly inside your walls, you can go and explore the outside world a bit, we might learn a lot from them!

Those impatient early birds can try locally with something like Cloudify… Ooops, Chef recipes do not work out of the box if you have an Apple user.

Giving a try to @cloudifysource… #fail when installing #Chef in os.getVendor()==Apple… jtimberman.housepub.org/blog/2012/07/2…

— Roman Valls (@braincode) March 22, 2013

There are some hiccups that could be circumvented with Virtualbox though:

@braincode check this project from @fastconnect ow.ly/1TRJsg. We plan to have this as a native capability in Cloudify as well

— Uri Cohen (@uri1803) March 22, 2013

But wait, there are other alternatives that don’t depend on your workstation/laptop whacky configurations, they might not be as powerful (HPCCloud users, move along), but enough for starters. Heroku is a classic, with an excellent CLI tool and mandatory version control that can get you started very quickly in the cloud PaaS arena:

Then, another interesting platform is dotcloud, with its recently published cloud stack/backend, following the same steps as heroku:

Until a few days ago, free for best effort instances were offered, which meant “put your application here, we will run it if we have enough resources left”. Now you can build your own dotcloud in your organization, if PaaS is what you need.

Conclusion

Being a IaaS cloud provider can be a bumpy experience, it is not trivial. In this post I wanted to outline three different worldviews: the hurried cloud in-house service provider, the more automated in-house provider and the industry-quality for-pay provider. The goal is clear yet a tricky one to implement in reality. We want to reach a point where:

1. No human intervention and support is required to instantiate basic and small clouds. Obsessive automation culture is paramount to reach that stage.
2. Enable user auto-provisioning as a result of the first point.
3. Commonly used services wrapped through public deployment recipes (cloudify or chef are examples). Encourage re-use and improvement of those recipes.
4. Enforce good practices in the developer/user side such as version control as the only way to deploy applications.
5. Enforce security from outside the instances, don’t get in the way of the users unless bad things are happening or about to happen. Make sure you have implemented some non-intrusive security measures before announcing the cloud service.
6. Provide a consistent API/CLI toolbox/toolbelt to control various aspects of the instances, like the industry-quality providers have.

Sometimes education can be a daunting process. It is quite obvious from the student side, we all have gone through exercises, corrections, learning what we did wrong on some of them, fixing and learning from those errors, rinse and repeat. That’s how it generally works.

On the teacher’s side, correcting assignments is easy and unbiased unless the number of students is considerably large. At
one of the sessions of our now official KTH course “DD3436 Scientific Programming in Python for Computational Biology” I was given the task to hold a session on software testing and continuous integration in Python… for around 50 students.

So I wanted to find a simple way to teach Python without going through the old beaten track of the boring fibonnacci function and endless lists of exercises. Something like little incremental goals that kept students hooked until they finished the training for the session.

Then I discovered the python koans by Greg Malcolm. By themselves, they are a very good way to learn Python basics and unit testing.

But what would happen if those pykoans respected standard UNIX exit codes and had a basic Travis-CI integration? Indeed, that brings us testing and continuous integration for free!

And then, as a teacher, what if there were easy means to monitor students progress and correct assignments via some API’s and JSON?

The end result is that teaching basic Python along with good programming practices such as TDD and CI is much easier nowadays. Teachers get to help students better without correcting tons of assignments manually and students get green TDD lights as they go.

In my experience on both sides of the iron, as a sysadmin and developer, none of them work as one would like to.

But first, let’s explore what several HPC centers have adopted as a solution and why… and most importantly, how to fix it eventually.

The good

Widely used in different research facilities, the module system allows users to choose different versions of several software. The approach is simple, just type “module load program/1.0″ and off you go.

On the sysadmin side, it’s the same old familiar spell “tar xvfz && make && make install”, and a “vim program” to define the module script that will set PATH, LD_LIBRARY or other variables and whatnot.

Consequently, the time required to wrap a software is minimal, conferring sysadmins with speedy quick hack superpowers. After all, velocity in research does matter, and getting things done to let research continue its way is mandatory.

Moreover, user-coded modules can be shared easily within the same cluster by simply tweaking MODULEPATH variable. What’s the catch ? Technical debt and most importantly, lack of automation.

The bad

Software packaging is a time consuming task that shouldn’t be kept inside institutional cluster firewalls, but openly published. Indeed, a single program could have been re-packaged a number of times on each academic cluster for each university department that has HPC resources. When new versions come up for each package the sysadmin has to take care of bumping it by creating directories and additional recipes. How does one justify this time investment ? It just doesn’t scale. Skip to “solutions?” section for some relief.

From a technical perspective, using package systems that are not shipped with the operating system introduces an extra layer of complexity. More often than not, updates on the base distribution will break compiled programs that rely on old libraries. Stacking package managers should be considered harmful.

Ruby, python and perl have their own mature way to install packages for most UNIXes, stacking package managers by rpm-packaging python or ruby modules, has several bad consequences. Granted, there are some concerns on uniformity, updates and security, but those again can be solved by the individual package managers.

But getting back to the module system, how well does it play with cloud computing ?

It doesn’t, thankfully !

One would have to install all the modules, and re-package the software for the virtual instances. In contrast, existing package systems, be it rpm, deb, pip, gem or lein solved that by themselves. On top of that, the module system will tweak crucial system variables such as $PATH or $LD_LIBRARY_PATH with bad side effects for python virtual environments or any other user-defined PATHs.

Lastly, from a human resources perspective, writing modules does not add value or expertise to your IT toolbox. On the other hand, search engines have something to say when looking up search terms such as job experience packaging software rpm deb. Actually, being involved in open source communities via packaging can give you some very valuable insights on how open source projects work.

The ugly

With aging and relatively unmantained software, some bugs arise. Under some circumstances here’s what occurs:

$ modulecmd bash purge
*** glibc detected *** modulecmd: free(): invalid next size (fast): 0x0000000001b88050 ***
======= Backtrace: =========
(...)

$ export MODULEPATH=AAAAAAAAAAAAAAAAAAAAAAAAA:AAAAAAAAAAAAAAAAAAAAA:/bin/bash && modulecmd bash purge
    *** glibc detected *** modulecmd: corrupted double-linked list: 0x00000000009c4600 ***

I would like to light a candle for those who dare running modulecmd with suid bit. I can only think of one sysadmin that could do that while being totally self-confident.

Solutions?

Here’s some brainstorming that might help in the long run:

1. Instead of complicating infrastructure, just state the software versions you are running in your publications. In python, pip freeze helps. Want an older version ? DIY.
2. Use CDE, rbenv and virtualenv before and after publishing if concerned about platform updates during your research.
3. Use virtual machine images to reproduce experiments.
4. If you really need the module system, at least publish the modules somewhere for people to reuse them.
5. If you are a sysadmin, get started with FPM as a first approach with the world of package management for your distribution.
6. Try to get those packages accepted upstream (best!) and/or create your own rpm/deb repo.
7. Learn puppet and/or chef.

Please report any issues via comments !

1. Firsly, follow my earlier post on how to setup your own python virtual environment on UPPMAX.
2. Once you have a prompt similar to: (devel) hostname ~$, you can continue, else, jump to 1.
3. pip install drmaa Mercurial PyYAML
4. Add the following env variables to your .bashrc:

   export DRMAA_LIBRARY_PATH=/bubo/sw/apps/build/slurm-drmaa/lib/libdrmaa.so
   export DRMAA_PATH=$DRMAA_LIBRARY_PATH
   

5. Create a file ~/.slurm_drmaa.conf with the contents:

   job_categories: {
         default: "-A <your project_account> -p devel"
   }
   

6. hg clone http://bitbucket.org/brainstorm/galaxy-central
7. Edit universe_wsgi.ini from the provided sample so that it contains:

   admin_users = <your_admin_user>@example.com
   enable_api = True
   start_job_runners = drmaa
   default_cluster_job_runner = drmaa://-A <your project_account> -p devel
   

8. On your local machine: ssh -f <your_user>@<uppmax> -L 8080:localhost:8080 -N
9. On your local machine: Fire up your browser and connect to http://localhost:8080

As a betatester you may expect some issues when running galaxy in that way. Firstly, keep in mind that it’ll not perform as fast as a production-quality setup, it’s just a developer instance. Furthermore the node you’re in might have time limit restrictions, meaning that your instance will be killed in 30 minutes if you don’t reserve a slot beforehand as Martin recommended on the section “Run galaxy on a node”.