Some PEP‘s have revolved around the problem of software versioning and dependency tracking.

So, in addition to having some blueprints such as those proposed by the Semantic Versioning guidelines, one needs specifics on how to integrate those practices in our day to day work with version control systems.

Setuptools saves the day by introducing versioning via git tags. In a post by Douglas Creager a strategy to use setuptools with git tags is devised. The workflow for tagging a new version results in:

1. Tag your release via git tag if the changes are significant.
2. Run python setup.py install, to bump the version on the filesystem.
3. git push.

The following code makes it happen:

# Fetch version from git tags, and write to version.py.
# Also, when git is not available (PyPi package), use stored version.py.
version_py = os.path.join(os.path.dirname(__file__), 'version.py')

try:
    version_git = subprocess.check_output(["git", "describe"]).rstrip()
except:
    with open(version_py, 'r') as fh:
        version_git = open(version_py).read().strip().split('=')[-1].replace('"','')

version_msg = "# Do not edit this file, pipeline versioning is governed by git tags"
with open(version_py, 'w') as fh:
    fh.write(version_msg + os.linesep + "__version__=" + version_git)

setup(name="yourpythonpackage",
      version="{ver}".format(ver=version_git),

As an addition to the git tags workflow proposed by Douglas, the ‘__version__’ attribute will be stored in version.py file. This allows the versions to be tracked even when our git repository is not available (i.e, via PyPi package installation), or when such a version needs to be queried from inside your own package.

Thanks Guillermo and Brad for the feedback and suggestions on this strategy.

This post is intended to be shortened over time, eventually becoming an automated procedure… a wiki-post from dahlo’s magic until upstream patches settle down. All commands are issued on the cluster, unless otherwise stated.

Please report any issues via comments !

1. Firsly, follow my earlier post on how to setup your own python virtual environment on UPPMAX.
2. Once you have a prompt similar to: (devel) hostname ~$, you can continue, else, jump to 1.
3. pip install drmaa Mercurial PyYAML
4. Add the following env variables to your .bashrc:

   export DRMAA_LIBRARY_PATH=/bubo/sw/apps/build/slurm-drmaa/lib/libdrmaa.so
   export DRMAA_PATH=$DRMAA_LIBRARY_PATH
   

5. Create a file ~/.slurm_drmaa.conf with the contents:

   job_categories: {
         default: "-A <your project_account> -p devel"
   }
   

6. hg clone http://bitbucket.org/brainstorm/galaxy-central
7. Edit universe_wsgi.ini from the provided sample so that it contains:

   admin_users = <your_admin_user>@example.com
   enable_api = True
   start_job_runners = drmaa
   default_cluster_job_runner = drmaa://-A <your project_account> -p devel
   

8. On your local machine: ssh -f <your_user>@<uppmax> -L 8080:localhost:8080 -N
9. On your local machine: Fire up your browser and connect to http://localhost:8080

As a betatester you may expect some issues when running galaxy in that way. Firstly, keep in mind that it’ll not perform as fast as a production-quality setup, it’s just a developer instance. Furthermore the node you’re in might have time limit restrictions, meaning that your instance will be killed in 30 minutes if you don’t reserve a slot beforehand as Martin recommended on the section “Run galaxy on a node”.

UPDATE:

This documentation below has been superseeded by a much simpler, generalized and automated alternative: VirtualEnv-burrito.

Continue reading →