Dealing with software package management can be a daunting task, even for experienced sysadmins. From the long forgotten graft, going through the modern and insanely tweakable portage to the (allegedly) multiplatform pkgsrc or the very promising xbps, several have tried to build an easy to use, community-driven, simple, with good dependency-handling, optimal, reliable, generic and portable packaging system.

In my experience on both sides of the iron, as a sysadmin and developer, none of them work as one would like to.

But first, let’s explore what several HPC centers have adopted as a solution and why… and most importantly, how to fix it eventually.

Continue reading →

When one is developing a daemonized service, it’s rather usual to encounter minor errors that require no further attention than just restarting the daemon. That could be like not being able to connect to a remote machine for some time:


Traceback (most recent call last):
(...)
File "python2.6/urllib2.py", line 1170, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "python2.6/urllib2.py", line 1145, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 111] Connection refused>


Granted, we want to fix this on the code so that the daemon does not die, but meanwhile it’s good to have a safety net that we can rely on. That’s were supervisord comes in handy. Let’s see how it’s done.

Continue reading →

I’ve just pushed a set of trivial modules system scripts that will hopefully ease your deployment of Cloudera Distribution for Hadoop 3 Beta 4 on your university cluster… partly, at least. This sad “partly” made me think about the current state of things on IT and HPC.

Over time I’ve learnt that there are several unexpected issues when deploying hadoop on custom clusters that you don’t own. Those are mainly related to software management policies, non-root access (being auto-deployment unfriendly), quotas and queueing or “batch” systems.

Ignoring most of these “fixable” issues, it becomes apparent that the most juicy problem for a sysadmin trying to get the most of hadoop-related tools is the batch system. Be it SGE, SLURM or other non-DRMAA compliant exotic batch system implementations, you’ll have to deal with annoying integration quirks at some point, granted.

Making it all work can be challenging to say the least… but the question is: does it have to be that hard ?

Continue reading →

I would have preferred that my Android 1.6 device supported OpenVPN out of the box. Unfortunately, this is only available for rooted devices and a bit of suffering. Instead, I went for configuring IPsec inside L2TP VPN server. All of it stuffed into an old and low-end Soekris net4511 board running Voyage Linux.

First, I will just redirect you to the well-documented, lengthy but primary resource:

Using a Linux L2TP/IPsec VPN server

On the client side, this post is quite complete:

Adding VPN connections to Android 1.6 (Donut)

If you’re feeling impatient and brave, perhaps you’ll succeed with the configuration files that follow (they worked for me)… since those are highly dependant on your network setup, YMMV, a lot.

Before jumping right into the meat and to avoid confusion, let’s see what is the game all these evil daemons are going to play:

1. A client (my android phone), connects to the server on port 4500.
2. IPsec server (OpenSWAN) responds and asks for the PSK.
3. If the previous “gatekeeper” is ok with you, control is handed over L2TP, the other “tunnel keeper” who will ask for another password.
4. If L2TP is satisfied with your answer, PPPD, the ancient UNIX beast will be waken up and ask for… your user and password !
5. Congrats ! You’re survived the gates, now you’re on your home network from your smartphone, ain’t it cool ?

Continue reading →

Just had a conversation with one good friend, rambling on the possibility of having a server-side crypted Maildir INBOX.

It seems that dovecot has a nice server-side mail compression plugin… how come there are no implementations of a session-based cyphering of mail storage based on this same principle ?

The use case is quite simple to imagine:

1. Mails keep coming to the MDA, on plain text, as usual.
2. User logs into the IMAP server with her credentials.
3. A keypair stored on user’s maildir, for instance, is unlocked using the login password.
4. New mails are cyphered and all subsequent read/write operations are performed through this (de)cyphering mechanism.
5. User logs out and the key is forgotten, shrinking the window of opportunity on possible sneakers or forensic forces.

EDIT: Of course, keeping just the public key on the server is way smarter in this case:

1. The user’s GnuPG public key is stored on her maildir.
2. All incoming emails are cyphered as they arrive with the previous public key.
3. The user logs in and sees all her mailbox cyphered, ready to be decyphered with his private key residing on her mail client.
4. Forensic analysis/spying on emails gets just a little bit harder .

He argued that just a quick UNIX pipework (using qmail) should be sufficient, but I rather preferred to keep the MTA out of the equation. The reason is that the MTA just “sinks” mail to the mailbox, while the MDA usually has both read and write access to emails, so to me it makes more sense to keep this “plugin” on the MDA side…

Is the idea clear at this point ? Is this already invented and passed under my radar ? Anyone has suggestions on top of that ?