Publishing my server’s `/etc` repository

, 2018-01-07.

Motivation

Fairly soon after I started playing with my own private server (I seem to have purchased it in November 2014 and the first Git commit dates to February 2015), I started tracking its configuration in a Git repository. I didn’t add everything in /etc (that is, the .gitignore is just *), but I felt that I wanted to be able to have a history of changes, with the ability to roll them back later if necessary, or at least understand why a setting was in place. I started tracking just the configuration of the email services, but later added other things as well. When I moved servers, I found the repository really useful as well.

However, eventually I found myself posting more and more parts of the repository publicly, usually via GitHub gists: I had managed to put together this nice bit of configuration, and I wanted others to benefit from it as well! Sometimes I posted just the configuration files, but sometimes I even posted the full Git commit (i. e., in git format-patch format) because I wanted to include the commit message as well. At this point, it seemed only logical to publish the repository. The rest of this post describes how I went about doing that.

Sensitive information

The first step was, of course, to deal with any sensitive information that might lurk in the repository. I had never added any really sensitive information to it (things like SSH server keys, X.509 certificate keys, the shadow file…), but I couldn’t just assume that I could publish everything as-is: for a while, I had used the repository with the full expectation that it would not be published, as evidenced in the commit message of commit 6eb85275be:

This repository will stay private anyways, and I don’t think there’s actually any very sensitive information in the torrc. (The keys etc. are in the data directory under /var.)

So, when I had some spare time during 34C3, I cloned the /etc repository to my laptop and went through the entire git log --patch, looking for anything that I might not want to make public.

On default configuration and empty `/etc`

Going through the log took far longer than it should have because Postfix, Dovecot and Apache all place enormous amounts of default configuration in /etc which I had added to the repository. This just reminded me of how much I prefer the systemd approach to configuration, so I wanted to elaborate a bit on that. (Though I’m sure systemd didn’t invent it, that is where I was introduced to it, and I’m not aware of many other programs which are configured this way.)

With systemd-style configuration, /etc does not contain any default configuration files: those live in /usr/lib, and if they’re good enough for you, then you don’t need anything in /etc. Otherwise you can either amend or entirely overwrite configuration files in /etc – for instance, the file /etc/systemd/system/foo.service completely replaces the file /usr/lib/systemd/system/foo.service, whereas /etc/systemd/system/foo.service.d/bar.conf merely augments it with extra directives.

If Apache followed this convention, for instance, it might be configured out of the box to serve /srv/http/ (or, closer to the Debian convention, /var/www/) over HTTP on localhost. For a basic version of my website, I would only need to configure the server name, the interface to bind to, and the location of my certificate. (For a more complete version, I would also need to enable PHP, enable indexes on a few directories, specify some redirects, etc.)

There is no reason that my /etc needs to be spammed with crap like

which MIME types MSIE 6 can handle in DEFLATEd form,
that .es means Spanish and not ECMAScript,
which bytes Just System Word Processor Ichitaro files start with,
where the .so file for module which provides Basic Authentication via LDAP is located,
that RCS files should not be listed in a directory index,
that RealPlayer 4.0 does not support HTTP/1.1,

or any of a thousand other things which are now all in my /etc repository. (Just to be clear: I’ve picked a lot of obsure or legacy stuff here, but I don’t mind that it’s obscure or legacy per se – I just don’t want it in /etc, which is supposed to be server-specific.)

Of course, the big problem is not that these files bloat the repository, or that it took longer to review the history because of it. But I also have to find my way through all of this cruft any time I want to change the config, or even just debug it to understand where something might be going wrong. And where it gets really ugly is during upgrades, especially full distribution upgrades – when I need to merge all of the changes that upstream made to the files (including indentation changes, hooray!) with my own changes. If the new version of the package supports a new option that should be on by default, then just make that the built-in default, or add the configuration in /usr/lib where the package manager’s stuff belongs, but don’t create a merge conflict in my configuration directory just because we both appended to the end of the config file!

This aside is already getting too long, and I’m not sure where I was going with this anyways, but – if you’re working on a program, please take this to heart. Load your configuration from /etc, falling back to /usr/lib, and place all the default configuration in the latter directory. Support drop-in files, including in /usr/lib so that other packages can also augment the configuration! (Also, for ease of debugging, provide ways for the system administrator to see the final config that your program understands. systemd does this with systemctl show and systemctl cat, Apache with apachectl -S and apachectl -t.)

In the end, I didn’t find any smoking guns, but I did decide that I wanted to redact the postfix/virtual file – I don’t want a full list of all valid email addresses on my domain to be public – and I also needed to rephrase one commit message which mentioned the not-yet-public name of a certain project. (I also took the opportunity to fix innocent typos in two other commit messages.)

To redact the postfix/virtual file while still keeping it under version control, I decided to use git-crypt, which transparently encrypts files in a Git repository using Git’s filter mechanism (see gitattributes(5)).

Rewriting history

As I was going to rewrite history, I also wanted to update all commit hashes I had used in commit messages, so I found all commits whose messages might have mentioned another commit with git log --grep '[0-9a-f]\{7,\}' and added those that weren’t false positives to the text file where I had already collected the commits I needed to redact for other reasons.

Then, in a temporary clone of the repository (/etc needed to remain functional, of course!), I started the big rebase: git rebase --interactive --root. I marked the root commit (the one that added postfix/virtual) for editing, and all the others where the commit message needed rephrasing to reword.

When the first commit was reached, I added the following postfix/.gitattributes file to the commit:

virtual filter=git-crypt diff=git-crypt
virtual-mailbox-users filter=git-crypt diff=git-crypt

I amended the commit and then checked with git cat-file -p @:postfix/virtual | less that the content recorded in Git was really encrypted. Then I continued the rebase.

Unfortunately, one consequence of the encryption is that patches cannot be applied, so the rebase was interrupted on every commit that touched the encrypted files. Fortunately, this wasn’t too hard to resolve: I could just restore the unencrypted file from the original commit (when Git said that it could not apply 641464f... Postfix: add lots of virtual addresses, I ran git cat-file -p 641464f:postfix/virtual >| postfix/virtual), add it to the index, and it would be encrypted through the regular filter process once git rebase --continue amended the commit.

For the other commits, I just found the new commit hash in the git log and replaced the hash in the commit message.

At the end, I added a README.md (hidden in the .github directory), uploaded it to GitHub, and done! And here it is.