Understanding systemd credentials

Date Tags systemd

This documents my journey to understand systemd CREDENTIALS. A very short summary is that the mechanism provides credentials in a $CREDENTIALS_DIRECTORY. $CREDENTIALS_DIRECTORY itself is exposed as an environment variable.

Why files?

Those who have been around a while my remember that 12factor recommended to keep config in environment variables. More modern tooling has realized this is not a good practice, because environment variables can easily leak. That is why container solutions and systemd use a special directory with files.

Small demo

Let's assume a very small test application that simply dumps the credential cred:

#!/bin/bash

dump() {
        cat "${CREDENTIALS_DIRECTORY}/cred"
}

dump

After making it executable, we can run this with the LoadCredential property:

chmod +x test
echo foo > secret
systemd-run --user --collect --property LoadCredential="cred:$PWD/secret" "$PWD/test"

The output can be found in the journal:

$ journalctl --user -t test
May 28 13:40:06 gijs test[988421]: foo

We have just exposed a file on the filesystem to the process. Note that systemd for system services reads the credential as root, meaning we don't need to think about file permissions. There are also options to store the credential itself encrypted, but that's not what I'm after now so that's left as an excercise for the reader.

Rotating secrets

Improving certificate handling was my primary motivation for this. You probably know those have an end date and thus need to be rotated. Big question is then whether the credential gets reread on every read or if its exposed once.

Let's enhance our test application:

#!/bin/bash

dump() {
        cat "${CREDENTIALS_DIRECTORY}/cred"
}

trap dump INT EXIT

dump
sleep 100

It now dumps the credential on start up and when interrupted or on exit.

Let's call it, rotate the secret and kill it:

echo foo > secret
systemd-run --user --collect -p LoadCredential="cred:$PWD/secret" "$PWD/test"
sleep 1 # systemd starts it asynchronously
echo bar > secret
pkill test

Now we can see two new entries in the journal:

$ journalctl --user -t test
May 28 13:40:06 gijs test[988421]: foo
May 28 13:42:51 gijs test[988704]: foo
May 28 13:42:52 gijs test[988717]: foo

Clearly the value is not reread by systemd and only on start up.

Dynamic loading

Depending on how your application is set up, it may need to be more dynamic. Luckily systemd can also read from an AF_UNIX socket. Using systemd credentials to pass secrets from Hashicorp Vault to systemd services is a blog post that greatly helped me to understand this, though I think the code is more complex than it needs to be.

How do you implement this? You still use credential:path where credential is the name and path is some path that is a UNIX socket.

Like in the aforementioned blog, we'll use Python but simplify it a bit. First we'll implement a main:

import asyncio
import logging
import os
import socket
import sys
from pathlib import Path

from systemd.daemon import listen_fds

log = logging.getLogger()


async def main():
    socket_fds = listen_fds()
    if len(socket_fds) == 0:
        log.info("No sockets passed, exiting")
        sys.exit(0)
    elif len(socket_fds) > 1:
        log.warning("More than 1 socket passed, not supported")
        sys.exit(1)
    sock = socket.socket(fileno=socket_fds[0])

    server = await asyncio.start_unix_server(client_connected_cb, sock=sock)
    await server.serve_forever()


if __name__ == "__main__":
    try:
        asyncio.run(main())
    except KeyboardInterrupt:
        pass
    except Exception as e:
        log.exception(e)
        sys.exit(2)

This gives a basic UNIX socket server (using asyncio.start_unix_server) that's listening on whatever socket systemd gives to it, allowing us to later use systemd's socket activation. None of this is special, but it's essential.

The interesting part is how we handle each client, which is implemented in client_connected_cb.

async def client_connected_cb(_reader, writer):
    _, _, service, credential = (
        writer.get_extra_info("peername").decode("utf-8").split("/")
    )
    log.info("Got connection from %s, credential %s", service, credential)
    try:
        path = await get_credential(service, credential)
        with path.open('rb') as f:
            loop = asyncio.get_event_loop()
            await loop.sendfile(writer.transport, f)
        await writer.drain()
    except FileNotFoundError:
        log.error('No credential %s found for %s', credential, service)
    finally:
        writer.close()
        await writer.wait_closed()

This is obviously based on the blog post, but with some clear differences. The first part determines who requested the credential using asyncio.BaseTransport.get_extra_info. Sadly, socket.getpeername isn't very helpful in understanding the resulting data. It is actually the address it binds to, which will typically be /run/$service/$credential. This is an area I still need to investigate further.

The second part is what I modified from the original post. Here we rely on a to-be-written API get_credential to return a pathlib.Path. That is opened and sent via sendfile. Finally everything is cleaned up.

A very simple implementation of get_credential is reading files from a directory:

BASE_DIRECTORY = Path(os.environ['STATE_DIRECTORY'])


async def get_credential(service, credential):
    service, _ = service.split('.')
    return BASE_DIRECTORY / service / credential

More complex examples can be thought of.

All of this is further implemented in my foreman-credentials repository.

Service example

Obviously none of this is of any use without a service using it. My initial target was to enhance Foreman's smart-proxy, which uses Ruby webrick.

Since that's much more complex, an example wouldn't speak for itself. Here is a minimal example:

#!/usr/bin/env ruby

require 'openssl'
require 'webrick'
require 'webrick/https'

def credential(cred)
  File.read(File.join(ENV['CREDENTIALS_DIRECTORY'], cred))
end

options = {
  DocumentRoot: nil,
  Port: 8443,
  SSLEnable: true,
  SSLCertificate: OpenSSL::X509::Certificate.new(credential('server-certificate')),
  SSLPrivateKey: OpenSSL::PKey.read(credential('server-key')),
}

server = WEBrick::HTTPServer.new(options)
server.mount_proc '/' do |req, res|
  res.body = 'Hello, world!'
end

trap 'INT' do
  server.shutdown
end

server.start

Then in a service file you can use (in combination with certbot:

[Service]
LoadCredential=server-certificate:/etc/letsencrypt/live/example.org/cert.pem
LoadCredential=server-key:/etc/letsencrypt/live/example.org/privkey.pem

Here static credentials are fine, because there's no reload implemented. Once that is implemented, then the dynamic server implementation becomes useful.