Foreman SSL explained

By default the foreman-installer installs using the [Puppet CA]. While options are provided to replace these, it's not obvious. In this blog we'll see how Foreman uses certificates and how to to replace them.

While the Puppet CA is very convenient for a server admin using Puppet, it's inconvenient for clients connecting to the UI because often it's not in the certificate store. For those not using Puppet, the CA service is quite heavy. Others may need to use different certificates for compliance reasons. Then there are those who do use Puppet CA but want to understand the application. Whatever the reason, we'll go step by step in building a deployment.

Generating certificates

Foreman can use certificates signed by any Certificate Authority, even self signed. For this example ownca is used, a shell script around wrapping openssl. The installation is done using wget and extracted to /etc/ownca.

yum -y install wget openssl
mkdir /etc/ownca
cd /etc/ownca
chmod +x ownca

This set of scripts first needs to generate a Certificate Authority (CA).

# ./ownca ca
Generating a RSA private key
writing new private key to 'private/cakey.crt'

A CA is allowed to sign other certificates. In this case it's a bunch of files, but sometimes it describes a company that issues certificates. Either way, there is always a public certificate; in this case /etc/ownca/cacert.crt.

The Foreman instance running on $HOSTNAME ( here) needs a certificate:

# ./ownca cert $HOSTNAME
Generating a RSA private key
writing new private key to './'
Using configuration from ./openssl.cnf
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
commonName            :PRINTABLE:''
Certificate is to be certified until May 22 11:31:13 2021 GMT (365 days)

Write out database with 1 new entries
Data Base Updated

These steps created the following files that are needed later:

  • /etc/ownca/cacert.crt is our Certificate Authority
  • /etc/ownca/ is our Certificate
  • /etc/ownca/ is our Private Key

Inspecting certificates

Certificates are not just a blob of bytes, there's a lot of information in them. Extracting this information is not always obvious so this chapter explains how to read the information, starting with the CA certificate.

openssl has various subcommands which can be listed using openssl help. X.509 is the standard that defines the format of public key certificates. This is also the openssl subcomand to inspect them. The next command reads cacert.crt and shows the text representation.

# openssl x509 -in cacert.crt -noout -text
        Version: 3 (0x2)
        Serial Number:
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = OwnCA
            Not Before: May 22 11:30:48 2020 GMT
            Not After : May 20 11:30:48 2030 GMT
        Subject: CN = OwnCA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
            X509v3 Subject Key Identifier:
            X509v3 Authority Key Identifier:

    Signature Algorithm: sha256WithRSAEncryption

This output is cryptic, but there's a few things to look for. First of all, there's the following block

        Issuer: CN = OwnCA
            Not Before: May 22 11:30:48 2020 GMT
            Not After : May 20 11:30:48 2030 GMT
        Subject: CN = OwnCA

This is the most important information. It tells us who issued the certificate, when it's valid and the subject. Certificates are identified by their subject. The exact number of fields there can vary, depending on the CA. Note that Issuer and Subject match, indicating that the certificate signed itself (self-signed certificate). A CA certificate that signed itself is commonly known as a root certificate because computer scientists see this as a tree structure, where things start at the root.

In the extensions we also see CA:TRUE which means it's allowed to sign certificates:

        X509v3 extensions:
            X509v3 Basic Constraints:

When a certificate has CA:TRUE but is signed by another certificate, it's typically referred to as an Intermediate CA.

The host certificate can also be inspected:

# openssl x509 -in $HOSTNAME/$HOSTNAME.crt -noout -text
        Version: 3 (0x2)
        Serial Number:
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = OwnCA
            Not Before: May 22 11:31:13 2020 GMT
            Not After : May 22 11:31:13 2021 GMT
        Subject: CN =
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
            X509v3 Subject Key Identifier:
            X509v3 Key Usage:
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication, Code Signing, E-mail Protection
            X509v3 Subject Alternative Name:
    Signature Algorithm: sha256WithRSAEncryption

This is largely the same as the CA certificate, but with some differences. For starters, the certificate is not allowed to sign other certificates (CA:FALSE).

More importantly the issuer and subject differ. The issuer is our CA and the subject is our hostname. Foreman still uses Common Name (CN) a lot, in particular around authentication. You might spot a potential problem with this. Common Name can only be a single name but various services have multiple. That's why modern tooling has started to use the Subject Alternative Name extension:

        X509v3 extensions:
            X509v3 Subject Alternative Name:

This is the same name as our Common Name. Ensure that the Common Name is there or things will break. Additional names can be added, but is beyond the scope of this explanation.

Installing Foreman

This guide is using CentOS 8 and the Foreman 3.1 but this is applicable to many versions and all supported operating systems.

yum -y install
yum -y install foreman-installer

This doesn't install EPEL, but you may need it for some plugins or older releases.

First the permissions will need to be fixed. For this, a dedicated group is created:

groupadd foreman-certs
chgrp foreman-certs $OWNCA/$HOSTNAME/$HOSTNAME.key

Now that we've prepared the certificates, the installer will do the hard work. Just running foreman-installer without arguments will install various components, but that's not what we want right now. It's customized to only install Foreman itself without a Foreman Proxy.

foreman-installer \
    --no-enable-foreman-cli \
    --no-enable-foreman-proxy \
    --no-enable-puppet \
    --foreman-user-groups foreman-certs \
    --foreman-server-ssl-ca $OWNCA/cacert.crt \
    --foreman-server-ssl-chain $OWNCA/cacert.crt \
    --foreman-server-ssl-cert $OWNCA/$HOSTNAME/$HOSTNAME.crt \
    --foreman-server-ssl-key $OWNCA/$HOSTNAME/$HOSTNAME.key \
    --foreman-server-ssl-crl "" \
    --foreman-client-ssl-ca $OWNCA/cacert.crt \
    --foreman-client-ssl-cert $OWNCA/$HOSTNAME/$HOSTNAME.crt \
    --foreman-client-ssl-key $OWNCA/$HOSTNAME/$HOSTNAME.key

Now that looks like a lot of duplication so we'll go over all options. We can recognize two groups: server and client.

The server is what will be used by Apache and the is most visible. Since Apache reads these files as root, file permissions are generally not that important. SELinux policies can limit it so be aware of that. The Certificate Revocation List (CRL) is unset. CA vs chain is explained later with examples.

However, Foreman also runs a bundled Websockify that will spawn for VNC consoles to VMs. It's serving its own certificates and these options control this. Websockify runs as the Foreman user so it needs to read these files, which is why we created the group. The server and websocket should use the same certificates.

There are also client certificates that are used to connect to the Foreman Proxy (also known as Smart Proxy). In this case, the certificate and key are used as client certificates rather than server. This means they're used for identification. Later we'll see how that's verified.

Our example uses the same certificates for both server and client, but this isn't required. Foreman Proxies can use a different CA. This can be useful when using a public commercial Certificate Authority for the Foreman UI but a custom CA (like Puppet CA) for internal communication. In that case --foreman-server-ssl-ca must point to the public facing CA and --foreman-server-ssl-chain the internal CA. The --foreman-client-ssl-* options must belong to the internal CA.

After the installer has completed, we can inspect what's actually being served.

# openssl s_client -connect $HOSTNAME:443 -CAfile /etc/ownca/cacert.crt
depth=1 CN = OwnCA
verify return:1
depth=0 CN =
verify return:1
Certificate chain
 0 s:CN =
   i:CN = OwnCA
 1 s:CN = OwnCA
   i:CN = OwnCA
Server certificate
subject=CN =

issuer=CN = OwnCA

Acceptable client certificate CA names
CN = OwnCA

This is slightly truncated for readability.

The first thing to notice, is the Certificate chain. At the highest level we see our server certificate with its subject (s:) and issuer (i:). Our server also serves the CA chain below that. In this case it's only our root CA as specified in --foreman-server-ssl-chain. If your chain includes intermediate CAs, you will see those as well.

Then the Server certificate is shown, which matches --foreman-server-ssl-cert. Again, the subject and issuer are shown.

Lastly there's Acceptable client certificate CA names that matches --foreman-server-ssl-ca. Foreman uses client certificates in various places to authenticate. Any such certificate must be signed by this CA.

Visualizing the deployment

This can be summarized in a diagram.

Foreman Architecture

Services listening externally have a bold border. Red means it's unencrypted and blue is encrypted using our new CA. Concretely:

  • Apache listens on *:80 (unencrypted)
  • Apache listens on *:443 (encrypted)
  • Websockify allocates a free port in the 5910 - 5930 (encrypted) range when a VNC console is started

Starting Foreman 2.1, Apache works as a reverse proxy to Foreman. The Foreman service runs unencrypted on and only Apache is supposed to connect to it. This is implemented using systemd socket activation. The systemd service foreman.socket listens on a TCP socket and passes it to foreman.service as a file descriptor. No HTTP requests are dropped this way, even if foreman.service is restarted. In the future Apache to Foreman connections may be changed to use a unix socket for better security.

The VNC console part also can use some improvements. The RFC Consolidating The Console was started, but so far nobody has worked on implementation.

Adding a Foreman Proxy

By default, foreman-installer installs a Foreman Proxy, but in this blog it's split off to a separate host. This is done to better illustrate how things talk to eachother. Foreman can have many Foreman Proxies so this is a common scenario even when a proxy also exists on the Foreman host.

First a certificate is needed for our host We run this on in the ownca directory.

[root@foreman ownca]# ./ownca cert
Generating a RSA private key
writing new private key to './'
Using configuration from ./openssl.cnf
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
commonName            :PRINTABLE:''
Certificate is to be certified until May 26 13:07:09 2021 GMT (365 days)

Write out database with 1 new entries
Data Base Updated

Of course this can also be inspected but for brevity this is left as a user excercise. When doing so, check that the issuer is still the same CA while the subject now has the new hostname.

The installer can register a new Foreman Proxy using the REST API. Authentication is done using OAuth. By default the installer generates a random key and secret when installing Foreman but they can also be explicitly specified. To retrieve the values, you can go to the Foreman settings page. In this case we'll use the command line on # foreman-rake -- config -k oauth_consumer_key
supersecret # foreman-rake -- config -k oauth_consumer_secret

Copy the files cacert.crt, and to the proxy host. Make sure the user foreman-proxy can read these. It's also important that the key is not world readable.

Also follow the same installation steps to get foreman-installer and run the installer on # foreman-installer \
    --no-enable-foreman \
    --no-enable-foreman-cli \
    --no-enable-puppet \
    --foreman-proxy-manage-puppet-group false \
    --foreman-proxy-puppet false \
    --foreman-proxy-puppetca false \
    --foreman-proxy-tftp false \
    --foreman-proxy-foreman-base-url \
    --foreman-proxy-trusted-hosts $HOSTNAME \
    --foreman-proxy-trusted-hosts \
    --foreman-proxy-oauth-consumer-key supersecret \
    --foreman-proxy-oauth-consumer-secret alsosecret \
    --foreman-proxy-ssl-ca /path/to/cacert.crt \
    --foreman-proxy-ssl-cert /path/to/$HOSTNAME.crt \
    --foreman-proxy-ssl-key /path/to/$HOSTNAME.key

Here we specify the server certs, but don't explicitly configure the client certificates. That's because the same certificates as the server are used automatically. When using a different CA for Foreman, --foreman-proxy-foreman-ssl-ca must be specified as well.

There are some options to disable Puppet integration and even tftp to get a minimal installation. This only leaves the logs feature, chosen because it requires no additional dependencies and Foreman refuses to register proxies without any features.

It's also possible to specify an additional group for the foreman-proxy user with --foreman-proxy-groups mygroup. One example is having all files in /path/to/mycerts and setting permissions using chown root:mygroup /path/to/mycerts && chmod 0750 /path/to/mycerts. It is up to you to ensure mygroup exists prior to running the installer.