Foreman SSL explained
By default the foreman-installer installs using the [Puppet CA]. While options are provided to replace these, it's not obvious. In this blog we'll see how Foreman uses certificates and how to to replace them.
While the Puppet CA is very convenient for a server admin using Puppet, it's inconvenient for clients connecting to the UI because often it's not in the certificate store. For those not using Puppet, the CA service is quite heavy. Others may need to use different certificates for compliance reasons. Then there are those who do use Puppet CA but want to understand the application. Whatever the reason, we'll go step by step in building a deployment.
Generating certificates
Foreman can use certificates signed by any Certificate Authority, even self signed. For this example ownca is used, a shell script around wrapping openssl. The installation is done using wget and extracted to /etc/ownca.
yum -y install wget openssl
mkdir /etc/ownca
cd /etc/ownca
wget https://raw.githubusercontent.com/ekohl/ownca/master/ownca
chmod +x ownca
This set of scripts first needs to generate a Certificate Authority (CA).
# ./ownca ca
Generating a RSA private key
.........+++++
....................+++++
writing new private key to 'private/cakey.crt'
-----
A CA is allowed to sign other certificates. In this case it's a bunch of files, but sometimes it describes a company that issues certificates. Either way, there is always a public certificate; in this case /etc/ownca/cacert.crt.
The Foreman instance running on $HOSTNAME (foreman.example.com here) needs a certificate:
# ./ownca cert $HOSTNAME
Generating a RSA private key
..............................................................................................................................+++++
.....+++++
writing new private key to './foreman.example.com/foreman.example.com.key'
-----
Using configuration from ./openssl.cnf
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
commonName :PRINTABLE:'foreman.example.com'
Certificate is to be certified until May 22 11:31:13 2021 GMT (365 days)
Write out database with 1 new entries
Data Base Updated
These steps created the following files that are needed later:
/etc/ownca/cacert.crtis our Certificate Authority/etc/ownca/foreman.example.com/foreman.example.com.crtis our Certificate/etc/ownca/foreman.example.com/foreman.example.com.keyis our Private Key
Inspecting certificates
Certificates are not just a blob of bytes, there's a lot of information in them. Extracting this information is not always obvious so this chapter explains how to read the information, starting with the CA certificate.
openssl has various subcommands which can be listed using openssl help. X.509 is the standard that defines the format of public key certificates. This is also the openssl subcomand to inspect them. The next command reads cacert.crt and shows the text representation.
# openssl x509 -in cacert.crt -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
0e:30:4a:47:ba:c5:dc:3e:77:ee:43:9e:eb:fd:c0:1a:49:6a:44:83
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = OwnCA
Validity
Not Before: May 22 11:30:48 2020 GMT
Not After : May 20 11:30:48 2030 GMT
Subject: CN = OwnCA
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:bc:3b:48:39:8c:52:3b:c2:a5:db:1f:e5:e9:26:
e0:b1:80:72:01:e4:af:41:0c:27:50:a7:21:41:5f:
13:ba:f6:ac:63:3d:aa:c1:5b:c0:bb:e2:a5:df:27:
b8:e2:1a:80:e8:12:a8:b1:12:b9:59:7d:07:0f:22:
14:c9:ea:fa:6e:ce:60:0b:cf:a6:2c:86:97:82:09:
1c:4c:1f:f8:a0:27:18:a5:26:20:59:a1:ed:69:45:
76:3d:5d:8b:b4:8b:46:78:62:a7:08:c5:c1:49:0d:
f4:c2:b6:e3:cc:e3:41:f0:2e:79:e4:04:b2:ce:8e:
7f:2d:6f:ae:84:4a:13:ad:2e:d4:47:59:3b:2a:58:
bb:21:b7:f2:dc:74:b9:d5:05:e4:7d:83:87:fc:c2:
5a:b1:bb:66:cf:f4:14:ff:a3:c0:14:5e:fc:98:11:
37:ea:75:fe:15:8b:8f:46:7f:10:28:8a:8c:e3:05:
1b:f1:68:ed:82:8d:e1:10:26:2c:dc:a7:5c:67:7b:
da:2a:70:3e:ee:42:55:ad:a8:93:35:aa:f1:7b:7f:
10:49:61:df:35:83:40:e9:8a:d3:1e:6d:7d:3c:63:
15:8b:ed:5d:a9:49:ce:8d:2f:a9:2c:04:ee:ca:c0:
3d:79:05:3a:a3:2e:82:a3:15:6a:e6:b8:eb:05:cc:
a3:ef
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:TRUE
X509v3 Subject Key Identifier:
10:EA:F3:ED:37:90:58:CE:50:0A:A4:96:96:25:B2:80:70:EC:AD:34
X509v3 Authority Key Identifier:
keyid:10:EA:F3:ED:37:90:58:CE:50:0A:A4:96:96:25:B2:80:70:EC:AD:34
DirName:/CN=OwnCA
serial:0E:30:4A:47:BA:C5:DC:3E:77:EE:43:9E:EB:FD:C0:1A:49:6A:44:83
Signature Algorithm: sha256WithRSAEncryption
20:d8:94:4c:6e:c7:91:7f:80:7d:d4:f8:be:be:a4:6f:9f:3e:
39:2b:8d:ba:b8:2a:52:cc:fd:80:b3:a4:72:2e:c5:cc:5f:53:
43:03:29:0e:4d:b4:db:d0:12:fb:6c:0e:db:9e:0d:03:60:6c:
8f:34:bc:88:ba:ec:a4:50:a5:d4:fd:bd:70:75:ad:72:b1:af:
ba:f1:c6:cd:ea:dd:0c:a0:fa:d5:b0:9a:ad:67:61:1b:1a:f0:
8b:1b:fc:32:c2:29:e5:6a:e1:07:7b:0c:e2:52:2e:8b:91:1b:
8b:28:d5:dd:64:a5:9c:96:81:4c:0d:da:cf:a2:3f:54:a7:76:
56:cf:1c:80:0f:24:b5:61:fd:66:41:86:82:4f:5d:ba:94:0a:
2f:87:2f:be:c2:ea:a1:85:2b:00:da:a4:b3:5a:3d:64:74:6f:
33:a3:d5:08:19:64:be:07:bf:52:27:f3:43:a9:29:81:ad:de:
16:ea:aa:cc:3a:93:f0:65:09:dc:0d:67:9a:e9:7d:70:b7:ff:
6e:5e:c3:ed:43:e8:04:70:f3:a7:f3:33:ce:92:83:a4:59:c0:
9b:0c:3d:87:c3:d7:2e:a8:42:01:82:fa:e2:0f:c7:57:b5:ce:
ae:18:27:c9:9c:85:dc:cf:05:91:3f:39:84:c9:0d:15:e8:4d:
2d:09:6e:83
This output is cryptic, but there's a few things to look for. First of all, there's the following block
Issuer: CN = OwnCA
Validity
Not Before: May 22 11:30:48 2020 GMT
Not After : May 20 11:30:48 2030 GMT
Subject: CN = OwnCA
This is the most important information. It tells us who issued the certificate, when it's valid and the subject. Certificates are identified by their subject. The exact number of fields there can vary, depending on the CA. Note that Issuer and Subject match, indicating that the certificate signed itself (self-signed certificate). A CA certificate that signed itself is commonly known as a root certificate because computer scientists see this as a tree structure, where things start at the root.
In the extensions we also see CA:TRUE which means it's allowed to sign certificates:
X509v3 extensions:
X509v3 Basic Constraints:
CA:TRUE
When a certificate has CA:TRUE but is signed by another certificate, it's typically referred to as an Intermediate CA.
The host certificate can also be inspected:
# openssl x509 -in $HOSTNAME/$HOSTNAME.crt -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
06:7c:1e:d8:1c:a1:8a:b0:8b:5a:b8:63:c5:d6:57:60:97:af:57:87
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = OwnCA
Validity
Not Before: May 22 11:31:13 2020 GMT
Not After : May 22 11:31:13 2021 GMT
Subject: CN = foreman.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:cd:42:f5:36:9a:6b:05:40:2a:4c:2f:e6:30:7a:
9d:c4:fd:3d:81:b1:fe:be:57:69:29:0f:d2:a4:7f:
af:a0:91:ee:b6:34:50:ca:35:dd:d4:27:d6:54:90:
e6:9e:51:50:ab:5d:44:f2:e8:06:f9:ee:09:5f:05:
fe:d3:c3:08:71:6f:80:01:d1:42:3d:c8:3f:36:80:
89:25:0e:c3:6f:b1:e7:35:86:d1:51:10:8c:e7:58:
de:94:4e:ee:b0:74:96:6c:43:8d:4d:29:17:99:b7:
f2:f2:17:8c:3e:35:ab:52:25:ce:bd:3c:66:29:ec:
01:e8:aa:97:07:4e:32:f7:e8:0c:6b:2d:26:f1:05:
64:a8:7a:b3:f8:9e:b6:3d:a0:d6:97:f2:84:82:ba:
88:24:3c:95:b6:e6:ef:c0:a8:3d:26:63:41:9a:a7:
cc:e1:d7:a7:cc:27:04:4a:9f:54:e3:57:2b:2e:f0:
24:03:12:63:e5:18:02:31:bf:e6:46:48:fc:5d:dd:
83:46:5a:49:86:91:83:75:39:3e:31:72:2d:26:7e:
c9:9f:b0:dc:9e:9b:20:d1:8d:ea:88:31:c2:cf:78:
a1:35:3e:b3:f4:19:fd:df:f0:11:5b:22:4f:84:d3:
f7:8e:fc:0b:9e:8a:97:40:6f:c8:8b:b0:59:44:48:
fb:83
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Subject Key Identifier:
83:0E:92:A6:95:25:1A:52:84:92:97:00:C7:CF:93:D8:40:7C:E5:71
X509v3 Key Usage:
Digital Signature, Non Repudiation, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication, Code Signing, E-mail Protection
X509v3 Subject Alternative Name:
DNS:foreman.example.com
Signature Algorithm: sha256WithRSAEncryption
a7:a1:c2:36:a1:48:ae:07:0f:d6:b7:bb:19:8a:55:c9:41:42:
c7:43:a0:2e:b9:0d:32:fa:9e:a3:6c:fd:7d:5b:46:2f:20:a0:
a0:79:9e:44:be:24:37:2e:f7:80:1a:cc:d7:ad:05:94:79:9a:
bc:4c:4d:fa:a4:73:4c:8d:48:94:4e:eb:0f:fb:7d:26:8a:ab:
0b:01:31:11:1f:53:8e:64:63:df:14:ec:aa:a7:32:79:ce:26:
39:b3:03:a3:be:2f:00:63:61:c1:11:00:90:7f:f2:d2:b3:27:
96:54:ab:3c:93:60:13:e1:a3:6d:49:06:9b:3f:a5:81:95:6b:
38:0a:f6:e5:04:59:4b:75:15:86:e3:91:36:a0:d2:a2:9a:f4:
29:a6:5a:fb:42:19:a9:fa:f6:7e:6a:91:8b:c4:64:b9:e9:d1:
20:60:42:99:0f:c3:11:d2:dc:42:c0:63:da:20:c6:e7:75:dd:
50:2f:17:2b:9f:5e:58:00:8f:ea:f9:c2:f2:23:1e:68:69:31:
26:ed:ad:9e:b6:91:f5:e9:6e:41:81:76:16:3b:7d:08:a3:ab:
c9:90:2f:d1:45:7f:10:3b:8f:41:95:0b:d8:2c:bc:3c:02:61:
97:31:b9:bd:ad:85:3c:a5:69:98:a4:9f:10:e7:7e:c4:9b:5d:
0f:45:18:c3
This is largely the same as the CA certificate, but with some differences. For starters, the certificate is not allowed to sign other certificates (CA:FALSE).
More importantly the issuer and subject differ. The issuer is our CA and the subject is our hostname. Foreman still uses Common Name (CN) a lot, in particular around authentication. You might spot a potential problem with this. Common Name can only be a single name but various services have multiple. That's why modern tooling has started to use the Subject Alternative Name extension:
X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:foreman.example.com
This is the same name as our Common Name. Ensure that the Common Name is there or things will break. Additional names can be added, but is beyond the scope of this explanation.
Installing Foreman
This guide is using CentOS 8 and the Foreman 3.1 but this is applicable to many versions and all supported operating systems.
yum -y install https://yum.theforeman.org/releases/3.1/el8/x86_64/foreman-release.rpm https://yum.puppet.com/puppet-release-el-8.noarch.rpm
yum -y install foreman-installer
This doesn't install EPEL, but you may need it for some plugins or older releases.
First the permissions will need to be fixed. For this, a dedicated group is created:
OWNCA=/etc/ownca
groupadd foreman-certs
chmod 640 $OWNCA/$HOSTNAME/$HOSTNAME.key
chgrp foreman-certs $OWNCA/$HOSTNAME/$HOSTNAME.key
Now that we've prepared the certificates, the installer will do the hard work. Just running foreman-installer without arguments will install various components, but that's not what we want right now. It's customized to only install Foreman itself without a Foreman Proxy.
OWNCA=/etc/ownca
foreman-installer \
--no-enable-foreman-cli \
--no-enable-foreman-proxy \
--no-enable-puppet \
--foreman-user-groups foreman-certs \
--foreman-server-ssl-ca $OWNCA/cacert.crt \
--foreman-server-ssl-chain $OWNCA/cacert.crt \
--foreman-server-ssl-cert $OWNCA/$HOSTNAME/$HOSTNAME.crt \
--foreman-server-ssl-key $OWNCA/$HOSTNAME/$HOSTNAME.key \
--foreman-server-ssl-crl "" \
--foreman-client-ssl-ca $OWNCA/cacert.crt \
--foreman-client-ssl-cert $OWNCA/$HOSTNAME/$HOSTNAME.crt \
--foreman-client-ssl-key $OWNCA/$HOSTNAME/$HOSTNAME.key
Now that looks like a lot of duplication so we'll go over all options. We can recognize two groups: server and client.
The server is what will be used by Apache and the is most visible. Since Apache reads these files as root, file permissions are generally not that important. SELinux policies can limit it so be aware of that. The Certificate Revocation List (CRL) is unset. CA vs chain is explained later with examples.
However, Foreman also runs a bundled Websockify that will spawn for VNC consoles to VMs. It's serving its own certificates and these options control this. Websockify runs as the Foreman user so it needs to read these files, which is why we created the group. The server and websocket should use the same certificates.
There are also client certificates that are used to connect to the Foreman Proxy (also known as Smart Proxy). In this case, the certificate and key are used as client certificates rather than server. This means they're used for identification. Later we'll see how that's verified.
Our example uses the same certificates for both server and client, but this isn't required. Foreman Proxies can use a different CA. This can be useful when using a public commercial Certificate Authority for the Foreman UI but a custom CA (like Puppet CA) for internal communication. In that case --foreman-server-ssl-ca must point to the public facing CA and --foreman-server-ssl-chain the internal CA. The --foreman-client-ssl-* options must belong to the internal CA.
After the installer has completed, we can inspect what's actually being served.
# openssl s_client -connect $HOSTNAME:443 -CAfile /etc/ownca/cacert.crt
CONNECTED(00000003)
depth=1 CN = OwnCA
verify return:1
depth=0 CN = foreman.example.com
verify return:1
---
Certificate chain
0 s:CN = foreman.example.com
i:CN = OwnCA
1 s:CN = OwnCA
i:CN = OwnCA
---
Server certificate
-----BEGIN CERTIFICATE-----
<CERTIFICATE>
-----END CERTIFICATE-----
subject=CN = foreman.example.com
issuer=CN = OwnCA
---
Acceptable client certificate CA names
CN = OwnCA
<MORE CONTENT>
This is slightly truncated for readability.
The first thing to notice, is the Certificate chain. At the highest level we see our server certificate with its subject (s:) and issuer (i:). Our server also serves the CA chain below that. In this case it's only our root CA as specified in --foreman-server-ssl-chain. If your chain includes intermediate CAs, you will see those as well.
Then the Server certificate is shown, which matches --foreman-server-ssl-cert. Again, the subject and issuer are shown.
Lastly there's Acceptable client certificate CA names that matches --foreman-server-ssl-ca. Foreman uses client certificates in various places to authenticate. Any such certificate must be signed by this CA.
Visualizing the deployment
This can be summarized in a diagram.
Services listening externally have a bold border. Red means it's unencrypted and blue is encrypted using our new CA. Concretely:
- Apache listens on
*:80(unencrypted) - Apache listens on
*:443(encrypted) - Websockify allocates a free port in the 5910 - 5930 (encrypted) range when a VNC console is started
Starting Foreman 2.1, Apache works as a reverse proxy to Foreman. The Foreman service runs unencrypted on 127.0.0.1:3000 and only Apache is supposed to connect to it. This is implemented using systemd socket activation. The systemd service foreman.socket listens on a TCP socket and passes it to foreman.service as a file descriptor. No HTTP requests are dropped this way, even if foreman.service is restarted. In the future Apache to Foreman connections may be changed to use a unix socket for better security.
The VNC console part also can use some improvements. The RFC Consolidating The Console was started, but so far nobody has worked on implementation.
Adding a Foreman Proxy
By default, foreman-installer installs a Foreman Proxy, but in this blog it's split off to a separate host. This is done to better illustrate how things talk to eachother. Foreman can have many Foreman Proxies so this is a common scenario even when a proxy also exists on the Foreman host.
First a certificate is needed for our host foreman-proxy.example.com. We run this on foreman.example.com in the ownca directory.
[root@foreman ownca]# ./ownca cert foreman-proxy.example.com
Generating a RSA private key
................................+++++
...........................+++++
writing new private key to './foreman-proxy.example.com/foreman-proxy.example.com.key'
-----
Using configuration from ./openssl.cnf
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
commonName :PRINTABLE:'foreman-proxy.example.com'
Certificate is to be certified until May 26 13:07:09 2021 GMT (365 days)
Write out database with 1 new entries
Data Base Updated
Of course this can also be inspected but for brevity this is left as a user excercise. When doing so, check that the issuer is still the same CA while the subject now has the new hostname.
The installer can register a new Foreman Proxy using the REST API. Authentication is done using OAuth. By default the installer generates a random key and secret when installing Foreman but they can also be explicitly specified. To retrieve the values, you can go to the Foreman settings page. In this case we'll use the command line on foreman.example.com:
foreman.example.com # foreman-rake -- config -k oauth_consumer_key
supersecret
foreman.example.com # foreman-rake -- config -k oauth_consumer_secret
alsosecret
Copy the files cacert.crt, foreman-proxy.example.com.crt and foreman-proxy.example.com.key to the proxy host. Make sure the user foreman-proxy can read these. It's also important that the key is not world readable.
Also follow the same installation steps to get foreman-installer and run the installer on foreman-proxy.example.com:
foreman-proxy.example.com # foreman-installer \
--no-enable-foreman \
--no-enable-foreman-cli \
--no-enable-puppet \
--foreman-proxy-manage-puppet-group false \
--foreman-proxy-puppet false \
--foreman-proxy-puppetca false \
--foreman-proxy-tftp false \
--foreman-proxy-foreman-base-url https://foreman.example.com \
--foreman-proxy-trusted-hosts $HOSTNAME \
--foreman-proxy-trusted-hosts foreman.example.com \
--foreman-proxy-oauth-consumer-key supersecret \
--foreman-proxy-oauth-consumer-secret alsosecret \
--foreman-proxy-ssl-ca /path/to/cacert.crt \
--foreman-proxy-ssl-cert /path/to/$HOSTNAME.crt \
--foreman-proxy-ssl-key /path/to/$HOSTNAME.key
Here we specify the server certs, but don't explicitly configure the client certificates. That's because the same certificates as the server are used automatically. When using a different CA for Foreman, --foreman-proxy-foreman-ssl-ca must be specified as well.
There are some options to disable Puppet integration and even tftp to get a minimal installation. This only leaves the logs feature, chosen because it requires no additional dependencies and Foreman refuses to register proxies without any features.
It's also possible to specify an additional group for the foreman-proxy user with --foreman-proxy-groups mygroup. One example is having all files in /path/to/mycerts and setting permissions using chown root:mygroup /path/to/mycerts && chmod 0750 /path/to/mycerts. It is up to you to ensure mygroup exists prior to running the installer.