This blog post will describe how to set up a Vault server using self signed SSL certificates.
This tutorial is not an introduction to Vault and will require you to have some previous knowledge of both Vault and Vagrant.
This blog post uses Vagrant and CentOS 6 for demonstration purposes.
What is Vault?
Vault is a tool made by Hashicorp that stores and manages secrets.
Why do I care?
Mainly because encryption is hard and solutions are usually expensive. Vault however is completely free and open source. The fact that is it open source is important as this allows users to have full visibility as to how and why it does certain things. Those same users can also help to make the product better and more secure if they find issues.
How do I get started?
There are two ways to use Vault. Dev mode and Production mode. In this example we will be looking at using Vault with self signed certificates. This will be the best representation of how Vault works in Production mode.
This is important for a couple of reasons. In Dev mode, Vault reads and writes secrets in memory and also has TLS disabled. While this is a great way of learning the Vault mechanics, it's not a good example of testing real world scenarios.
Generate some SSL certificates!
This section is heavily based off of https://langui.sh/2009/01/18/openssl-self-signed-ca/ and can be performed locally on your machine or on the Vagrant machine. For Windows users I suggest using the Vagrant machine for the tutorial.
Step 1: Generate a root certificate first of all:
$ openssl req -newkey rsa:2048 -days 3650 -x509 -nodes -out root.cer
Step 2: Generate a cert and a key:
$ openssl req -newkey rsa:1024 -nodes -out vault.csr -keyout vault.key
Step 3: Create an empty certindex file and a serialfile containing a hexidecimal value
$ echo 000a > serialfile
$ touch certindex
Now we create our own Certificate Authority in order to sign the certificate:
Be sure to edit the following paths in this file to point at the certs you have just generated:
# The root certificate we generated in Step 1
# The key file we created in Step 2
# The certindex file we created in Step 3
# The serial file we created in Step 3
[ ca ]
default_ca = myca
[ myca ]
new_certs_dir = /tmp
unique_subject = no
certificate = /path/to/root.cer
database = /path/to/certindex
private_key = /path/to/vault.key
serial = /path/to/serialfile
default_days = 365
default_md = sha1
policy = myca_policy
x509_extensions = myca_extensions
copy_extensions = copy
[ myca_policy ]
commonName = supplied
stateOrProvinceName = supplied
countryName = supplied
emailAddress = optional
organizationName = supplied
organizationalUnitName = optional
[ myca_extensions ]
basicConstraints = CA:false
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always
subjectAltName = IP:127.0.0.1
keyUsage = digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth
It's important to set the
subjectAltName to be the IP or hostname of the Vault server. You can add multiple IP's/hostnames here. The following link explains why this is important.. When running a Vault server on the same machine you're performing commands on, 127.0.0.1/localhost are the values to use (this is what we will be doing).
If the Vault server was living on an external machine from the one we are using, we would use the IP /hostname of that machine. In the case of a cluster of Vault servers, you want to specify all hostnames and IP addresses needed.
Now let's create the signed certificate!
$ openssl ca -batch -config vault-ca.conf -notext -in vault.csr -out vault.crt
Now let's go and clone down the sample repo: https://github.com/samdunne/vault-ssl-tutorial
$ cd /path/to/repo
$ vagrant up
This will bring up a machine with Consul preinstalled (which we are using for the purposes of this tutorial as our backend storage).
Let's get Vault up and running
So lets create some files and folders to allow vault to run.
First let's create the directory where we will store the vault configuration files.
$ sudo mkdir /etc/vault.d
Next we will install Vault on the machine.
$ cd /tmp
$ wget http://bit.ly/1lwm84c -O vault.zip
$ unzip vault.zip >/dev/null
$ sudo chmod +x vault
$ sudo mv vault /usr/local/bin/vault
Let's create a file called
/etc/vault.d/configuration.json and put the following contents into it:
Trying to start the vault server at this point will result in an error:
$ vault server -config=/etc/vault.d
Error initializing listener of type tcp: error loading TLS cert: open /var/lib/vault/ssl/vault.crt: no such file or directory
Add SSL certificates
Let's create the directory where we will be keeping the certificates.
$ sudo mkdir -p /var/lib/vault/ssl
Let's start the server again to check if it can read our certificates correctly:
$ vault server -config=/etc/vault.d
==> Vault server configuration:
Log Level: info
Mlock: supported: true, enabled: false
Backend: consul (HA available)
Advertise Address: https://127.0.0.1:8200
Listener 1: tcp (addr: "127.0.0.1:8200", tls: "enabled")
==> Vault server started! Log data will stream in below:
So all seems to be up and running. But it isn't finished just yet. What happens when we try to initialise Vault?
First lets start the Vault server in the background so we can interact with it:
$ nohup vault server -config=/etc/vault.d &
Now try to initialise the vault:
$ vault init
Error initializing Vault: Put https://127.0.0.1:8200/v1/sys/init: x509: certificate signed by unknown authority
So as we can see, Vault doesn't trust our certs. We can always bypass this by doing the following:
$ vault status -tls-skip-verify
Error checking seal status: Error making API request.
URL: GET https://127.0.0.1:8200/v1/sys/seal-status
Code: 400. Errors:
* server is not yet initialized
But this is not what we want. We want vault to trust our CA.
Update the CA trust
This is the key to the entire exercise. This is how we will get Vault to trust our self signed certificates. We are going to update the CA trust on the machine to include our certificate.
First we will enable the ability to update the CA trust:
$ sudo update-ca-trust enable
Next create the file:
/etc/pki/ca-trust/source/anchors/ca.crt. Copy the contents of
root.cer into this file.
And finally tell the machine to extract the new CA certificates.
$ sudo update-ca-trust extract
So now when we try to initialise Vault we should have no issues:
$ vault init
Key 1: 66db37f3ff6e5a0543f11d1d3a4b3d76d2f4821dde899e0a760dcce2eff7be7501
Key 2: 62e3585ebcf97bf93deade15695f140d9f5c99c6c65892ea846d2e0a4129ece202
Key 3: 9b22f8b3fa27e0abbfa794ef8b916f1584defdaad30d07fca2a81b0dc05d8fd303
Key 4: 2e6874aca1b7c438830d2a6c024544fffef6f2a34d77b4aeb5498e89e8df4e2e04
Key 5: d7a9d441e7695f6a01406096e08b3fe7e57496cf582221b8938cbb8e69ab2d1f05
Initial Root Token: debd9cda-ce74-a67c-41a0-50f6aa58a9f0
Vault initialized with 5 keys and a key threshold of 3. Please
securely distribute the above keys. When the Vault is re-sealed,
restarted, or stopped, you must provide at least 3 of these keys
to unseal it again.
Vault does not store the master key. Without at least 3 keys,
your Vault will remain permanently sealed.
To see that it's working, we can navigate to the Consul web ui and see the new vault:
Is it possible to automate this?
I'm glad you asked! This question was put to me by David Cusack when we were looking at using Vault. What we discovered was some good and some bad.
Unsealing makes the process of automating a Vault install difficult. Automated tools can easily install, configure, and start Vault, but unsealing it is a very manual process.
Chef has helped us automate this to a certain extent. However, the solution is quite basic and could definitely be improved upon. I'll write a further blog post detailing our progress in the next few weeks.