We recently worked on building out a Hashicorp Vault deployment to support secrets management across parts of our infrastructure. Overall, Vault is fairly straightforward to understand and operate once you become comfortable with its architecture and terminology. While setup and configuration really isn’t very difficult, we knew that we wanted to automate many aspects of our deployment to make it as zero-touch as possible. Building out this automation within our business constraints made for an interesting project. This article will discuss some of the tools that we built and the challenges that we faced while integrating Vault in our environment.
Background
Prior to our efforts with Vault, our secrets management strategy was usually closely coupled to our configuration management strategy using Puppet. Secrets might make their way into a configuration file via a Puppet file server mount point containing the secrets file. Access to this file would be restricted to the systems engineering team.
Decoupling secrets management from configuration management has several benefits, as described in a great talk by Seth Vargo at PuppetConf 2016. While this was a large undertaking, there were a few notable benefits of going this route:
- Static secret rotation becomes easier. Instead of needing to file a systems engineering ticket to update a static secret, such as an external service API key, we can enact Vault policies that allow authorized users to simply rotate the secret on their own.
- We can use dynamic secrets for the use cases where they make sense.
- Data is encrypted at rest, unlike a static file on one of our Puppet masters.
- Vault policies permit very fine-grained access to secrets in a way that Puppet and Foreman do not.
- Vault provides very detailed audit logs about who accessed a secret, what they did (e.g., read, modify, delete, etc.), and when they did it.
The setup for a host-implementation looks pretty simple:
Two binaries run on each host:
- Vault Agent - This is simply the regular Vault binary running in agent mode. Agent mode allows you to specify some authentication material (such as a certificate) and obtain a Vault token. This token can optionally be written to disk.
- Consul Template - Consul Template allows you to dynamically render configuration files using secrets that are stored in Vault (or configuration data stored in Consul, as the name suggests).
The actual implementation is relatively easy, but automating all of this to work within the limits of Puppet is a bit more involved. Some of the questions that we had to consider include:
- How do Vault policies become associated with a host? This shouldn’t be a manual process, but should be expressed through our configuration management platform (Puppet).
- How do we manage policies at all? How can we ensure that policies are sane when they make it into Vault, allow other teams to contribute policies that they need for their use cases, and avoid losing track of the policies that we have in Vault?
Key Concepts
Vault terminology can be a bit confusing, so let’s get some core ideas out of the way. To interact with secrets in Vault, you need a token. A token is a unique identifier that is sent to Vault with each request. The token must have the appropriate policies associated with it, granting (or denying) permissions to secrets in Vault. Vault also has the concept of an entity and an entity-alias. This allows a user (human or machine) to log in to Vault via multiple methods (referred to as authentication backends in Vault parlance) and still have the necessary policies associated with their token. An example of this entire flow is instructive:
- Bob logs in to Vault using either Vault’s GitHub or Okta authentication backends.
- An entity-alias for both GitHub and Okta tie Bob’s log in to a single entity in Vault
- This entity has policies associated with it.
- When Bob authenticates, he receives a token with all of the necessary policies applied to it.
- Bob can now access secrets with his token
If you’re thoroughly confused, the Vault tutorials have a pretty good walkthrough of this in action.
Based on this information, if we want our servers to interact with Vault, then we need:
- An authentication method for the machines to use that does not require human intervention.
- A set of policies that grant a machine access to only those secrets that it should have access to.
- An entity for each machine with the appropriate policies
- An entity-alias tying the authentication mechanism in Step 1 to the entity for the machine
Machine Authentication for Vault
The first thing that we wanted to achieve is a way for our hosts to authenticate to Vault with minimal effort. Ideally, we would like such an authentication scheme to:
- Allow us to uniquely identify hosts for authorization purposes
- Deploy easily without much intervention at server build or deploy time (i.e., having to find a way to distribute something like a password to all of our machines would be untenable)
- “Just work”
Since we use Puppet throughout our environment, each machine already has some information that we trust to uniquely identify hosts: a certificate signed by our Puppet Masters. Each Puppet certificate contains the FQDN of the server in the certificate’s CN, which we can then use to uniquely identify servers throughout our fleet and assign policies to them in Vault. We already trust this certificate and private key for configuring our servers, so using it for Vault authentication is a natural extension of this trust.
As mentioned in the intro, Vault can be run in agent mode. With the simple configuration below, a machine can authenticate with Vault using its Puppet certificate and write the obtained token to a file sink, which is just a fancy way of saying that the token will be written to a file on the filesystem (at /etc/vault-agent/puppet-token).
root@testbox:/etc/vault-agent# cat config.hcl
vault {
address = "https://vault.example.com"
client_cert = "/etc/puppetlabs/puppet/ssl/certs/testbox-1.example.com.pem"
client_key = "/etc/puppetlabs/puppet/ssl/private_keys/testbox-1.example.com.pem"
}
auto_auth {
method "cert" {
config = {
type = "cert"
mountpath="auth/cert/"
}
}
sink "file" {
config = {
path = "/etc/vault-agent/puppet-token"
}
}
}
On the Vault side, we leverage a certificate authentication backend. Vault is configured to trust the Puppet CA certificate and allow any host to authenticate with its local Puppet certificate. The tokens generated via this authentication method have a maximum lifetime of 24 hours, and they receive a default policy of managed-puppet-default. More on policies later.
$ vault list auth/cert/certs
Keys
----
puppet
$ vault read auth/cert/certs/puppet
Key Value
--- -----
allowed_common_names <nil>
allowed_dns_sans <nil>
allowed_email_sans <nil>
allowed_names <nil>
allowed_organizational_units <nil>
allowed_uri_sans <nil>
certificate -----BEGIN CERTIFICATE-----
< removed for brevity >
-----END CERTIFICATE-----
display_name puppet
max_ttl 24h
required_extensions <nil>
token_bound_cidrs []
token_explicit_max_ttl 0s
token_max_ttl 24h
token_no_default_policy false
token_num_uses 0
token_period 0s
token_policies [managed-puppet-default]
token_ttl 0s
token_type defaultThe managed-puppet-default policy that hosts receive by default doesn’t grant any actual permissions to read other secrets in Vault. To grant access to secrets, we need to find a way to associate policies with our servers.
Policies
Vault policies are written in Hashicorp Configuration Language or JSON and specify a set of permissions on a path. For example, if I have secrets in Vault at the blog-article/ path, I can grant full access to this path (and all sub-paths) with a policy like the one below:
path "blog-article/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
I won’t dig into policies in too much detail for this explanation, as they’re pretty straightforward. In our environment, a few design considerations became immediately apparent:
- Policies won’t change too often. A host, app, or user will likely be granted sets of permissions only occasionally, and these will persist.
- Policies should be tightly controlled: they should be peer-reviewed and placed in source control to ensure that we can track any changes.
- Policy drift can be problematic. We need a way to ensure that policies in the production Vault deployment don’t change without us knowing.
We addressed all of these considerations by writing a policy import script, using Python and the HVAC library, that handles both policy importation and helps us to identify policy drift. The script, along with the policies themselves, are stored in a Git repository. Within that repository, there’s a “policies” directory. Each file in the directory is named “managed-$policy_name” to denote that it is being managed by the policy importer.
For example, we might have a managed-api-servers policy that we apply to hosts that serve an API and need access to a specific set of relevant secrets. This would be stored at the policies/managed-api-servers.hcl location in our repository. The prefix allows us to exclude prefixes, if needed (e.g., if we need to manage policies via a different mechanism in the future).
The policy importer also identifies any policies in Vault that do not have a corresponding entry in the policy directory. For example, let’s say someone added a my-test-policy to Vault manually via the command line (something that permissions shouldn’t allow anyway). The policy importer will detect that this policy does not have a corresponding policy file, and will trigger an alert to let us know.
Finally, there’s the human element of this. Automatically importing policies doesn’t do us any good if anyone can write any policy that they want, including dangerous policies that grant unnecessarily broad permissions. We addressed this problem via GitLab. Any requests to add a new policy involve a merge request to our repository, which is protected to ensure that even maintainers cannot silently push directly to the master branch. MRs require a minimum of two approvals from the teams that manage Vault. Once approved and merged, the policy importer is automatically kicked off via a CI/CD pipeline.
Theory is interesting, but let’s take a look at this in action. First, let’s see what policies we have:
root@vault-dev# ls -l policies/
total 20
-rw-r--r-- 1 root root 95 Dec 3 15:27 managed-blog-article.hcl
-rw-r--r-- 1 root root 991 Dec 3 15:13 managed-certificate-importer.hcl
-rw-r--r-- 1 root root 536 Dec 3 15:13 managed-policy-importer.hcl
-rw-r--r-- 1 root root 403 Dec 3 15:13 managed-puppet-default.hcl
-rw-r--r-- 1 root root 623 Dec 3 15:13 managed-vault-admin.hcl
The policies directory above has 5 policies. For this example, managed-blog-article does not yet exist in Vault and we’re going to add it using the importer. Let’s take a look at that policy and verify that it doesn’t already exist in Vault:
root@vault-dev# cat policies/managed-blog-article.hcl
path "blog-article/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
root@vault-dev# vault policy read managed-blog-article
No policy named: managed-blog-article
As promised, we can see that the policy doesn’t yet exist in Vault. In a normal workflow, we would:
- Add this policy in a separate branch of the repository
- Submit a merge request to master
- Have the merge request approved
- Upon merge, the policy importer would run as part of a CI/CD pipeline
Since this is a blog article, I’ll skip to step 4 and manually run the importer:
root@vault-dev# ./script/vault-policy-importer.py --vault-addr https://vault.example.com --app-role-id dd6de864-c405-912e-630d-623033cae801 --app-role-secret-id 8a13fa38-2697-369a-87f7-b4b513dfe252
Successfully imported managed-certificate-importer.hcl
Successfully imported managed-blog-article.hcl
Successfully imported managed-vault-admin.hcl
Successfully imported managed-policy-importer.hcl
Successfully imported managed-puppet-default.hcl
There are policies in Vault that are not in Git. All policies must be in source control.
Offending policies: {'some-new-policy'}
Exiting.
The policy importer is written to allow both certificate and approle authentication, with approle authentication being used in the example above. Notice that all of the policies were successfully imported, and the importer also warned us about a policy that wasn’t in Git: some-new-policy.
Taking a look at the list of policies in Vault, we can see that some-new-policy does indeed exist. It must have been added outside of the Git workflow and should be deleted. Once we have confidence in this general approach, we’ll probably have the policy importer automatically delete anything that isn’t in Git. In the meantime, we’re being cautious.
root@vault-dev# vault policy list
default
gitlab-ci-somepolicy1
gitlab-ci-somepolicy2
managed-blog-article
managed-certificate-importer
managed-policy-importer
managed-puppet-default
managed-vault-admin
some-new-policy
root
root@vault-dev# vault policy delete some-new-policy
Success! Deleted policy: some-new-policyYou’ll also notice that there are a few policies in the list that do not have corresponding policy files in the repository, but the importer doesn’t complain about them: default, root, gitlab-ci-somepolicy1, and gitlab-ci-somepolicy2. We designed the importer to have an exclusion list so that we can have certain policies added via other means. For example, policies matching the gitlab-ci-* pattern are ignored because they are managed via a different method.
This gets our policies into Vault with tight control over the policies themselves, but we still need a way to associate these policies with individual (or groups of) servers in an automated way. Let’s take a look at how we address this in the next section.
Associating policies with hosts
Associating policies with a host is conceptually simple, but we wanted a strategy that would automate this process and be consistent with our current configuration management strategy. Our primary configuration management tool is Puppet. While Puppet does an excellent job for configuration management, it lacks any orchestration capabilities and doesn’t easily integrate with orchestration-based workflows. This introduces some challenges that we had to overcome to ensure that Vault policy management could fit into our overall infrastructure with its current Puppet-centric design.
Since Puppet is our primary configuration management tool, we wanted a way to ensure that the policies used by a host could be represented in our Puppet code. Implementing another set of tools would result in a great deal of scope creep just for managing Vault. While we may use other tooling someday, most of our infrastructure processes are centered on concepts that are tied to our Foreman and Puppet infrastructure: host groups, environment-specific Puppet manifests, and others. Duplicating this tooling into another orchestration tool just for the purpose of deploying Vault would be untenable, both technically and organizationally.
At a high level, we were faced with the following automation scenario:
- All hosts have Puppet manifests associated with them. Generally, these are applied at a hostgroup level in Foreman.
- Puppet will run on a host at regular intervals.
- During that Puppet run, the catalog for that host will be compiled. Somehow, we need to associate a Vault policy with the host.
To solve this problem, we wrote a custom Puppet module called vault_agent. Aside from handling installation and general configuration of Vault agent, this module also exposes a defined resource type called a policy. The policy defined resource type can then be called from other manifests, such as the usual ones that we write to maintain our infrastructure. More specifically, it can be exported. If you’re unfamiliar with exported Puppet resources, then be sure to check out the official documentation. Exported resources can be collected by other Puppet manifests, including those run on other hosts.
Let’s say I want all hosts using a certain Puppet manifest to have the managed-blog-article policy applied to them in Vault. To accomplish this, I can simply export the vault_agent::policy defined type that we created. Note that I use $facts[‘fqdn’] to ensure that the resource declaration is unique, as required by Puppet.
@@vault_agent::policy { "$facts['fqdn']_managed-blog-article":
fqdn => $facts['fqdn'],
policy => ‘managed-blog-article,
}
The Puppet manifests for each host (or app environment, or group of hosts, etc.) will export one or more policies that should be applied to the host. The Puppet manifests on our Vault server then collect these exported policies. By collecting the Vault policies, it can then apply them for each host. Collection syntax, like most things in the Puppet DSL, is ugly:
Vault_agent::Policy <<| |>>You’re probably thinking: OK, you can export policies for each host and collect them on the Vault server. But how exactly does that get the policies into Vault itself? Good question! To understand how this works, you need to take a look at what exactly this vault_agent::policy defined type really is under the hood:
define vault_agent::policy (
String $fqdn,
String $policy,
) {
concat::fragment { "${fqdn}_${policy}":
target => '/etc/vault-certificate-importer/host_policies.txt',
content => inline_template("<%= @fqdn %> <%= @policy %>\n"),
}
}If you’re familiar with Puppet, you’ll see that this is just a simple concat fragment resource. The Puppet concat module allows you to create multiple fragments and combine them together into one file (in this case, /etc/vault-certificate-importer/host_policices.txt).
The /etc/vault-certificate-importer/host_policies.txt file is just a newline delimited file of host/policy pairs:
testbox-1.example.com managed-some-policy-1
testbox-1.example.com managed-some-policy-2
testbox-2.example.com managed-some-policy-3We then have a script that reads this file and reconciles the list of expected policies with the actual list of policies in Vault. This approach works because exporting and collecting resources occurs at catalog compilation time, which occurs on the Puppet master and obtains exported resources from PuppetDB. To add a new policy for a host, a user must have access to the Puppet masters or the Git repositories that we use to store our Puppet manifests. This kind of access is tightly controlled in our environment.
The Importer Script
Our policy defined type, which is really just a thin wrapper around the Puppet concat resource, will lay down a file with host/policy pairs in it. We now need a way to get those policies into Vault, as well as perform some other housekeeping activities. To do this, we implemented a script that imports all policies in /etc/vault-certificate-importer/host_policies.txt. The script also handles reconciling any entities that are showing up in Vault but aren’t in the policy file (we want everything to be in Puppet and, by extension, the policy file).
When a host authenticates to Vault, an entity is automatically created if it doesn’t already exist. Vault also creates an entity-alias that ties the entity to the authentication method. For example, a server in our environment might have an entity-alias entry that looks like the one below:
root@vault-dev# vault read identity/entity-alias/id/fe908dd3-e0bd-b590-43df-b841bc6e75eb
Key Value
--- -----
canonical_id 4c1f647c-b171-dd02-17a1-652f520cac9a
creation_time 2019-11-11T20:38:24.397350451Z
id fe908dd3-e0bd-b590-43df-b841bc6e75eb
last_update_time 2019-11-11T20:38:24.397350451Z
merged_from_canonical_ids <nil>
metadata <nil>
mount_accessor auth_cert_6ed5379a
mount_path auth/cert/
mount_type cert
name testbox-1.example.com
namespace_id rootNotice that the name of the entity-alias is the FQDN of the host, which is also the CN of the certificate. When this host (testbox-1.example.com) authenticated to Vault using its Puppet certificate, Vault automatically created both the entity and the alias. The entity has an ID of 4c1f647c-b171-dd02-17a1-652f520cac9a as denoted by the canonical-id field in the alias.
Looking at the entity itself, we can see a list of policies (in this case, only one policy) associated with it:
root@vault-dev# vault read identity/entity/id/4c1f647c-b171-dd02-17a1-652f520cac9a
Key Value
--- -----
aliases [map[canonical_id:4c1f647c-b171-dd02-17a1-652f520cac9a creation_time:2019-11-11T20:38:24.397350451Z id:fe908dd3-e0bd-b590-43df-b841bc6e75eb last_update_time:2019-11-11T20:38:24.397350451Z merged_from_canonical_ids:<nil> metadata:<nil> mount_accessor:auth_cert_6ed5379a mount_path:auth/cert/ mount_type:cert name:testbox-1.example.com]]
creation_time 2019-11-11T20:38:24.313799346Z
direct_group_ids []
disabled false
group_ids []
id 4c1f647c-b171-dd02-17a1-652f520cac9a
inherited_group_ids []
last_update_time 2019-12-02T14:43:45.773552178Z
merged_entity_ids <nil>
metadata map[]
name testbox-1.example.com
namespace_id root
policies [managed-blog-article]As you can see, the policies are applied to the entity, not the entity-alias. So, from a programmatic standpoint, we need to:
- Look up the entity-alias based on the FQDN of a host
- Obtain the entity ID from the entity-alias
- Apply policies to the entity
The script should also account for entities that don’t yet exist. For example: maybe Puppet ran on a host and the Vault Agent started up, but the host hasn’t been able to reach Vault due to a missing firewall rule. In this case, the host and associated policies will show up in /etc/vault-certificate-importer/host_policies.txt, but there won’t yet be an entity and entity-alias for the host. In this case, the script simply creates one.
To implement this, we wrote another script using Python and the HVAC library. HVAC provides handy Python bindings to most of the Vault API. In our case, the list of steps above are implemented almost exactly as they sound using calls to the HVAC identity API. The script also imports the Puppet CRL and provides us with a list of hosts that aren’t in the policy file (and are unmanaged for some reason).
We obviously don’t want this script to run all the time, so we trigger runs via a systemd path unit. The path unit watches for changes on the host_policies.txt file and triggers a oneshot service that runs the script.
Wait a Second…
Astute readers will notice that there is a flaw in our Puppet logic for exporting Vault policies. Essentially, the workflow looks like this:
- Puppet runs on a host and installs Vault Agent. The catalog compilation also exports a policy resource.
- The host, excited about its newfound Vault Agent abilities, immediately starts the agent and successfully authenticates. After all, it has a valid certificate signed by Puppet.
- At some point in the future, Puppet runs on our Vault servers. It updates the policy file by collecting all of the host policies, and this triggers a certificate importer run.
- The certificate importer runs and ensures that policies are correctly associated with the entity for each host.
The flaw exists between steps 2 and 4. For that duration of time (i.e., from when Puppet runs on a host to when it runs on the Vault server) the host does not actually have any policies (other than a default) in Vault. Therefore, if the host attempts to read any secrets from Vault, then those reads will fail.
This is a hard one to solve using only Puppet. Ideally, the certificate importer logic wouldn’t exist at all, but Puppet lacks any ability to perform orchestration (or trigger external orchestration jobs without writing your own Puppet functions, which we wanted to avoid for now).
After giving this some thought, we decided that it’s not a huge deal in our environment. We rarely spin up servers that go into service immediately. Instead, we chose to mitigate this with a lower Puppet run interval on the Vault servers (Puppet is set to run every 30 minutes, instead of our higher default). This closes the delay window to 30 minutes, which is an acceptable duration for our current business needs. Looking toward the future, this will inevitably become a problem that we’ll have to optimize for, likely by writing custom Puppet functions to trigger external orchestration tooling.
Consul Template
So far, I’ve only discussed the authentication and authorization of hosts with Vault. With all of the above automation, a host can authenticate with Vault and receive a token with policies that allow it to do something. But it’s the something that we all came here for: the ability to grab secrets out of Vault and render them to files, such as application configuration files.
Consul Template makes this largely painless. It has the ability to:
- Read a Puppet token from a file, such as the Vault Agent sink at
/etc/vault-agent/token - Read template files that are written in a special templating language. These template files can reference secrets that are stored in Vault
- Authenticate with Vault, obtain any necessary secrets, and render the template file to an output file, such as an application config file
- Optionally run arbitrary commands, such as service restarts, after a render
- Periodically check in with Vault to see if the secret has changed and re-render files
We decided to write a thin Puppet module wrapper for Consul Template. Our Puppet module allows for installing and enabling the service with a base config, as well as providing a defined resource type for the templates themselves. I won’t cover the base class here, since it does exactly what it sounds like: installs Consul Template and lays down a base configuration that allows the service to talk with Vault. The template resource is more interesting:
consul_template::template { 'symfony_parameters_yml':
ensure => present,
source => '/etc/consul-template/templates/symfony_parameters_yml.tpl',
template_file_location => "puppet:///modules/${module_name}/symfony_parameters_yml.tpl",
destination => '/var/www/vhosts/app/current/app/config/parameters.yml',
command => '/var/www/vhosts/app/current/app/bin/console cache:clear',
}The above resource declaration is all that’s needed to define a template in Puppet. Under the hood, this resource type will create a configuration file for Consul Template to tell it about the template and lay down the template file. It exposes options for all of the Consul Template parameters, such as the destination for the rendered file and an optional command to run after the file is rendered.
First, let’s take a look at a secret that is stored in Vault:
$ vault kv get app-secrets/web_app_secrets
====== Metadata ======
Key Value
--- -----
created_time 2019-12-10T21:55:02.72671449Z
deletion_time n/a
destroyed false
version 1
======= Data =======
Key Value
--- -----
api_key zaYbz7JfnB88f6fU
db_password 7XZELebRJ3XK4zB6After a Puppet run, all of the configuration is in place to kick off Consul Template. First, let’s take a look at the Consul Template configuration file. This tells Consul Template about the file to render. All of the config options available in the config are also available as parameters in the defined type, but I omitted them in the example above for brevity.
root@appserver-1:/etc/consul-template# cat conf.d/symfony_parameters_yml.hcl
template {
source = "/etc/consul-template/templates/symfony_parameters_yml.tpl"
destination = "/var/www/vhosts/app/current/app/config/parameters.yml"
create_dest_dirs = false
error_on_missing_key = true
backup = true
left_delimiter = "{{"
right_delimiter = "}}"
command = "/var/www/vhosts/app/current/app/bin/console cache:clear"
command_timeout = "60s"
wait {
min = "2s"
max = "10s"
}
}Next, there’s the actual template itself:
root@appserver-1:/etc/consul-template# cat templates/symfony_parameters_yml.tpl
parameters:
app_name: my_app
log_level: info
{{ with secret "app-secrets/web_app_secrets" }}
db_password: {{ .Data.data.db_password }}
api_key: {{ .Data.data.api_key }}
{{ end }}
Assuming all went well, we’ll have a configuration file that has been properly rendered at the destination:
root@appserver-1:~# cat /var/www/vhosts/app/current/app/config/parameters.yml
parameters:
app_name: my_app
log_level: info
db_password: 7XZELebRJ3XK4zB6
api_key: zaYbz7JfnB88f6fUAnd that’s it! This is just for static key/value secrets in Vault. I haven’t even gotten to dynamic secrets, which are way cooler and enable very short-lived secrets. A topic for another time.
Final Thoughts
Vault is very powerful and its core concepts are straightforward and approachable. Making it all work with your configuration management tool of choice in a way that is easy to manage is a bit more difficult. In this post, I walked you through some of the ways we solved Vault deployment challenges in an automated way that allows us to keep careful track of our policies, automatically authenticate hosts with Vault, and deploy both Vault Agent and Consul Template using Puppet. It’s possible (indeed, very likely) that our automation strategy will evolve over time as we find aspects that could use some tuning. Hopefully, this insight into our Vault management strategy gives you some ideas to implement on your own. Have better ideas? Let us know what you think in the comments!
