Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
INSIGHTS
12 min read
Published on 09/08/2019
Last updated on 06/18/2024
Inject secrets directly into Pods from Vault revisited
Share
A key part of the Banzai Cloud Pipeline platform, has always been our strong focus on security. We incorporated Vault into our architecture early on in the design process, and we have developed a number of support components to be easily used with Kubernetes. We love what Vault enables us to do, but, as with many things security-related, strengthening one part of our system exposed a weakness elsewhere. For us, that weakness was Vault ->
If you are familiar with Kubernetes secrets, you know that these secrets are stored in etcd. When we say that we intend to bypass K8s security, we mean that we don't intend to touch etcd at all; the problem with etcd is that when data is encrypted at rest, it is encrypted with a global key (see the relevant documentation). That's less than ideal in a multi-tenant cluster, where independent and unrelated users might potentially gain access to the secrets of others. Also, if you already have a security team that's operating a certified Vault installation, they're probably not going to be happy about placing an unencrypted secret in an intermediary location (Kubernetes secrets, i.e. etcd).
Banzai Cloud's Pipeline platform already used Kubernetes webhooks to provide a range of advanced features (security scans, spot instance scheduling, annotating webhooks, etc.), and it occured to us that using a webhook to inject secrets directly into Kubernetes containers from Vault would be a good way of bypassing etcd.
Let's dive into how it works.
K8s secrets, which is the standard way in which applications consume secrets and credentials on Kubernetes. Any secret that is securely stored in Vault and then unsealed for consumption will eventually end up as a K8s secret, and with much less protection and security than we'd like. K8s secrets use base64 encoding that, while better than nothing, does not satisfy our standards, and likely fails to satisfy the standards of most enterprise clients as well. As a result, we've developed a solution wherein we can bypass the K8s secrets mechanism and inject secrets directly into Pods from Vault.
Vault -> Kubernetes secrets -> Pod
If you are familiar with Kubernetes secrets, you know that these secrets are stored in etcd. When we say that we intend to bypass K8s security, we mean that we don't intend to touch etcd at all; the problem with etcd is that when data is encrypted at rest, it is encrypted with a global key (see the relevant documentation). That's less than ideal in a multi-tenant cluster, where independent and unrelated users might potentially gain access to the secrets of others. Also, if you already have a security team that's operating a certified Vault installation, they're probably not going to be happy about placing an unencrypted secret in an intermediary location (Kubernetes secrets, i.e. etcd).
Banzai Cloud's Pipeline platform already used Kubernetes webhooks to provide a range of advanced features (security scans, spot instance scheduling, annotating webhooks, etc.), and it occured to us that using a webhook to inject secrets directly into Kubernetes containers from Vault would be a good way of bypassing etcd.
Let's dive into how it works.
Kubernetes mutating webhook for injecting secrets
Our mutating admission webhook injects an executable into containers (in a non-intrusive way) inside Pods, which then request secrets from Vault through special environment variable definitions. This project was inspired by a number of other projects (e.g.channable/vaultenv, hashicorp/envconsul), but one thing that makes it unique is that it is a daemonless solution.
First, the Kubernetes webhook checks if a container has environment variables with values that correspond to a specific schema. Then it reads the values for those variables directly from Vault at start-up:
env: - name:
AWS_SECRET_ACCESS_KEY value:
"vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY"vault-env is attached to it as an in-memory volume. That volume is mounted to all containers with the appropriate environment variable definitions.
The init-container also changes the command of the container to run vault-env, instead of running the application directly. vault-env starts up, connects to Vault (using the Kubernetes Auth method), checks that the environment variables have a reference to a value stored in Vault (vault:secret/....) and replaces that with a corresponding value from Vault's secrets backend. Afterward, vault-env executes the original process (with syscall.Exec()), which uses the secret that was originally stored in Vault.
Using this solution prevents secrets stored in Vault from landing in Kubernetes secrets (and in etcd).
vault-env was designed to work on Kubernetes, but there's nothing stopping it from being used outside of Kubernetes as well. It can be configured with the standard Vault client's environment variables, since there's a standard Go Vault client underneath.
Currently, the Kubernetes Service Account-based Vault authentication mechanism is used by vault-env, which requests a Vault token in return for the Service Account of the container it's being injected into. But our implementation is going to change in order to allow the use of the Vault Agent's Auto-Auth feature very soon. This will allow users to request tokens in init-containers with all the authentication mechanisms supported by Vault Agent, so they won't be handcuffed to the Kubernetes Service Account-based method.
Why is this more secure than using Kubernetes secrets or using any other custom sidecar container?
Our solution is particularly lightweight and uses only existing Kubernetes constructs like annotations and environment variables. No confidential data ever persists on the disk - not even temporarily - or in etcd. All secrets are stored in memory, and only visible to the process that requests them. If you want to make this solution even more robust, you can disablekubectl exec-ing in running containers. If you do so, no one will be able to hijack injected environment variables from a process.
Additionally, there is no persistent connection with Vault, and any Vault token used to read environment variables is flushed from memory before the application starts, in order to minimize attack surface.
A complete example
This example will guide you through setting up a fully functional Vault installation with the Banzai Cloud Vault operator, and help you to create an example deployment that will be mutated by the webhook so that environment variables can be injected into Pods:# These examples require Helm 3 and kubectl:
# Add the Banzai Cloud Helm repository
helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
# Create a namespace for the bank-vaults components called vault-infra
# Namespace labeling is required, because the webhook's mutation is based on label selectors
kubectl create namespace vault-infra
kubectl label namespace vault-infra name=vault-infra
# Install the vault-operator to the vault-infra namespace
helm upgrade --namespace vault-infra --install vault-operator banzaicloud-stable/vault-operator --wait
# Clone the bank-vaults project
git clone git@github.com:banzaicloud/bank-vaults.git
cd bank-vaults
# Create a Vault instance with the operator which has the Kubernetes auth method configured
kubectl apply -f operator/deploy/rbac.yaml
kubectl apply -f operator/deploy/cr.yaml
# Now you have a fully functional Vault installation on top of Kubernetes,
# orchestrated by the `banzaicloud/vault-operator` and `banzaicloud/bank-vaults`.
# Next, install the mutating webhook with Helm into its own namespace (to bypass the catch-22 situation of self mutation)
helm upgrade --namespace vault-infra --install vault-secrets-webhook banzaicloud-stable/vault-secrets-webhook --wait
# Set the Vault token from the Kubernetes secret
# (strictly for demonstrative purposes, we have K8s unsealing in cr.yaml)
export VAULT_TOKEN=$(kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root} | base64 --decode)
# Tell the CLI that the Vault Cert is signed by a custom CA
kubectl get secret vault-tls -o jsonpath="{.data.ca\.crt}" | base64 --decode > $PWD/vault-ca.crt
export VAULT_CACERT=$PWD/vault-ca.crt
# Tell the CLI where Vault is listening (the certificate has 127.0.0.1 as well as alternate names)
export VAULT_ADDR=https://127.0.0.1:8200
# Forward the TCP connection from your Vault pod to localhost (in the background)
kubectl port-forward service/vault 8200 &
# Write a secret into Vault, which will be injected as an environment variable
vault kv put secret/accounts/aws AWS_SECRET_ACCESS_KEY=s3cr3t
# Apply the deployment with special environment variables
# It will be mutated by the webhook
kubectl apply -f deploy/test-deployment.yamlapiVersion: apps/v1
kind: Deployment
metadata:
name: hello-secrets
spec:
replicas: 1
selector:
matchLabels:
app: hello-secrets
template:
metadata:
labels:
app: hello-secrets
annotations:
vault.security.banzaicloud.io/vault-addr: "https://vault:8200"
vault.security.banzaicloud.io/vault-tls-secret: "vault-tls"
spec:
serviceAccountName: default
containers:
- name: alpine
image: alpine
command:
[
"sh",
"-c",
"echo $AWS_SECRET_ACCESS_KEY && echo going to
sleep... && sleep 10000",
]
env:
- name: AWS_SECRET_ACCESS_KEY
value: "vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY"apiVersion: v1
kind: Pod
metadata:
name: hello-secrets-575554499f-26894
labels:
app: hello-secrets
annotations:
vault.banzaicloud.io/vault-addr: "https://vault:8200"
vault.security.banzaicloud.io/vault-tls-secret: "vault-tls"
spec:
initContainers:
- name: copy-vault-env
command:
- sh
- -c
- cp /usr/local/bin/vault-env /vault/
image: banzaicloud/vault-env:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /vault/
name: vault-env
containers:
- name: alpine
command:
- /vault/vault-env
args:
- sh
- -c
- echo $AWS_SECRET_ACCESS_KEY $ && echo going to
sleep... && sleep 10000
image: alpine
imagePullPolicy: Always
env:
- name: AWS_SECRET_ACCESS_KEY
value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY
- name: VAULT_ADDR
value: https://vault:8200
- name: VAULT_SKIP_VERIFY
value: "false"
- name: VAULT_PATH
value: kubernetes
- name: VAULT_ROLE
value: default
- name: VAULT_IGNORE_MISSING_SECRETS
value: "false"
- name: VAULT_CACERT
value: /vault/tls/ca.crt
volumeMounts:
- mountPath: /vault/
name: vault-env
- mountPath: /vault/tls/ca.crt
name: vault-tls
subPath: ca.crt
volumes:
- emptyDir:
medium: Memory
name: vault-env
- name: vault-tls
secret:
secretName: vault-tlsAWS_SECRET_ACCESS_KEY variable is only visible inside the alpine container.
Using charts without an explicit container.command and container.args
The Webhook is now capable of determining the container's entry point and command with the help of image metadata queried from the image registry. This data is cached until the webhook Pod is restarted. If the registry is publicly accessible (without authentication), you don't need to do anything, but, if the registry requires authentication, the necessary credentials have to be made available in the Pod'simagePullSecrets section, or in the Pod's ServiceAccount.
NOTE: Future improvement: on AWS and GKE and other cloud providers get a credential dynamically with the cloud-specific SDK
MySQL example:
# Put the MySQL passwords into Vault
vault kv put secret/mysql MYSQL_ROOT_PASSWORD=s3cr3t MYSQL_PASSWORD=3xtr3ms3cr3t
# Install the MySQL chart with root and a user password sourced from Vault
helm upgrade --install mysql stable/mysql \
--set mysqlRootPassword=vault:secret/data/mysql#MYSQL_ROOT_PASSWORD \
--set mysqlPassword=vault:secret/data/mysql#MYSQL_PASSWORD \
--set "podAnnotations.vault\.security\.banzaicloud\.io/vault-addr"=https://vault:8200 \
--set "podAnnotations.vault\.security\.banzaicloud\.io/vault-tls-secret"=vault-tls \
--wait
# Open a connection towards the MySQL Service
kubectl port-forward service/mysql 3306 &
# Read the MySQL user password's secret from Vault
# Make sure you still have the port-forward from the previous example
vault read secret/data/mysql
Key Value
--- -----
data map[MYSQL_PASSWORD:3xtr3ms3cr3t MYSQL_ROOT_PASSWORD:s3cr3t]
metadata map[created_time:2019-09-05T13:03:42.980780517Z deletion_time: destroyed:false version:1]
# Open up the MySQL shell with the root user and its corresponding password from Vault `s3cr3t`
mysql -h 127.0.0.1 -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 84
Server version: 5.7.14 MySQL Community Server (GPL)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
# And here's the magic: there are still no secrets in the env vars!
kubectl exec -it mysql-749cfddc67-5slcb bash
root@mysql-749cfddc67-5slcb:/\# env | grep MYSQL.*PASSWORD
MYSQL_PASSWORD=vault:secret/data/mysql#MYSQL_PASSWORD
MYSQL_ROOT_PASSWORD=vault:secret/data/mysql#MYSQL_ROOT_PASSWORDvault-env it will fork a new process which will have those environment variables correctly set:
/vault/vault-env env | grep MYSQL.*PASSWORD
2019/09/05 13:42:09 Received new Vault token
2019/09/05 13:42:09 Initial Vault token arrived
MYSQL_PASSWORD=3xtr3ms3cr3t
MYSQL_ROOT_PASSWORD=s3cr3t
Consequentially, it's advised you disallow the
"pods/exec" Kubernetes RBAC rule snippet for users you don't want doing this:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
namespace: default
name: pod-execer
rules:
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]Getting secrets data from Vault and transplanting it into a Kubernetes secret
You can mutate secrets by setting annotations and defining the proper Vault path in the secret data:apiVersion: v1
kind: Secret
metadata:
name: sample-secret
annotations:
vault.security.banzaicloud.io/vault-addr: "https://vault.default.svc.cluster.local:8200"
vault.security.banzaicloud.io/vault-role: "default" # In case of Secrets the webhook's ServiceAccount
is used
vault.security.banzaicloud.io/vault-skip-verify: "true"
vault.security.banzaicloud.io/vault-path: "kubernetes"
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: eyJhdXRocyI6eyJodHRwczovL2RvY2tlci5pbyI6eyJ1c2VybmFtZSI6InZhdWx0OnNlY3JldC9kYXRhL2RvYkubectl create secret docker-registry dockerhub --docker-username="vault:secret/data/dockerrepo#DOCKER_REPO_USER" --docker-password="vault:secret/data/dockerrepo#DOCKER_REPO_PASSWORD"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-addr="https://vault.default.svc.cluster.local:8200"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-role="default"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-skip-verify="true"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-path="kubernetes"Multiple (and dynamic) secret backends not just KV
Currently,vault-env supports reading Values from the KV backend, but we have added support for dynamic secrets as well - database URLs with temporary usernames and passwords for batch or scheduled jobs, for example. This feature is implemented with consul-template's Vault component and is based on the work of Jürgen Weber. It deserved its own blog-post, and is described in detail in Vault webhook - complete secret support with consul-template.
Extensions in the works
Something we're still working on is templating (transforming/combining secret values), an extension based on the Go and Sprig templates; this is a frequently requested freature. The webhook is now available as anintegrated service (we used to call them posthooks) of the Pipeline platform, which means that you can install it on a Kubernetes clusters via Pipeline (with the UI or with the CLI). It will then configure Vault with the authentications and policies necessary for it to work with our webhook - track this issue here.
For more information, or if you're interested in contributing, check out the Bank-Vaults repo - the Vault Swiss army knife and operator for Kubernetes, and/or give us a GitHub star if you think the project deserves it!
Learn more about Bank-Vaults:
- Secret injection webhook improvements
- Backing up Vault with Velero
- Vault replication across multiple datacenters
- Vault secret injection webhook and Istio
- HSM support
- Injecting dynamic configuration with templates
- OIDC issuer discovery for Kubernetes service accounts
- Show all posts related to Bank-Vaults
About Banzai Cloud Pipeline
Banzai Cloud’s Pipeline provides a platform for enterprises to develop, deploy, and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operations teams alike. Strong security measures — multiple authentication backends, fine-grained authorization, dynamic secret management, automated secure communications between components using TLS, vulnerability scans, static code analysis, CI/CD, and so on — are default features of the Pipeline platform.
Subscribe to
the Shift!
Get emerging insights on emerging technology straight to your inbox.
