Securing Vault with HSM on Kubernetes

From the beginning, Bank-Vaults has been one of the core building blocks of Pipeline - Banzai Cloud's container management platform for hybrid clouds. Today we are happy to announce the release of Bank-Vaults 1.0, and the official launch of Bank-Vaults as a product with commercial support. Additionally, we have taken the step of adding Bank-Vaults support for hardware security modules, usually abbreviated as HSMs.

Bank-Vaults 1.0

Bank-Vaults was first released two years ago as a Vault operator for Kubernetes, a CLI tool and a Go library. Initially we have built Bank-Vaults for our own needs, as the central component managing secrets for our Pipeline container management platform customers, however the external adoption (and the community around it) has grown exponentially (over 1000 production deployments, 3 million Docker Hub downloads, close to 100 contributors). As part of the Pipeline platform our customers always had support for Bank-Vaults, but there was an increased demand to support it as an individual product/project as well. Bank-Vaults 1.0 just brings that with some notable new features as well as:

• Secret injection webhook improvements
• Backing up Vault with Velero
• Vault replication across multiple datacenters
• Vault secret injection webhook and Istio
• Mutate any kind of k8s resources
• HSM support

HSM Support

As Pipeline became increasingly popular among commercial and investment banks, there was increased demand that we add support for the banking industry standard safeguard mechanisms that manage digital keys. Bank-Vaults already supported multiple KMS alternatives for encrypting and storing unseal-keys and root-tokens. However, there remained a concerted interest in Bank-Vaults adding HSM support; hardware security modules (or HSMs) offer an industry-standard way of encrypting your data in standard on-premise environments. You can use HSMs to generate and store the private keys used by Bank-Vaults. The main selling point of HSM devices is their speed: they render the average PC capable of completing more cryptographic operations. The main benefit of an HSM is the increased protection of private keys and speed in handling cryptographic operations. These allow for the encryption of protected information without exposing private keys (they are not extractable). Bank-Vaults currently supports the PKCS11 software standard for communication with HSMs. As a bonus, HSMs help you fulfill some compliance requirements (for example, PCI DSS), so from now on you can meet those requirements with Bank-Vaults.

HSM implementation in Bank-Vaults

Bank-Vaults had to be changed in a few places to support HSM devices for unseal-key and root-token encryption. These are as follows:

• a new encryption/decryption Service had to be implemented, named hsm in the bank-vaults CLI
• the bank-vaults Docker image now includes SoftHSM (for testing) and OpenSC tooling
• the operator had to be made aware of HSM and its nature

HSMs offer an encryption mechanism, but the unseal-keys and root-tokens have to be stored somewhere after they are encrypted. Some HSM devices can be used to store a limited amount of arbitrary data (like Nitrokey HSM). Additionally, Bank-Vaults offers a storage backend alternative for the storage of encrypted unseal keys. In future versions, we'll add some more storage backends, but these two currently meet all our customers' requirements. Bank-Vaults also makes it possible to use previously generated cryptographic encryption keys on an HSM device, or to generate a key pair on the fly if none exists with the specified label in the specified slot. Since Bank-Vaults is written in Go, we used the miekg's PKCS11 wrapper for Go to pull in the PKCS11 library. To be more precise, we used the p11 high-level wrapper, which wraps miekg/pkcs11 and makes it easier to use, especially when compared to a more straightforward C wrapper like miekg/pkcs11. This has made our lives a lot easier. Thank you, Miek Gieben.

SoftHSM support for testing

SoftHSMv2 is a great piece of software developed by OpenDNSSEC. If you are planning to implement and test how software interacts with PKCS11, this is the way to do it.

# Initialize SoftHSM to create a working example (only for dev).
# The HSM device is emulated with a previously generated keypair in the image.
brew install softhsm
softhsm2-util --init-token --free --label bank-vaults --so-pin banzai --pin banzai
pkcs11-tool --module /usr/local/lib/softhsm/libsofthsm2.so --keypairgen --key-type rsa:2048 --pin banzai --token-label bank-vaults --label bank-vaults

You can interact with SoftHSM via the following unsealConfig snippet in Vault CR (when using vault-operator):

# This example relies on the SoftHSM device initialized in the Docker image.
unsealConfig:
  hsm:
    # The HSM SO module path (softhsm is built into the Bank-Vaults image)
    modulePath: /usr/lib/softhsm/libsofthsm2.so
    tokenLabel: bank-vaults
    pin: banzai
    keyLabel: bank-vaults

To run the entirety of the SoftHSM-based example in Kubernetes, run the following commands:

kubectl create namespace vault-infra
helm upgrade --install vault-operator banzaicloud-stable/vault-operator --namespace vault-infra
kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/cr-hsm-softhsm.yaml

NitroKey HSM support (OpenSC)

Nitrokey HSM is a USB HSM device based on the OpenSC project. We use NitroKey to develop and perform tests on real hardware-based HSMs while working on HSM support in Bank-Vaults. This device is not a cryptographic accelerator. It only supports key generation and private key operations (sign and decrypt). Public key operations should be performed through extracting the public key and working on a computer, and, accordingly, that's how those operations are implemented in Bank-Vaults. It is not possible to extract private keys from NitroKey HSM, since the device is tamper-resistant.

Courtesy of our friends at NitroKey, between March 16-23, 2020 there is a 10% discount available for Nitrokey HSM 2. Use the "Bank-Vaults" promotion code, and grab your device today.

This device supports only RSA-based encryption/decryption. As such, this mode of encryption is implemented in Bank-Vaults. Bank-Vaults supports ECC keys as well, but only for sign/verification operations. Install OpenSC and initialize the NitroKey HSM stick:

brew install opensc
sc-hsm-tool --initialize --label bank-vaults --pin banzai --so-pin banzaicloud
pkcs11-tool --module /usr/local/lib/opensc-pkcs11.so --keypairgen --key-type rsa:2048 --pin banzai --token-label bank-vaults --label bank-vaults

Check that you have a keypair object in slot 0:

pkcs11-tool --list-objects

Using slot 0 with a present token (0x0)
Public Key Object; RSA 2048 bits
  label:      bank-vaults
  ID:         a9548075b20243627e971873826ead172e932359
  Usage:      encrypt, verify, wrap
  Access:     none

pkcs15-tool --list-keys

Using reader with a card: Nitrokey Nitrokey HSM
Private RSA Key [bank-vaults]
	Object Flags   : [0x03], private, modifiable
	Usage          : [0x0E], decrypt, sign, signRecover
	Access Flags   : [0x1D], sensitive, alwaysSensitive, neverExtract, local
	ModLength      : 2048
	Key ref        : 1 (0x01)
	Native         : yes
	Auth ID        : 01
	ID             : a9548075b20243627e971873826ead172e932359
	MD:guid        : a6b2832c-1dc5-f4ef-bb0f-7b3504f67015

Setup on Minikube for testing (optional)

If you want to try this out on your Mac/in OSX by running Docker in a VM, you need to complete some extra steps before mounting your HSM device to Kubernetes; on Linux, you can skip the next section, where we configure HSM devices. We are going to use Minikube to test and validate the NitroKey setup; the following steps are necessary in order to mount them into the minikube Kubernetes cluster:

# Specify VirtualBox as the VM backend
minikube config set vm-driver virtualbox

# You need to install the Oracle VM VirtualBox Extension Pack for USB 2.0 support, so make sure it is installed (and double-check its license)
VBoxManage list extpacks

# Create a minikube cluster with the virtualbox driver and stop it (we need to modify the VM)
minikube start
minikube stop

# Enable USB 2.0 support for the minikube VM
VBoxManage modifyvm minikube --usbehci on

# Find the vendorid and productid for your Nitrokey HSM device
VBoxManage list usbhost

VENDORID=0x20a0
PRODUCTID=0x4230

# Create a filter for that device
VBoxManage usbfilter add 1 --target minikube --name "Nitrokey HSM" --vendorid ${VENDORID} --productid ${PRODUCTID}

# Restart the minikube VM
minikube start

# Now plug the USB device into your computer

# Check that minikube captured your NitorKey HSM
minikube ssh lsusb | grep ${VENDORID:2}:${PRODUCTID:2}

Now your minikube Kubernetes cluster has access to the HSM device through the USB.

Configure the Bank-Vaults operator to use NitroKey HSM-based unsealing {#unsealing-hsm}

In the vault-operator, unsealConfig is a little different for OpenSC HSM devices; there are certain things that the operator needs to be aware of in order to correctly communicate with the device:

# This example relies on an OpenSC HSM (NitroKey HSM) device initialized and plugged into the Kubernetes Node.
unsealConfig:
  hsm:
    # We need the OpenSC daemon to communicate with the device
    daemon: true
    # The HSM SO module path (opensc is built into the bank-vaults image)
    modulePath: /usr/lib/opensc-pkcs11.so
    # slotId is preferable to tokenLabel for OpenSC
    # (OpenSC appends/prepends some extra stuff to labels)
    slotId: 0
    pin: banzai # This can also be specified in the BANK_VAULTS_HSM_PIN environment variable from a Secret
    keyLabel: bank-vaults

Kubernetes node setup

Some HSM vendors offer network daemons that allow their HSM equipment to reach different servers. Unfortunately, there is no defined networking standard for PKCS11 access and thus Bank-Vaults must be scheduled to the same node the HSM device is attached to (unless you're using a Cloud HSM). Since the HSM is a hardware device connected to a physical node, Bank-Vaults has to find its way to that node. To make this work, we'll create an HSM extended resource on the Kubernetes nodes for which the HSM device is plugged in. Extended resources must be advertised in integer amounts. For example, a Node can advertise four HSM devices, but not four and a half. We need to patch the node to specify that it has an HSM device as a resource. Because of the integer constraint, and because all Bank-Vaults related Pods have to land on a Node where an HSM resource is available, we need to advertise two units for one device; one will be allocated by each Vault Pod and one by the Configurer. If you would like to run Vault in HA mode - multiple Vault instances in different nodes - you'll need to have multiple HSM devices with the same key and slot setup plugged into those nodes.

kubectl proxy &

NODE=minikube

curl --header "Content-Type: application/json-patch+json" \
     --request PATCH \
     --data '[{"op": "add", "path": "/status/capacity/nitrokey.com~1hsm", "value": "2"}]' \
     http://localhost:8001/api/v1/nodes/${NODE}/status

Going forward, this resource can be requested in PodSpec:

# When using the NitroKey HSM example, that resource has to be part of the resource scheduling request.
resources:
  hsmDaemon:
    requests:
      cpu: 100m
      memory: 64Mi
      nitrokey.com/hsm: 1
    limits:
      cpu: 200m
      memory: 128Mi
      nitrokey.com/hsm: 1

Apply the modified setup from scratch:

kubectl delete vault vault
kubectl delete pvc vault-file-vault-0
kubectl delete secret vault-unseal-keys
kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/cr-hsm-nitrokey.yaml

Check the logs that are unsealed by the NitroKey HSM device:

kubectl logs -f vault-0 bank-vaults

time="2020-03-04T13:32:29Z" level=info msg="HSM Information {CryptokiVersion:{Major:2 Minor:20} ManufacturerID:OpenSC Project Flags:0 LibraryDescription:OpenSC smartcard framework LibraryVersion:{Major:0 Minor:20}}"
time="2020-03-04T13:32:29Z" level=info msg="HSM Searching for slot in HSM slots [{ctx:0xc0000c0318 id:0}]"
time="2020-03-04T13:32:29Z" level=info msg="found HSM slot 0 in HSM by slot ID"
time="2020-03-04T13:32:29Z" level=info msg="HSM TokenInfo {Label:bank-vaults (UserPIN)\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 ManufacturerID:www.CardContact.de Model:PKCS#15 emulated SerialNumber:DENK0200074 Flags:1037 MaxSessionCount:0 SessionCount:0 MaxRwSessionCount:0 RwSessionCount:0 MaxPinLen:15 MinPinLen:6 TotalPublicMemory:18446744073709551615 FreePublicMemory:18446744073709551615 TotalPrivateMemory:18446744073709551615 FreePrivateMemory:18446744073709551615 HardwareVersion:{Major:24 Minor:13} FirmwareVersion:{Major:3 Minor:3} UTCTime:\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}"
time="2020-03-04T13:32:29Z" level=info msg="HSM SlotInfo for slot 0: {SlotDescription:Nitrokey Nitrokey HSM (DENK02000740000         ) 00 00 ManufacturerID:Nitrokey Flags:7 HardwareVersion:{Major:0 Minor:0} FirmwareVersion:{Major:0 Minor:0}}"
time="2020-03-04T13:32:29Z" level=info msg="found objects with label \"bank-vaults\" in HSM"
time="2020-03-04T13:32:29Z" level=info msg="this HSM device doesn't support encryption, extracting public key and doing encrytion on the computer"
time="2020-03-04T13:32:29Z" level=info msg="no storage backend specified for HSM, using on device storage"
time="2020-03-04T13:32:29Z" level=info msg="joining leader vault..."
time="2020-03-04T13:32:29Z" level=info msg="vault metrics exporter enabled: :9091/metrics"
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:	export GIN_MODE=release
 - using code:	gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET    /metrics                  --> github.com/gin-gonic/gin.WrapH.func1 (3 handlers)
[GIN-debug] Listening and serving HTTP on :9091
time="2020-03-04T13:32:30Z" level=info msg="initializing vault..."
time="2020-03-04T13:32:30Z" level=info msg="initializing vault"
time="2020-03-04T13:32:31Z" level=info msg="unseal key stored in key store" key=vault-unseal-0
time="2020-03-04T13:32:31Z" level=info msg="unseal key stored in key store" key=vault-unseal-1
time="2020-03-04T13:32:32Z" level=info msg="unseal key stored in key store" key=vault-unseal-2
time="2020-03-04T13:32:32Z" level=info msg="unseal key stored in key store" key=vault-unseal-3
time="2020-03-04T13:32:33Z" level=info msg="unseal key stored in key store" key=vault-unseal-4
time="2020-03-04T13:32:33Z" level=info msg="root token stored in key store" key=vault-root
time="2020-03-04T13:32:33Z" level=info msg="vault is sealed, unsealing"
time="2020-03-04T13:32:39Z" level=info msg="successfully unsealed vault"

You will also find the unseal keys and root token on the HSM:

pkcs11-tool --list-objects

Using slot 0 with a present token (0x0)
Public Key Object; RSA 2048 bits
  label:      bank-vaults
  ID:         a9548075b20243627e971873826ead172e932359
  Usage:      encrypt, verify, wrap
  Access:     none
Data object 2168561792
  label:          'vault-test'
  application:    'vault-test'
  app_id:         <empty>
  flags:           modifiable
Data object 2168561168
  label:          'vault-unseal-0'
  application:    'vault-unseal-0'
  app_id:         <empty>
  flags:           modifiable
Data object 2168561264
  label:          'vault-unseal-1'
  application:    'vault-unseal-1'
  app_id:         <empty>
  flags:           modifiable
Data object 2168561360
  label:          'vault-unseal-2'
  application:    'vault-unseal-2'
  app_id:         <empty>
  flags:           modifiable
Data object 2168562304
  label:          'vault-unseal-3'
  application:    'vault-unseal-3'
  app_id:         <empty>
  flags:           modifiable
Data object 2168562400
  label:          'vault-unseal-4'
  application:    'vault-unseal-4'
  app_id:         <empty>
  flags:           modifiable
Data object 2168562496
  label:          'vault-root'
  application:    'vault-root'
  app_id:         <empty>
  flags:           modifiable

If you would like to clean up the HSM after testing:

PIN=banzai

# Delete the unseal keys and the root token
for label in "vault-test" "vault-root" "vault-unseal-0" "vault-unseal-1" "vault-unseal-2" "vault-unseal-3" "vault-unseal-4"
do
  pkcs11-tool --delete-object --type data --label ${label} --pin ${PIN}
done

# Delete the encryption key
pkcs11-tool --delete-object --type privkey --label bank-vaults --pin ${PIN}

Additional HSM implementations

AWS CloudHSM also supports the PKCS11 API, so it should also work, though it will require a custom Docker image. If you're interested in contributing, check out the Bank-Vaults repository, or give us a GitHub star.

Learn more about Bank-Vaults:

• Secret injection webhook improvements
• Backing up Vault with Velero
• Vault replication across multiple datacenters
• Vault secret injection webhook and Istio
• HSM support
• Injecting dynamic configuration with templates
• OIDC issuer discovery for Kubernetes service accounts
• Show all posts related to Bank-Vaults