Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
15 min read
Published on 08/08/2020
Last updated on 02/05/2024
Hardened FIPS compliant Istio with Backyards
Share
Companies providing services for government, healthcare, or finance sectors must ensure that their products adhere to government approved security requirements that ensure the protection of sensitive data. The Federal Information Processing Standards (FIPS) outlines the rules for cryptographic modules employed as mechanism that implement security. A cryptographic module is a combination of hardware and/or software that handles cryptographic functions such as encryption, decryption, digital signatures, authentication techniques and random number generation.
The FIPS 140-2 standard provides four increasing, qualitative levels of security: Level 1, Level 2, Level 3, and Level 4. These levels are intended to cover a wide range of potential applications and environments into which cryptographic modules may be employed.
In this post we'll discuss the approach we took to reaching FIPS 140-2 Security Level 1 compliance for our Istio distribution, Backyards (now Cisco Service Mesh Manager).
While doing the same check with the Backyards FIPS 140-2 compliant version, we can see that it uses the validated BoringCrypto library:
While the FIPS 140-2 compliant Backyards Istio proxy image contains a custom built compliant version of Envoy:
While the FIPS 140-2 compliant Backyards version uses BoringCrpto with strict FIPS settings:
Check the
We can access detailed information about a TLS endpoint by running testssl.sh.
The following is the output of such a check against the
If you do the same thing on the cluster with the FIPS 140-2 compliant Backyards mesh, the outputs will look totally different.
As you can see our FIPS compliant Istio build only allows connection with TLS1.2+ and only with FIPS compliant ciphers. Also, the only allowed elliptic curve algorithm is the FIPS compliant P-256/prime256v1.
Note that there is a newer/updated FIPS version, FIPS 140-3. FIPS 140-3 testing will begin on September 22, 2020.
tl;dr
- we are using a Go build where the standard crypto library is replaced with an FIPS compliant library (BoringCrypto)
- we are building Envoy with the same FIPS compliant library (BoringCrypto)
- Backyards (now Cisco Service Mesh Manager) delivers a custom Istio build, using the same FIPS compliant library (BoringCrypto)
- Backyards (now Cisco Service Mesh Manager) includes lots of automations and convenience features. One is certificate management which is closely related to FIPS. Backyards (now Cisco Service Mesh Manager) now comes with a version of cert-manager built with the same FIPS compliant library (BoringCrypto)
- as of now, the hardened FIPS compliant version of Backyards (now Cisco Service Mesh Manager) is tested with FIPS 140-2 compliant cipher suites (and rejects anything else)
- although FIPS 140 allows other ciphers, we only have GCM ciphers enabled, because only those can prevent the SSL LUCKY13 timing attack
FIPS at Banzai Cloud
While, for now, we're discussing how we built a FIPS 140-2 compliant Istio with Backyards (now Cisco Service Mesh Manager), several of the steps we'll be covering are shared by other products we are working to make FIPS compliant:- PKE, our CNCF certified Kubernetes distribution,
- Pipeline, the container management platform for hybrid clouds (supporting 5 major cloud providers, bare metal and VMware),
- Supertubes, production ready Kafka on Kubernetes,
- Bank-Vaults, the Vault Swiss Army knife and secret injection webhook,
- and One Eye, the ultimate observability tool for Kubernetes.
You might want to check out how we manage AWS GovCloud Kubernetes clusters with Pipeline
FIPS challenges
FIPS Security Level 1 mandates the usage of approved security algorithms or functions used by cryptographic modules for generating keys that are used to protect sensitive data. This means that any exchange of sensitive data between applications must occur over secure channels, using only FIPS validated cryptographic modules. For the list of FIPS validated cryptographic modules see Cryptographic Module Validation Program (using Google as a vendor example).For a guide on how to read an FIPS Validation Listing, follow this linkIn microservices architecture, as its name implies, multiple services (applications) are employed to implement scalable, flexible, fault-tolerant and secure products that serve complex business requirements. The components that such a system comprise are often written in different programming languages using different technologies. Ensuring that each of these components use an FIPS validated cryptographic module can be easily complicated:
- there might be no FIPS validated library (cryptographic module) available that is compatible with the component
- third party components may not allow replacing the used cryptography with a FIPS validated one
- each component may require its own custom solution which increases the maintenance burden
FIPS 140-2 encryption using Istio
If we take a look at a typical microservice deployment running on our Istio distribution, Backyards (now Cisco Service Mesh Manager), we can see that the above mentioned challenges apply to it as well. Also, in the Istio community there is a longstanding feature request of using FIPS 140-2 compliant versions of Istio (which has resulted in lots of discussion on forums, internal WG or Slack). It is our pleasure, then, to announce that the wait is over: Backyards (now Cisco Service Mesh Manager) and all the components that ship as part of the [Banzai Cloud Istio distribution] are now built with FIPS compliant crypto modules.In a nutshell
So what steps did we take? What details make Istio FIPS compliant? As discussed in the tl;dr: section:- we are using a Go build where the standard crypto library is replaced with an FIPS compliant library (BoringCrypto)
- we are building Envoy with the same FIPS compliant library (BoringCrypto)
- Backyards (now Cisco Service Mesh Manager) delivers a custom Istio build, using the same FIPS compliant library (BoringCrypto)
- Backyards (now Cisco Service Mesh Manager) includes lots of automations and convenience features, including certificate management. Backyards (now Cisco Service Mesh Manager) now comes with a version of cert-manager built with the same FIPS compliant library (BoringCrypto)
- the hardened FIPS compliant version of Backyards (now Cisco Service Mesh Manager) is now tested with FIPS 140-2 compliant cipher suites (and rejects anything else)
- although FIPS 140 allows for other ciphers, we only have GCM ciphers enabled, since only they can prevent an SSL LUCKY13 timing attack
Note: as FIPS introduces lots of restrictions on the accepted cipher suites and can introduce cryptographic incompatibilities, the default version of Backyards (now Cisco Service Mesh Manager) is still built with the standard SSL libraries. If you'd like to use an FIPS compliant version of Istio/Backyards (now Cisco Service Mesh Manager), get in touch with us.
Under the hood
The bottom line for FIPS compliance from the application standpoint is that it must use FIPS verified crypto libraries.Go-based applications
Unfortunately, the native crypto in Go is not FIPS validated, so we need a way to use a validated crypto library with Go. Luckily, it's Google's BoringSSL to the rescue.BoringSSL is a fork of OpenSSL that is designed to meet Google’s needs.BoringSSL as a whole is not FIPS validated. However, there is a core library (called BoringCrypto) that has been FIPS validated. Google also maintains a Go fork that can use BoringCrypto for cryptographic functions.
The distribution name for a Go+BoringCrypto release has the form < GoVersion>b< BoringCryptoVersion>, where < GoVersion> is the Go version the release is based on, and < BoringCryptoVersion> is an integer that increments each time there is a new release with different BoringCrypto bits.You can use the Go+BoringCrypto distribution as a drop-in replacement, no code changes usually needed for simple compilation. Indeed, it takes more than just a plain re-compile to get our application to make use of the BoringCrypto. The application has to be re-compiled with specific build configuration:
- The build must be GOOS=linux, GOARCH=amd64.
- The build must have cgo enabled.
- The android build tag must not be specified.
- The cmd_go_bootstrap build tag must not be specified.
To check whether a given binary is using BoringCrypto, runFurthermore, we made changes in the Istio codebase to make sure that FIPS is enforced properly in each and every binary. Our build has a more strict TLS configuration than FIPS 140-2 requires, and places similar limitations on TLS versions and used ciphers. This required additional changes in the codebase, but it further enhanced application security. Compare the built binaries The compiled binaries can be verified by the aforementionedgo tool nmon it and check that it has symbols named*_Cfunc__goboringcrypto_*. The program rsc.io/goversion will report the crypto implementation used by a given binary when invoked with the -crypto flag.
goversion tool.
If we check the upstream binary of Istiod, for example, we'll find that it uses the standard non-FIPS compliant standard Go crypto:
❯ goversion ./pilot-discovery
pilot-discovery go1.14.2 (standard crypto)❯ goversion ./pilot-discovery
pilot-discovery go1.14.6b4 (boring crypto) +crypto/tls/fipsonlyEvery Istio component the FIPS compliant Backyards distribution deploys uses custom built binaries that are using FIPS 140-2 validated BoringCrypto library.
Envoy
Istio uses Envoy in its data-plane, so the next step is to add an FIPS compliant version of it as well. Envoy uses BoringSSL already and it can be built in an FIPS 140-2 compliant mode by using the--define boringssl=fips build option.
The correctness of the resulting FIPS build can be verified by checking the presence of BoringSSL-FIPS in the --version output.
Besides the FIPS compliant build of Envoy, we've also made changes in Istio to ensure that the generated Envoy configuration distributed through XDS has strict and compliant TLS settings.
Compare the Envoy builds
The upstream Istio proxy image contains a non-FIPS 140-2 compliant Envoy build:
~ ❯ docker run -ti --rm --entrypoint envoy istio/proxyv2:1.6.7 --version
envoy version: ccae1bd37085ecd78415dc06b50233b3b97e30c0/1.14.4/Clean/RELEASE/BoringSSL~ ❯ docker run -ti --rm --entrypoint envoy banzaicloud/istio-proxyv2:1.6.7-bzc-fips --version
envoy version: 136ca7b03ff5821697d68b77e7411dbc6bfdd5ef/1.14.4/Clean/RELEASE/BoringSSL-FIPS
cert-manager
Backyards installs cert-manager and relies on it for various certificate management related tasks (webhook and ingress certs for example). This was another stumbling block on our path to FIPS 14 0-2 compliance. We used the same tooling, configuration and code base changes as with Istio's components.If you are interested in cert-manager on Kubernetes, we recently blogged [about > it]({{< relref "/blog/cert-management-on-kubernetes.md" >}}), specifically about [cert management on > Istio]({{< relref "/blog/cert-management-on-istio.md" >}}) as well.Compare the built cert-manager binaries If we check the upstream version of the cert manager controller we'll find that it is uses standard Go crypto:
❯ goversion -crypto ./controller
./controller go1.14.2 (standard crypto)❯ goversion -crypto ./controller
./controller go1.14.2 (standard crypto)Showtime
We are going to use two Kubernetes clusters that were created with Banzai Cloud Pipeline. One cluster has an Istio mesh installed and configured using our FIPS 140-2 compliant Backyards version, while the other cluster has a non-compliant mesh installed, which will allow us to demonstrate the differences between the two. Backyards contains a microservice-based demo application which is also installed on both clusters and will be used in the following steps. We are going to use thefrontpage service which serves requests on port 8080. The Istio mesh is configured with strict auto mTLS on both clusters.
Since the most visible aspects of FIPS 140-2 compliance are the TLS settings, the last item in the demo toolbox today is a handy TLS/SSL testing tool called testssl.sh.
Deploy testssl.sh pod on both clusters
We deployed the testssl pods on each cluster into the default namespace using the following snippet:❯ kubectl create -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: testssl
namespace: default
spec:
containers:
- name: app
image: drwetter/testssl.sh
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 3000; done;" ]
EOFCheck the frontpage service endpoint
We can access detailed information about a TLS endpoint by running testssl.sh.
The following is the output of such a check against the frontpage service on the cluster that uses the non-FIPS compliant build. The workload runs inside an Istio mesh, thus the TLS here is provided by Istio sidecar Envoy proxy.
non FIPS compliant cluster ❯ kubectl exec -ti testssl -- testssl.sh https://frontpage.backyards-demo.svc.cluster.local.:8080
Start 2020-08-06 13:46:47 -->> 10.11.241.177:8080 (frontpage.backyards-demo.svc.cluster.local.) <<--
rDNS (10.11.241.177): frontpage.backyards-demo.svc.cluster.local.
Service detected: certificate-based authentication => skipping all HTTP checks
Testing protocols via sockets except NPN+ALPN
SSLv2 not offered (OK)
SSLv3 not offered (OK)
TLS 1 offered (deprecated)
TLS 1.1 offered (deprecated)
TLS 1.2 offered (OK)
TLS 1.3 offered (OK): final
NPN/SPDY not offered
ALPN/HTTP2 h2 (offered)
Testing cipher categories
NULL ciphers (no encryption) not offered (OK)
Anonymous NULL Ciphers (no authentication) not offered (OK)
Export ciphers (w/o ADH+NULL) not offered (OK)
LOW: 64 Bit + DES, RC[2,4], MD5 (w/o export) not offered (OK)
Triple DES Ciphers / IDEA not offered
Obsoleted CBC ciphers (AES, ARIA etc.) offered
Strong encryption (AEAD ciphers) with no FS offered (OK)
Forward Secrecy strong encryption (AEAD ciphers) offered (OK)
Testing server's cipher preferences
Has server cipher order? yes (OK) -- only for < TLS 1.3
Negotiated protocol TLSv1.3
Negotiated cipher TLS_AES_256_GCM_SHA384, 253 bit ECDH (X25519)
Cipher per protocol
Hexcode Cipher Suite Name (OpenSSL) KeyExch. Encryption Bits Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
-
SSLv3
-
TLSv1 (server order)
xc013 ECDHE-RSA-AES128-SHA ECDH 256 AES 128
x2f AES128-SHA RSA AES 128
xc014 ECDHE-RSA-AES256-SHA ECDH 256 AES 256
x35 AES256-SHA RSA AES 256
TLSv1.1 (server order)
xc013 ECDHE-RSA-AES128-SHA ECDH 256 AES 128
x2f AES128-SHA RSA AES 128
xc014 ECDHE-RSA-AES256-SHA ECDH 256 AES 256
x35 AES256-SHA RSA AES 256
TLSv1.2 (server order)
xcca8 ECDHE-RSA-CHACHA20-POLY1305 ECDH 253 ChaCha20 256
xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 253 AESGCM 128
xc013 ECDHE-RSA-AES128-SHA ECDH 253 AES 128
x9c AES128-GCM-SHA256 RSA AESGCM 128
x2f AES128-SHA RSA AES 128
xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 253 AESGCM 256
xc014 ECDHE-RSA-AES256-SHA ECDH 253 AES 256
x9d AES256-GCM-SHA384 RSA AESGCM 256
x35 AES256-SHA RSA AES 256
TLSv1.3 (no server order, thus listed by strength)
x1302 TLS_AES_256_GCM_SHA384 ECDH 253 AESGCM 256
x1303 TLS_CHACHA20_POLY1305_SHA256 ECDH 253 ChaCha20 256
x1301 TLS_AES_128_GCM_SHA256 ECDH 253 AESGCM 128
Testing robust forward secrecy (FS) -- omitting Null Authentication/Encryption, 3DES, RC4
FS is offered (OK) TLS_AES_256_GCM_SHA384
TLS_CHACHA20_POLY1305_SHA256
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-SHA
ECDHE-RSA-CHACHA20-POLY1305
TLS_AES_128_GCM_SHA256
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-SHA
Elliptic curves offered: prime256v1 X25519FIPS compliant cluster ❯ kubectl exec -ti testssl -- testssl.sh https://frontpage.backyards-demo.svc.cluster.local.:8080
Start 2020-08-06 14:32:22 -->> 10.63.243.56:8080 (frontpage.backyards-demo.svc.cluster.local.) <<--
rDNS (10.63.243.56): frontpage.backyards-demo.svc.cluster.local.
Service detected: certificate-based authentication => skipping all HTTP checks
Testing protocols via sockets except NPN+ALPN
SSLv2 not offered (OK)
SSLv3 not offered (OK)
TLS 1 not offered
TLS 1.1 not offered
TLS 1.2 offered (OK)
TLS 1.3 offered (OK): final
NPN/SPDY not offered
ALPN/HTTP2 h2 (offered)
Testing cipher categories
NULL ciphers (no encryption) not offered (OK)
Anonymous NULL Ciphers (no authentication) not offered (OK)
Export ciphers (w/o ADH+NULL) not offered (OK)
LOW: 64 Bit + DES, RC[2,4], MD5 (w/o export) not offered (OK)
Triple DES Ciphers / IDEA not offered
Obsoleted CBC ciphers (AES, ARIA etc.) not offered
Strong encryption (AEAD ciphers) with no FS offered (OK)
Forward Secrecy strong encryption (AEAD ciphers) offered (OK)
Testing server's cipher preferences
Has server cipher order? yes (OK) -- only for < TLS 1.3
Negotiated protocol TLSv1.3
Negotiated cipher TLS_AES_256_GCM_SHA384, 256 bit ECDH (P-256)
Cipher per protocol
Hexcode Cipher Suite Name (OpenSSL) KeyExch. Encryption Bits Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
-
SSLv3
-
TLSv1
-
TLSv1.1
-
TLSv1.2 (server order)
xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 256 AESGCM 128
xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 256 AESGCM 256
x9c AES128-GCM-SHA256 RSA AESGCM 128
x9d AES256-GCM-SHA384 RSA AESGCM 256
TLSv1.3 (no server order, thus listed by strength)
x1302 TLS_AES_256_GCM_SHA384 ECDH 256 AESGCM 256
x1303 TLS_CHACHA20_POLY1305_SHA256 ECDH 256 ChaCha20 256
x1301 TLS_AES_128_GCM_SHA256 ECDH 256 AESGCM 128
Testing robust forward secrecy (FS) -- omitting Null Authentication/Encryption, 3DES, RC4
FS is offered (OK) TLS_AES_256_GCM_SHA384
TLS_CHACHA20_POLY1305_SHA256
ECDHE-RSA-AES256-GCM-SHA384
TLS_AES_128_GCM_SHA256
ECDHE-RSA-AES128-GCM-SHA256
Elliptic curves offered: prime256v1Backyards provides gateway configuration features with Let's Encrypt support utilizing an FIPS 140-2 compliant build of cert-manager, so the same TLS settings apply not just in east-west mTLS communication, but for ingress TLS traffic as well.
FIPS 140-2 compliant Backyards TLS settings
Allowed TLS versions- TLS v1.2
- TLS v1.3
Although FIPS 140-2 would allow lower TLS versions under some circumstances, we disabled them for security reasons. TLS 1.0 and 1.1 are out-of-date protocols that do not support modern cryptographic algorithms, and they contain security vulnerabilities that may be exploited by attackers. The IETF is also planning to officially deprecate both protocols. In addition, the vast majority of encrypted Internet traffic is now over TLS 1.2, which was introduced over a decade ago.Allowed FIPS compatible ciphers
- ECDHE-RSA-AES128-GCM-SHA256
- ECDHE-RSA-AES256-GCM-SHA384
- ECDHE-ECDSA-AES128-GCM-SHA256
- ECDHE-ECDSA-AES256-GCM-SHA384
- AES128-GCM-SHA256
- AES256-GCM-SHA384
There are more ciphers allowed by FIPS 140-2. We only enable GCM ciphers, because only those ciphers can prevent a LUCKY13 timing attackAllowed Elliptic-curve algorithm
- P-256
Summary
As was mentioned in the beginning, ensuring that each of the components are using an FIPS validated cryptographic module is easily complicated if:- there might be no FIPS validated library (cryptographic module) available that is compatible with the component
- third party components may not allow replacing the used cryptography with an FIPS-validated one
- each component requires its own custom solution which increases the maintenance burden
Check out Backyards in action on your own clusters! Register for an evaluation version and run a simple install command! If you’d like to use a FIPS 140-2 compliant version of Istio/Backyards, get in touch with us. Or just take a look at some of the Istio features that Backyards automates and simplifies for you, and which we've already blogged about. .
Subscribe to
the Shift!
Get emerging insights on emerging technology straight to your inbox.
