Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
11 min read
Published on 04/26/2021
Last updated on 02/05/2024
What's new in Istio 1.9, a quick walkthrough
Share
Istio 1.9 contains some exciting new features and numerous enhancements. The core focus of the release, however, is to increase operational stability for production users.
In this post, we'll review what's new in Istio 1.9, and highlight a few potential snags to look out for when upgrading to the latest version.
In Istio 1.8, experimental features like smart DNS proxying and auto registration for VMs provided features to conveniently include workloads running in VMs as part of the Istio mesh. For Istio 1.9, the stability, testing and documentation were further improved for VM integration and the feature is now promoted to Beta.
One specific area where improvements were delivered for Istio 1.9 is in the multi-primary multi-cluster setup. Previously, the
In Istio 1.9, Alpha support was added for this Gateway API. On the long run, the goal is to use this API in Istio by default. To do that, an incremental and slow migration process is aimed to be used in Istio.
Right now the Gateway API can be extended by Istio resources either by:
The Istio sidecar injector webhook now better utilizes pod labels to determine if injection is required.
There were two limitations with the previous implementation:
To enhance that, in Istio 1.9 experimental support was added to use external authorization systems in a more convenient way. This is a high level view of how the feature works (borrowed from this blog post):
It is possible now to exchange a Kubernetes token for another token, which can be authenticated by XDS.
There's a new OIDC JWT authenticator added to Istio 1.9, which supports both JWKS-URI and OIDC discovery. The feature can be enabled with the
Prior to Istio 1.9, the
When debugging Istio, you often need to check the Envoy access logs to figure out what's wrong with your application. That's why it is welcome to see that the default access log format in Istio has been supplemented with two new fields, which might help debugging in some cases.
These fields are
New pprof profiling debug endpoints were added to pilot-agent for easier debugging of performance issues, memory leaks, etc.
gRPC logging can be enabled for pilot with the
If you are familiar with the
Last, but not least, to mitigate DockerHub's rate limiting restrictions, all Istio images are published to
Istio 1.9 changes
High impact changes (can cause issues when upgrading the mesh):- Supported Kubernetes versions changed
- EnvoyFilter XDS v2 removal
- Service tags added to trace spans
- Virtual Machine Integration promoted to Beta
- Request/Response Classification promoted to Beta
- Kubernetes Gateway API support
- Improved sidecar injector selection
- Integration with external authorization systems
- Enable Security Token Service (STS) token exchange for XDS flow
- Add OIDC JWT authenticator
- Remote fetch and load of WebAssembly (Wasm) HTTP filters
- Introduce DNS_AUTO_ALLOCATE option
- List sidecar injectors
- Enhance access logs
- Add pprof endpoint to pilot-agent
- Allow gRPC logging for pilot
- DestinationRule inheritance
- Mirroring of images on gcr.io
For the full list of changes please refer to the official change notes.
High impact changes
Supported Kubernetes versions changed
For Istio 1.8, the officially supported Kubernetes versions are 1.16, 1.17, 1.18 and 1.19. For Istio 1.9, these are 1.17, 1.18, 1.19 and 1.20.
In Istio 1.9, Kubernetes 1.16 is not officially supported, therefore it is recommended to upgrade to K8s 1.17 before upgrading to Istio 1.9!
EnvoyFilter XDS v2 removal
The XDS v2 API support was removed in the new Envoy version, hence all EnvoyFilters using the old v2 API need to be updated to v3 before upgrading to Istio 1.9.More info can be found in Istio upgrade docs.This feature increases the storage need per span. If that causes an issue, it can be disabled with the
PILOT_ENABLE_ISTIO_TAGS=false env var in istiod.
Under the radar changes
Virtual Machine Integration promoted to Beta
One specific area where improvements were delivered for Istio 1.9 is in the multi-primary multi-cluster setup. Previously, the WorkloadEntry resources needed to be manually copied to all the clusters so that the VMs could be reached from every cluster. Because of this, in the multi-cluster setup the auto registration feature did not work automatically, it needed these additional manual copying steps.
The solution for this in Istio 1.9 is that all WorkloadEntry resources will be read from all clusters in multi-cluster installations. This way auto registration works for multi-primary multi-cluster scenarios as well. This feature is disabled by default and can be enabled with the PILOT_ENABLE_CROSS_CLUSTER_WORKLOAD_ENTRY environment variable in istiod.
One more aspect to note here is that VMs can be attached to a mesh in single-network and multi-network setups as well.Here are a few links collected on the Istio webpage to get started with the VM integration feature.
Request/Response Classification promoted to Beta
There is always a need to more precisely monitor application traffic. That is why it's good to see Request Classification as a Beta feature in Istio 1.9.
The Request/Response classification feature makes it possible to easily collect telemetry data based on request parameters (e.g. URL paths) or response parameters (e.g. response codes). As an example, how many GET requests arrived for a specific URL path can be measured.The feature can be used with EnvoyFilters using the AttributeGen plugin. The resulting attribute can be added as a dimension to a standard Istio metric (e.g. istio_requests_total).
Kubernetes Gateway API support
In Kubernetes, there is an ongoing effort to restructure the old ingress API and have a new well-defined API for service networking. This effort is called the Gateway API project.
If this effort rings a bell to you, but you remember it being named to
Service APIs, don't worry, this one is not yet another project. These two are the same, only theService APIsproject was renamed toGateway APIin February 2021.
The goal of the project is to provide a good common API (which should cover all the common features of different ingress implementations), and built-in integration points for custom features in different implementations.
Implementing this common API opens up opportunity for out-of-the-box integrations between different tools implementing this API. Migrating between different ingress implementations using this API should be painless in the future.
In Istio 1.9, Alpha support was added for this Gateway API. On the long run, the goal is to use this API in Istio by default. To do that, an incremental and slow migration process is aimed to be used in Istio.
Right now the Gateway API can be extended by Istio resources either by:
- explicit composition: concrete reference to Istio resources from the k8s Gateway resource
- implicit composition: Istio can layer the resources together without concrete references
Here's a quick intro on the subject by John Howard presented at IstioCon and a getting started guide for the Gateway API on the Istio page.
Improved sidecar injector selection
- Opting out individual pods with the
sidecar.istio.io/inject: falselabel from an auto injected namespace still triggered the webhook and the pod was only filtered out in istiod's mutating webhook implementation. - It was not possible to opt in a single pod with
sidecar.istio.io/inject: truelabel if auto injection was not enabled for the whole namespace.
ObjectSelector field, instead istiod's mutating webhook endpoint implementation. This way the aforementioned two issues are solved and overall a better sidecar injection experience is provided based on namespace and pod labels.
The feature is not enabled by default in Istio 1.9, but can be tested using values.sidecarInjectorWebhook.useLegacySelectors=false setting. It is planned to be enabled by default in Istio 1.10.
Security features
Integration with external authorization systems
There are cases where Istio users would like to use their own external authorization system (e.g. OPA, OAuth2, etc.) for their applications and not the built-in authorization solutions provided by Istio. Prior to Istio 1.9, this could only be done with someEnvoyFilter magic, which was not straightforward, and as we know EnvoyFilters are not very reliable.
This feature can be enabled by using the CUSTOM action in Istio's Authorization Policy CR spec. Here's an example where the authorization of the requests arriving to the foo-gateway is delegated to the custom-authz authorization extension:
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: ext-authz
namespace: foo
spec:
selector:
matchLabels:
app: foo-gateway
action: CUSTOM
provider:
name: "custom-authz"Here's how you can get started with the feature, there is a blog post and a video, which explain the feature in greater detail.
Enable (Security Token Service) STS token exchange for XDS flow
Add OIDC JWT authenticator
JWT_RULE environment variable.
Experimental Features
Remote fetch and load of WebAssembly (Wasm) HTTP filters
Istio 1.9 introduced an experimental feature to fetch WASM modules from remote locations and dynamically load them without the need of restarting the proxies. This way custom C++ code can be injected to proxies easily. Allowing advanced use cases to be handled that are not possible with the existing Istio APIs. To use this feature a new EnvoyFilter.ApplyTo field calledEXTENSION_CONFIG was introduced. This new field distributes the provided configuration as an Envoy Extension Configuration Discovery Service (ECDS) resource. When the istio-agent in an istio-proxy container receives this update, it downloads the remote WASM filter from the remote URL and stores its content in its local file system.
Here's a (not complete!) example how this could be used:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: basic-auth-config
namespace: istio-system
spec:
configPatches:
- applyTo: EXTENSION_CONFIG
match:
context: GATEWAY
patch:
operation: ADD
value:
name: istio.basic_auth
typed_config:
"@type": type.googleapis.com/udpa.type.v1.TypedStruct
type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
value:
config:
vm_config:
vm_id: basic-auth
runtime: envoy.wasm.runtime.v8
code:
remote:
http_uri:
uri: https://github.com/istio-ecosystem/wasm-extensions/releases/download/1.9.2/basic-auth.wasm
# The configuration for the Wasm extension itself
configuration:
"@type": type.googleapis.com/google.protobuf.StringValue
value: |
{
"basic_auth_rules": [
{
"prefix": "/productpage",
"request_methods":[ "GET", "POST" ],
"credentials":[ "ok:test", "YWRtaW4zOmFkbWluMw==" ]
}
]
}Here's the getting started doc for the WASM remote fetch and load feature.
Introduce DNS_AUTO_ALLOCATE option
DNS_CAPTURE flag controlled both substituting for kube-dns for VMs and auto allocation of ServiceEntry addresses. In Istio 1.9, these features are now decoupled.
The DNS_CAPTURE flag now only controls kube-dns DNS, while the new experimental DNS_AUTO_ALLOCATE flag enables the automatic ServiceEntry address allocation.
More info can be found in theIstio docs on DNS proxying.
List sidecar injectors
An experimentalistioctl x injector list command was added to show which namespaces are using which Istio revision and how many proxies are up-to-date/need restart:
$ istioctl x injector list
NAMESPACE ISTIO-REVISION POD-REVISIONS
backend 18x 18x: 9
frontend 19x 18x: 4 NEEDS RESTART: 4Other notable changes
Enhance access logs
RESPONSE_CODE_DETAILS and CONNECTION_TERMINATION_DETAILS, this is how they might look like in logs:
- RESPONSE_CODE_DETAILS (
via_upstream,filter_chain_not_found,http1.codec_error,rbac_access_denied_matched_policy[none]):[2020-10-29T03:22:16.126Z] "GET /headers HTTP/1.1" 200 - via_upstream - "-" 0 547 5 4 "-" "curl/7.73.0-DEV" "24e58e43-e576-9048-829b-02e3f3da16f0" "httpbin:8000" "127.0.0.1:80" inbound|8000|| 127.0.0.1:51680 172.17.0.7:80 172.17.0.6:59022 outbound_.8000_._.httpbin.foo.svc.cluster.local default [2020-10-29T03:23:33.415Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.0.7:80 172.17.0.6:59800 - - [2020-10-29T03:24:13.520Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 172.17.0.7:80 172.17.0.6:60214 - - [2020-10-29T03:25:20.162Z] "GET /headers HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "curl/7.73.0-DEV" "07541777-6ee2-908f-a78b-867fe257ac80" "httpbin:8000" "-" - - 172.17.0.7:80 172.17.0.6:60898 outbound_.8000_._.httpbin.foo.svc.cluster.local - [2020-10-29T03:26:11.561Z] "GET /headers HTTP/1.1" 403 - rbac_access_denied_matched_policy[ns[foo]-policy[httpbin]-rule[0]] - "-" 0 19 0 - "-" "curl/7.73.0-DEV" "090a1897-d696-99b1-90f0-2673113ba258" "httpbin:8000" "-" - - 172.17.0.7:80 172.17.0.6:33204 outbound_.8000_._.httpbin.foo.svc.cluster.local - - CONNECTION_TERMINATION_DETAILS (
rbac_access_denied_matched_policy[ns[foo]-policy[httpbin]-rule[0]]):[2020-10-29T03:22:16.126Z] "GET /headers HTTP/1.1" 200 - via_upstream - "-" 0 547 5 4 "-" "curl/7.73.0-DEV" "24e58e43-e576-9048-829b-02e3f3da16f0" "httpbin:8000" "127.0.0.1:80" inbound|8000|| 127.0.0.1:51680 172.17.0.7:80 172.17.0.6:59022 outbound_.8000_._.httpbin.foo.svc.cluster.local default [2020-10-29T03:23:33.415Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.0.7:80 172.17.0.6:59800 - - [2020-10-29T03:24:13.520Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 172.17.0.7:80 172.17.0.6:60214 - - [2020-10-29T03:25:20.162Z] "GET /headers HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "curl/7.73.0-DEV" "07541777-6ee2-908f-a78b-867fe257ac80" "httpbin:8000" "-" - - 172.17.0.7:80 172.17.0.6:60898 outbound_.8000_._.httpbin.foo.svc.cluster.local - [2020-10-29T03:26:11.561Z] "GET /headers HTTP/1.1" 403 - rbac_access_denied_matched_policy[ns[foo]-policy[httpbin]-rule[0]] - "-" 0 19 0 - "-" "curl/7.73.0-DEV" "090a1897-d696-99b1-90f0-2673113ba258" "httpbin:8000" "-" - - 172.17.0.7:80 172.17.0.6:33204 outbound_.8000_._.httpbin.foo.svc.cluster.local -
Add pprof endpoint to pilot-agent
Allow gRPC logging for pilot
--log_output_level=grpc:debug setting.
DestinationRule inheritance
PeerAuthentication and Sidecar Istio resources, you might know that there are mesh-level, namespace-level, and workload-level customizations for them. The settings are inherited from their "parents" by default, and the "child" CRs can override those settings.
The same architecture now applies for DestinationRules as well, support for DestinationRule inheritance from mesh/namespace level were added. The feature is off by default and can be enabled with the PILOT_ENABLE_DESTINATION_RULE_INHERITANCE environment variable in istiod.
Mirroring of images on gcr.io
gcr.io/istio-release as well.
Takeaway
Istio 1.9 mostly improved operational stability, which will likely help increase production adoption in the future.
Subscribe to
the Shift!
Get emerging insights on emerging technology straight to your inbox.
