Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
13 min read
Published on 11/24/2020
Last updated on 02/05/2024
What's new in Istio 1.8, a quick walkthrough
Share
Istio 1.8 has just been released and is one of the best Istio releases so far. The new version contains exciting experimental features, numerous enhancements, as well as deprecations and removals. The core focus of the release, however, is to increase operational stability. In this post, we'll review what's new in Istio 1.8, and highlight a few potential snags to look out for when upgrading to the latest version.
Previously, the Envoy inbound cluster name format looked like this:
In the past, Istio introduced a detection timeout for certain "server first" protocols, such as the protocol used by MySQL, which triggers when there are no initial bytes of traffic to sniff. This protocol detection timeout has been disabled by default to avoid potential traffic failures on slow connections or in certain cases increased latency.
If you have misconfigured "server first" protocols, this change will cause immediate connection timeouts.
To avoid issues involving "server first" protocols, make sure to explicitly specify the protocol for your service through one of the following methods:
Mixer has been the control plane component to which application sidecar proxies reported their telemetry data since the early days of Istio. Mixer had the potential to cause resource consumption and latency issues in larger environments, which is why a new Mixerless telemetry model was introduced:
The Istio sidecar proxy now provides native support for DNS resolution with ServiceEntries, which is why the Istio CoreDNS plugin is deprecated in Istio 1.8 and will be removed in a future release.
For the EnvoyFilter API, it is recommended that you use the new Envoy filter names, as some filter names were deprecated and will be removed in a future release.
DNS is essential for any cloud native application, but the way it works in Kubernetes results in a few challenges in Istio:
To register VMs for the Istio mesh, so far the only way was to manually create a WorkloadEntry resource. This was previously a manual step which registered new VMs, in contrast to how the same process was automated for nodes in a Kubernetes cluster. It could cause issues, for example, in case of auto scaling VMs, a circumstance in which it's is not easy to register new VMs automatically.
There is a pre-alpha feature that solves this problem, making it possible to auto register VMs. This is done by a new WorkloadGroup resource, which is responsible for automatically creating the
Beginning with Kubernetes 1.18, there is a CSR API feature, which automates the request and retrieval of certificates from a Certificate Authority (CA). In Istio 1.8, experimental support has been added to allow Istio to integrate with external CAs, using the Kubernetes CSR API.
This feature only lays the foundation necessary to work with external CAs, now only istiod is supported (and Google CA somewhat), but new implementations should come in future releases as this feature matures both on the Istio and Kubernetes side.
Previously, the gateways read the certificates for external communications from Kubernetes secrets. These gateways were usually public facing entities, and, if they were compromised, the certificates could be read from the Kubernetes secrets that compromised the whole cluster.
In Istio 1.8 gateways fetch the certs from Istiod. They do not even have the privileges they need to read secrets from the cluster to reduce potential collateral damage.
This feature is interesting because it paves the way for us to be able to fetch certificates from external secret stores (e.g. Vault or cloud key stores) in future releases, which would be convenient for many Istio users already using other secret stores. Furthermore, this is a totally backward compatible change, and does not require any user interaction.
To specify the order of HTTP_ROUTE specs for the Envoy configurations, new
A new
Almost two years ago now, when we open sourced the Banzai Cloud Istio operator, we did so partly because the only option for installing Istio was through Helm, which had its issues. After a while,
Since the 1.6 release, there has been an interesting alpha feature in Istio, in which various useful pieces of information about Istio resources are exposed in their
As WebAssembly (Wasm) gains more and more traction in Istio, it is also gradually becoming required in order to monitor it properly. Therefore, it can come handy to use the newly added Grafana dashboard on Wasm related metrics.
Previously, there were cases in which requests did not reach a destination proxy (e.g. fault injection, circuit breaking, no proxy injected etc.), and there were no destination labels to identify what would have been the end target of the request. This is remedied in 1.8, which is a welcome bug fix.
If you're looking for a production ready Istio distribution based on the new Istio 1.8 release and with support for a safe and automated canary > > upgrade flow, check out the new Backyards > > 1.5 release.
Istio 1.8 changes
Caveat: The Istio 1.8.0 release has a few known issues pointed out in the official change notes. Either make sure you won't be affected by them or wait a few patch releases before upgrading your cluster to make sure these issues are sorted out.First of all, it's important that we point out that the supported Kubernetes versions for Istio 1.8 are 1.16, 1.17, 1.18 and 1.19. If you are running Istio 1.7 in your environment, you should already be on at least Kubernetes 1.16 (as that is also the oldest supported K8s version for Istio 1.7). As a result, you should be ready to upgrade to Istio 1.8 without having to upgrade your K8s cluster. High impact changes (can cause issues when upgrading the mesh):
- Inbound cluster name format changed
- Protocol detection timeout disabled by default
- AuthorizationPolicy has new remoteIpBlocks/notRemoteIpBlocks fields
- Trust Domain Validation enforced by default
- Completely removes Mixer
- Removes support to install add-ons with istioctl
- Istio CoreDNS Plugin Deprecated
- Envoy filter names deprecated
- Smart DNS proxying
- Auto registration for VMs
- Integrates with external CAs with K8s CSR API
- Istiod to handle gateway certificates
- EnvoyFilter enhancements
- Debug archive option added
- Helm3 Istio installation
- Observed generation added for Istio resource statuses
- Wasm Grafana dashboards added
- Metrics properly labeled when destination not reached
High impact changes
Inbound cluster name format changed
inbound|<service_port_number>|<service_port_name>|<service_hostname>. An example cluster name with this format was: inbound|80|http|httpbin.default.svc.cluster.local.
For Istio 1.8.0 this format has changed, the service port name and the service's hostname are now omitted, the new format looks like this: inbound|<service_port_number>||. For the example above the new format is simply: inbound|80||
There were issues when multiple services selected the same container port in the same pod and this change is an attempt to solve those issues. The offical release note states that: "For most users, this is an implementation detail, and will only impact debugging or tooling that directly interacts with Envoy configuration."
Well, from what we've seen so far this change causes undesirable behaviour in the following scenario. When there are two services, which use the same service port number and select the same pod, but they are targeting different port numbers inside the pod. Here's an example when a backend and a frontend container are in the same pod:
apiVersion: v1
kind: Service
metadata:
name: backyards-web
namespace: backyards-system
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: http-web
selector:
app.kubernetes.io/name: backyards
type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
name: backyards
namespace: backyards-system
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: http-app
selector:
app.kubernetes.io/name: backyards
type: ClusterIPdocker.io/istio/pilot:1.8.0 docker image:
"pilot_duplicate_envoy_clusters": {
"inbound|80||": {
"proxy": "backyards-8cdfc77b5-4jw44.backyards-system",
"message": "Duplicate cluster inbound|80|| found while pushing CDS"
}
}backyards-web or the backyards service that in both cases the same target port will be called inside the pod. So it will be impossible to reach the other container port through its service, which is definitely undesired behaviour and seems to be a bug.
We're planning to open an issue about this with more details to sort out this issue upstream. A workaround for this problem can be to use different service port numbers to avoid such conflicts. What we did in Backyards is that we reverted this pilot change in our own docker image, which we use in Backyards (now Cisco Service Mesh Manager), so that this issue can never happen.
UPDATE: The upstream issue can be tracked here. Kudos to John Howard, who picked up the issue and fixed it very quickly. The patch should already be in the Istio 1.8.1 release.
Protocol detection timeout disabled by default
- By the name of the port:
name: <protocol>[-<suffix>]. - In Kubernetes 1.18+, by the
appProtocolfield:appProtocol: <protocol>.
kind: Service
metadata:
name: httpbin
spec:
ports:
- number: 443
name: https-web
- number: 3306
name: db
appProtocol: mysqlFor more info, please refer to Istio's documentation.
The
ipBlocks/notIpBlocks fields on the AuthorizationPolicy resource are now used to allow/deny requests based on the source address of IP packets that arrive at the sidecar.
If you'd like to allow/deny requests based on their original source IP addresses (either because you use the X-Forwarded-For header or a proxy protocol), then update your AuthorizationPolicy resources to use the new remoteIpBlocks/notRemoteIpBlocks fields instead of the ipBlocks/notIpBlocks fields.
Trust Domain Validation enforced by default
By default, the Trust Domain Validation feature will reject any incoming requests to sidecars, if the request is neither from the same trust domain nor in theTrustDomainAliases list.
If you want to allow traffic to your mesh from different trust domains on Istio 1.8, you need to add it to your TrustDomainAliases list, otherwise it will be rejected.
Removals/deprecations
Complete removal of Mixer
- first, as an experimental feature in Istio 1.3
- then in Istio 1.4 the feature was made available under the name Telemetry V2
- in Istio 1.5 Telemetry V2 was turned on by default
- in Istio 1.6 the original Mixer-based telemetry model was deprecated in favor of the new Telemetry V2 model
Remove support to install add-ons with istioctl
Open-source software that enhances Istio's capabilities (like Prometheus, Grafana, Zipkin, Jaeger, and Kiali) can no longer be installed withistioctl, and it is recommended that they be installed separately.
If you want the easiest way to install all these integrated components - with a production-ready setup and with many additional features - check out Backyards (now Cisco Service Mesh Manager).
Istio CoreDNS Plugin deprecated
Envoy filter names deprecated
Experimental features
Smart DNS proxying
- It can be difficult for VMs outside the Kubernetes cluster to provide DNS resolution to route services inside the cluster
- If services do not have unique virtual IPs and are on the same port (e.g. databases), then DNS servers cannot distinguish between them
- In multi-cluster meshes we cannot resolve the IP addresses of a service on another cluster (this is why stub services are likewise created on remote clusters)
Auto registration for VMs
WorkloadEntry resources when new VMs are available.
The Istio 1.8 release laid the groundwork for a wide variety of features around VM support, which should become apparent in future releases as it matures.
Integrate with external CAs with K8s CSR API
Under the radar changes
Istiod to handle gateway certificates
EnvoyFilter enhancements
INSERT_FIRST, INSERT_BEFORE, INSERT_AFTER operations were added to the EnvoyFilter API. To be able to override http and network filters, a new REPLACE operation was added to the EnvoyFilter API.
Debug archive option added
istioctl bug-report command was added to be able to get an archive of the Istio cluster - mainly for debugging purposes.
Other notable changes
Install Istio with Helm3
istioctl, and then the official Istio operator, were introduced, and, slowly, the method of installing by Helm became unsupported. It turned out, however, that there was still demand among Istio users for a way of installing and upgrading Istio with Helm (mostly for users who had already deployed all their apps with Helm), and because of that, Helm 3 support was added for Istio 1.8.
As a result, it is possible to install and upgrade Istio with Helm 3, but I would note here that this is NOT recommended. Only in-place upgrades are supported with Helm, whilst the canary upgrade model is the recommended flow today.
Observed generation added for Istio resource status
Status field. These fields include, but are not limited to resource readiness, the number of data plane instances associated with the resource, and validation messages.
In Istio 1.8 a new observed generation field is also present, which, when it matches the generation in the resource's metadata, indicates that all Istio updates have been completed. This is useful for detecting when requested changes to an Istio configuration have been served and are ready to receive traffic.
Wasm Grafana dashboards added
Metrics properly labeled when a destination is not reached
Takeaway
Istio 1.8 largely increases operational stability and also lays the foundation for exciting new features in future releases. If you’d like to kickstart your Istio experience, try out Backyards (now Cisco Service Mesh Manager), our Istio distribution. Backyards (now Cisco Service Mesh Manager) operationalizes the service mesh to bring deep observability, convenient management, and policy-based security for modern container-based applications.
Want to know more? Get in touch with us, or delve into the details of the latest release. Or just take a look at some of the Istio features that Backyards automates and simplifies for you, and which we've already blogged about.
Subscribe to
the Shift!
Get emerging insights on emerging technology straight to your inbox.
