Published on 00/00/0000
Last updated on 00/00/0000
Published on 00/00/0000
Last updated on 00/00/0000
Share
COLLABORATIONS
13 min read
Published on 03/08/2023
Last updated on 06/18/2024
Getting Started with OpenTelemetry: KinD, Jaeger and the Spring PetClinic Application
Share
In the Getting Started Series opener, I referenced several links to help you get up to speed on the architecture of OpenTelemetry. If you didn't get to check that blog out, here it is: https://techblog.cisco.com/blog/opentelemetry-getting-started-series
Goal
This post provides a quick-start guide for deploying the basic components of OpenTelemetry (OTel), a way to interact with the tracing output (via Jaeger) and generate traces using a sample Spring PetClinic service that has OTel auto-instrumentation enabled.
This document does not go into a granular explanation of the OTel, Jaeger, and Spring Framework components. We will break the OTel components down into detail in future blogs.
Purpose
This document provides how-to steps to:
- Deploy a KinD Kubernetes cluster
- Deploy an OpenTelemetry (OTel) Collector
- Deploy a Jaeger All-in-One environment
- Deploy a sample application using the Spring PetClinic
Deployment Overview
Figure 1 illustrates the basic Kubernetes services, deployments, and pods used in this setup. Note: Not all components are shown, including the Jaeger and OTel operators.
Figure 1. Kubernetes Resource Overview
As shown in Figure 1, the Jaeger all-in-one deployment includes multiple services that represent Jaeger components such as the query, collector, agent (not shown), and headless collector (not shown). These components are managed via the Jaeger operator. In addition, the OTel Operator manages the various OTel collector components, which include the collector service and pod. Finally, the Spring PetClinic sample service leverages the OTel auto-instrumentation library for Java (discussed later).
Deployment Steps
- Create a KinD Cluster: (https://kind.sigs.k8s.io/docs/user/quick-start/)
- Deploy Cert Manager: (https://cert-manager.io/docs/installation/)
- Deploy the Jaeger Operator: (https://github.com/jaegertracing/jaeger-operator)
- Deploy the Jaeger all-in-one model
- Deploy the OTel Operator: (https://github.com/open-telemetry/opentelemetry-operator)
- Deploy the OTel Collector and aim the exporter towards the Jaeger all-in-one collector service.
- Deploy the OTel auto-instrumentation CRD and set the endpoint as the OTel collector service.
- Deploy the Spring PetClinic sample service and enable sidecar injection of the OTel Java auto-instrumentation library.
- Check for service tracing of the sample service.
Looks easy, right? Let's jump into the details.
Deployment Walk-thru
Create a KinD cluster (see the link above to install KinD on your machine):
kind create clusterDeploy Cert Manager (always check for the latest release):
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yamlDeploy the Jaeger Operator (always check for the latest release):
kubectl create namespace observability
kubectl apply -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.42.0/jaeger-operator.yaml -n observabilityDeploy the Jaeger All-in-One Strategy:
kubectl apply -f - <<EOF
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simplest
EOFIn another terminal session, port forward to the Jaeger simplest-query service on port 16686:
kubectl port-forward svc/simplest-query 16686:16686Deploy the OTel Operator:
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yamlDeploy the OTel Collector:
In this setup, the specific exporter configuration uses the previously deployed Jaeger collector and the endpoint connection uses the service name ("simplest-collector") and port of 14250. Ensure that the "insecure" flag is true. Additionally, the pipeline definition uses OTLP (OpenTelemetry Protocol) as the receiver protocol and Jaeger as the exporter.
kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel
spec:
config: |
receivers:
otlp:
protocols:
grpc:
http:
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
batch:
send_batch_size: 10000
timeout: 10s
exporters:
logging:
jaeger:
endpoint: "simplest-collector:14250"
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: []
exporters: [jaeger]
EOFDeploy the OTel Java Auto-instrumentation CRD:
The auto-instrumentation configuration points the library towards the previously deployed OTel collector service (otel-collector) on port 4317 and references the image locations for each language type.
kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
spec:
exporter:
endpoint: http://otel-collector:4317
propagators:
- tracecontext
- baggage
- b3
sampler:
type: parentbased_traceidratio
argument: "0.25"
java:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
nodejs:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
python:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
EOFDeploy the Spring PetClinic Sample Service:
In this example, the Spring PetClinic sample application deployment uses the auto-instrumentation configuration to perform a sidecar injection of the OTel Java library. In future blog posts, we will walk through other methods of instrumenting an application.
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: spring-petclinic
spec:
selector:
matchLabels:
app: spring-petclinic
replicas: 1
template:
metadata:
labels:
app: spring-petclinic
annotations:
sidecar.opentelemetry.io/inject: "true"
instrumentation.opentelemetry.io/inject-java: "true"
spec:
containers:
- name: app
image: ghcr.io/pavolloffay/spring-petclinic:latest
EOFWe will get into how manual and auto-instrumentation works in a future blog post, but let's take a quick look at how auto-instrumentation works in this example.
In the annotation shown above, we have indicated that we are using the sidecar method to spin up an init container when the spring-petclinic pod(s) are deployed and we want to inject the Java auto-instrumentation code into the spring-petclinic container. So, let's look at the pod with the two containers and what the injection is doing to the spring-petclinic container.
Find the name of your spring-petclinic pod and run a 'kubectl describe' on it. I have removed a bunch of extra output to focus on the two containers we want to discuss.
The first thing to note is that the annotations from our config above appear in the pod annotation section, and then we see two containers listed along with some environment data. The first container, the auto-instrumentation init container, is used to copy the '/otel-auto-instrumentation/javaagent.jar' file into the 'main body' container (in our case, the spring-petclinic container) and then run as a 'javaagent' (more on that in a second).
In the Environment section, there is the JAVA_TOOL_OPTIONS line that refers to the jar file that is copied and then executed in the spring-petclinic container. The other line to note is the OTEL_EXPORTER_OTLP_ENDPOINT line. That line indicates where the spring-petclinic container and the javaagent library will send the trace data (over OTLP). That endpoint is the OTel collector we deployed in a previous step.
# kubectl describe pod spring-petclinic-79f4794dd9-mjbfn
Name: spring-petclinic-79f4794dd9-mjbfn
Namespace: default
Priority: 0
Service Account: default
Node: kind-control-plane/172.18.0.2
Start Time: Tue, 28 Feb 2023 10:28:11 -0500
Labels: app=spring-petclinic
pod-template-hash=79f4794dd9
Annotations: instrumentation.opentelemetry.io/inject-java: true
sidecar.opentelemetry.io/inject: true
Init Containers:
opentelemetry-auto-instrumentation:
Container ID: containerd://310fbcf762dcb592f84330a7a9583174852a6abcdad505a9e4baa3cc78963ede
Image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
Containers:
app:
Container ID: containerd://32e0056ea02bb749654aeb310fb59bf46055fdc94470f5a54592e5a59b0bf55a
Image: ghcr.io/pavolloffay/spring-petclinic:latest
Environment:
JAVA_TOOL_OPTIONS: -javaagent:/otel-auto-instrumentation/javaagent.jar
OTEL_SERVICE_NAME: spring-petclinic
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317
OTEL_RESOURCE_ATTRIBUTES_POD_NAME: spring-petclinic-79f4794dd9-mjbfn (v1:metadata.name)
OTEL_RESOURCE_ATTRIBUTES_NODE_NAME: (v1:spec.nodeName)
OTEL_PROPAGATORS: tracecontext,baggage,b3
OTEL_TRACES_SAMPLER: parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG: 0.25
OTEL_RESOURCE_ATTRIBUTES: k8s.container.name=app,k8s.deployment.name=spring-petclinic,k8s.namespace.name=default,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),k8s.replicaset.name=spring-petclinic-79f4794dd9Now, let's look at the log for the spring-petclinic container and see what the init container did. Use the 'kubectl logs' command to view the logs for the spring-petclinic container.
The first line shows the javaagent running the auto-instrumentation code in the spring-petclinic container. The process copies the jar file and executes it according to the information in the Environment section above. After that, the usual business logic of the petclinic runs.
# kubectl logs spring-petclinic-79f4794dd9-mjbfn
Picked up JAVA_TOOL_OPTIONS: -javaagent:/otel-auto-instrumentation/javaagent.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2023-02-28 15:28:34:943 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.23.0
|\ _,,,--,,_
/,`.-'`' ._ \-;;,_
_______ __|,4- ) )_ .;.(__`'-'__ ___ __ _ ___ _______
| | '---''(_/._)-'(_\_) | | | | | | | | |
| _ | ___|_ _| | | | | |_| | | | __ _ _
| |_| | |___ | | | | | | | | | | \ \ \ \
| ___| ___| | | | _| |___| | _ | | _| \ \ \ \
| | | |___ | | | |_| | | | | | | |_ ) ) ) )
|___| |_______| |___| |_______|_______|___|_| |__|___|_______| / / / /
==================================================================/_/_/_/
:: Built with Spring Boot :: 2.5.4In an new terminal session, port forward to the Spring PetClinic deployment on port 8080:
kubectl port-forward deployment.apps/spring-petclinic 8080:8080Verification:
Drive this section on your own. You can't break anything, so have a blast!
Note: Before starting this section, make sure both of your port forwarding sessions are still running (8080 and 16686).
Verify that the Spring PetClinic service works by opening a browser to http://localhost:8080. Click through various links in the sample service to generate traffic and traces.
Verify that the Jaeger all-in-one deployment works by opening a browser to http://localhost:16686. After a few seconds (you may need to refresh the browser page), the Jaeger UI will show two services in the "Search" panel on the left. Select the "spring-petclinic" service and then click "Find Traces" at the bottom of the Search panel. If you have yet to use the PetClinic UI, there may only be basic single span results in the search. Click around the PetClinic UI to generate more spans.
Click through the spans and expand the "Tags" and "Process" sections to see the details of each span.
Walkthru of Jaeger and Spring PetClinic UIs:
In this section, we will look at the PetClinic UI and then dig into a couple of areas of the Jaeger UI to visualize some trace elements - the trace ID and span ID.
In the Spring PetClinic UI (http://localhost:8080/), click on the "Find Owners" tab at the top as shown in Figure 2.
Figure 2. Spring PetClinic - Find Owners
In the Jaeger UI (http://localhost:16686/), select "spring-petclinic" in the "service" field and click "Find Traces". A trace result should show something similar to "spring-petclinic: GET /owners" as shown in Figure 3. Click on the trace result.
Figure 3. Jaeger - Get Owners
After clicking on the trace result of GET /owners, you will see the trace detail view as shown in Figure 4. This view shows you the trace and span summary for GET /owners. This is an excellent view for quickly visualizing the call flow of the trace and each span that makes up the trace. You can also see the per-span timeline, which is great for quickly determining if a specific span has an issue that is causing high latency.
Figure 4. Jaeger - Trace Detail View
Click on one of the spans to get detailed information about the process in that span. For example, as shown in Figure 5, the span detail view shows all kinds of information about OpenTelemetry, the code function, and even the infrastructure environmental information such as the Kubernetes node info (node, pod, operating system).
Figure 5. Jaeger - Span Detail
Now, check out the logs of the OTel collector to verify its connection to Jaeger.
Verify that the OTLP receiver is started and the connection from OTel to the Jaeger is ready:
kubectl logs deployment.apps/otel-collectorThe output should show "Receiver started":
builder/receivers_builder.go:73 Receiver started.<strong> </strong>{"kind": "receiver", "name": "otlp"}And the backend is "READY":
jaegerexporter@v0.41.0/exporter.go:186 State of the connection with the Jaeger Collector backend {"kind": "exporter", "name": "jaeger", "state": "READY"}
Figure 6 illustrates the connection flow for each of the major components (Jaeger, OTel Collector and Spring Pet Clinic service).
Figure 6. Connection Flow
On the Jaeger All-in-One pod, check the connections to the query service (16686) and the OTel Collector -to- Jaeger connection (14250). Note: Output of netstat command has been summarized. There is an incoming connection from the local browser to the port forwarded service on 16686. There is also and incoming connection from the OTel Collector on port 14250.
# kubectl exec -it simplest-797dd8fc67-6l9q4 -- netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:47280 localhost:16686 ESTABLISHED
tcp 0 0 simplest-797dd8fc67-6l9q4:14250 10-244-0-11.otel-collector.default.svc.cluster.local:49768 ESTABLISHED
I am a day-in-the-life of a connection type of guy, so, lets verify the connection between the Spring PetClinic pod and the OTel Collector. Note: The default image does not include any network connection tools (e.g., netstat, lsof, etc.). Connect to a shell on the Spring PetClinic pod and install net-tools:
# kubectl exec -it spring-petclinic-79f4794dd9-mjbfn -- /bin/bash
apt update
apt install net-tools -yCheck the connections. The two primary connections to look for are the connections from the browser to the PetClinic UI (8080) and the connection from the Java OTel library process to the OTel Collector (4317):
root@spring-petclinic-79f4794dd9-mjbfn:/# netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:55028 localhost:8080 ESTABLISHED
tcp 0 0 10.244.0.12:34208 otel-collector.def:4317 ESTABLISHEDThat's it! You did it! You are now on your way to being an OpenTelemetry guru.
In future blogs, we will tear apart some of what we glossed over and explain how some of these components work, how to modify their configurations for specific use cases, and how to mix in different OTel SDKs (node.js, python, etc.). We will also talk about exporting traces, metrics, and logs to different back-ends and much more!
Shannon McFarland is a Distinguished Engineer and open-source advocate in Cisco’s Emerging Technology & Incubation organization. You can follow him on Twitter @eyepv6.
Subscribe to
the Shift!
Get emerging insights on emerging technology straight to your inbox.
