Kubernetes Operator
SUSE Observability
Last updated
SUSE Observability
Last updated
Here is the setup we'll be creating, for an application that needs to be monitored:
The monitored application / workload running in cluster A, auto-instrumented by the operator
The Open Telemetry operator in cluster A
A collector created by the operator
SUSE Observability running in cluster B, or SUSE Cloud Observability
The Open Telemetry operator offers some extra features over the normal Kubernetes setup:
It can auto-instrument your application pods for supported languages (Java, .NET, Python, Golang, Node.js), without having to modify the applications or docker images at all
It can be dropped in as a replacement for the Prometheus operator and start scraping Prometheus exporter endpoints based on service and pod monitors
The operator is installed with a Helm chart, so first configure the chart repository.
Let's create a otel-operator.yaml
file to configure the operator:
Now install the collector, using the configuration file:
This only installs the operator. Continue to install the collector and enable auto-instrumentation.
Use the same cluster name as used for installing the SUSE Observability agent if you also use the SUSE Observability agent with the Kubernetes stackpack. Using a different cluster name will result in an empty traces perspective for Kubernetes components and will overall make correlating information much harder for SUSE Observability and your users.
Now apply this collector.yaml
in the open-telemetry
namespace to deploy a collector:
Now we need to tell the operator how to configure the auto instrumentation for the different languages using another custom resource, of kind Instrumentation
. It is mainly used to configure the collector that was just deployed as the telemetry endpoint for the instrumented applications.
It can be defined in a single place and used by all pods in the cluster, but it is also possible to have a different Instrumentation
in each namespace. We'll be doing the former here. Note that if you used a different namespace or a different name for the otel collector the endpoint in this file needs to be updated accordingly.
Create an instrumentation.yaml
:
Now apply the instrumentation.yaml
also in the open-telemetry
namespace:
To instruct the operator to auto-instrument your applicaction pods we need to add an annotation to the pod:
Java: instrumentation.opentelemetry.io/inject-java: open-telemetry/otel-instrumentation
NodeJS: instrumentation.opentelemetry.io/inject-nodejs: open-telemetry/otel-instrumentation
Python: instrumentation.opentelemetry.io/inject-python: open-telemetry/otel-instrumentation
Go: instrumentation.opentelemetry.io/inject-go: open-telemetry/otel-instrumentation
Note that the value of the annotation refers to the namespace and name of the Instrumentation
resource that we created. Other options are:
"true" - inject and Instrumentation
custom resource from the namespace.
"my-instrumentation" - name of Instrumentation
custom resource in the current namespace.
"my-other-namespace/my-instrumentation" - namespace and name of Instrumentation
custom resource in another namespace.
"false" - do not inject
When a pod with one of the annotations is created the operator modifies the pod via a mutation hook:
It adds an init container that provides the auto-instrumentation library
It modifies the first container of the pod to load the instrumentation during start up and it adds environment variables to configure the instrumentation
Go auto-instrumentation requires elevated permissions. These permissions are set automatically by the operator:
Go to SUSE Observability and make sure the Open Telemetry Stackpack is installed (via the main menu -> Stackpacks).
We'll install in the open-telemetry
namespace and use the receiver API key generated during installation (see where to find it):
The operator manages one or more collector deployments via a Kubernetes custom resource of kind OpenTelemetryCollector
. We'll create one using the same configuration as used in the .
It uses the secret created earlier in the guide. Make sure to replace <otlp-suse-observability-endpoint:port>
with your OTLP endpoint (see for your endpoint) and insert the name for your Kubernetes cluster instead of <your-cluster-name>
:
The collector offers a lot more configuration receivers, processors and exporters, for more details see our . For production usage often large amounts of spans are generated and you will want to start setting up .
If you need to customize which containers should be instrumented use the .
After a short while and if your pods are getting some traffic you should be able to find them under their service name in the Open Telemetry -> services and service instances overviews. Traces will appear in the and in the for the service and service instance components. Span metrics and language specific metrics (if available) will become available in the for the components.
If you also have the Kubernetes stackpack installed the instrumented pods will also have the traces available in the .
You can add new charts to components, for example the service or service instance, for your application, by following . It is also possible to create using the metrics and setup to get notified when your application is not available or having performance issues.
The operator, the OpenTelemetryCollector
, and the Instrumentation
custom resource, have more options that are documented in the . For example it is possible to install an optional via the OpenTelemetryCollector
resource, it can be used to configure the Prometheus receiver of the collector. This is especially useful when you want to replace Prometheus operator and are using its ServiceMonitor
and PodMonitor
custom resources.