Open telemetry collector
SUSE Observability
Last updated
SUSE Observability
SUSE Observability
Last updated
The OpenTelemetry Collector offers a vendor-agnostic implementation to receive, process and export telemetry data. Applications instrumented with Open Telemetry SDKs can use the collector to send telemetry data to SUSE Observability (traces and metrics).
Your applications, when set up with OpenTelemetry SDKs, can use the collector to send telemetry data, like traces and metrics, to SUSE Observability or another collector (for further processing). The collector is set up to receive this data by default via OTLP, the native open telemetry protocol. It can also receive data in other formats provided by other instrumentation SDKs like Jaeger and Zipkin for traces, and Influx and Prometheus for metrics.
The collector is running close to your application, in the same Kubernetes cluster, on the same virtual machine, etc. This allows SDKs to quickly offload data to the collector, which can then do transformations, batching and filtering. It can be used by multiple applications and allows for easy changes to your data processing pipeline.
For installation guides use the different . The getting started guides provide a basic collector configuration to get started, but over time you'll want to customize it to your needs and add additional receivers, processors, and exporters to customize your ingestion pipeline to your needs.
The collector configuration defines pipelines for processing the different telemetry signals. The components in the processing pipeline can be divided in several categories, and each component has its own configuration. Here we'll give an overview of the different configuration sections and how to use them.
Receivers accept telemetry data from instrumented applications, here via OTLP:
There are many more receivers that accept data via other protocols, for example Zipkin traces, or that actively collect data from various sources, for example:
Host metrics
Kubernetes metrics
Prometheus metrics (OpenMetrics)
Databases
The data from the receivers can be transformed or filtered by processors.
To send data to the SUSE Observability backend the collector has exporters. There are exporters for different protocols, push- or pull-based, and different backends. Using the OTLP protocols it is also possible to use another collector as a destination for additional processing.
For each telemetry signal a separate pipeline is configured. The pipelines are configured in the service.pipeline section and define which receivers, processors and exporters should be used in which order. Before using a component in the pipeline it must first be defined in its configuration section. The batch processor, for example, doesn't have any configuration but still has to be declared in the processors section. Components that are configured but are not included in a pipeline will not be active at all.
Extensions are not used directly in pipelines for processing data but extend the capabilities of the collector in other ways. For SUSE Observability it is used to configure the authentication using an api key. Extensions must be defined in a configuration section before they can be used. Similar to the pipeline components an extension is only active when it is enabled in the service.extensions section.
An example that only masks specific attributes and/or values:
The getting started guides show how to deploy the collector to Kubernetes or using Linux packages for a production ready setup. It is also possible to run it, for example for tests, directly as a docker container to try it out:
This uses the collector contrib image which includes all contributed components (receivers, processors, etc.). A smaller, more limited version of the image is also available, but it has only a very limited set of components available:
Note that the Kubernetes installation defaults to the Kubernetes distribution of the collector image, ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s, which has more components than the basic image, but less than the contrib image. If you run into missing components with that image you can simply switch it to use the contrib image , ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib, instead.
In some cases HTTP requests for telemetry data can become very large and may be refused by SUSE Observability. SUSE Observability has a limit of 4MB for the gRPC protocol. If you run into HTTP requests limits you can lower the requests size by changing the compression algorithm and limiting the maximum batch size.
To reduce the HTTP request size can be reduced by adding configuration to the batch processor limiting the batch size:
The Open Telemetry documentation provides much more details on the configuration and alternative installation options:
Open Telemetry Collector configuration: https://opentelemetry.io/docs/collector/configuration/
Kubernetes installation of the collector: https://opentelemetry.io/docs/kubernetes/helm/collector/
Using the Kubernetes operator instead of the collector Helm chart: https://opentelemetry.io/docs/kubernetes/operator/
Open Telemetry sampling: https://opentelemetry.io/blog/2022/tail-sampling/
Some receivers support all 3 signals (traces, metrics, logs), others support only 1 or 2, for example the Prometheus receiver can only collect metrics. The opentelemetry-collector-contrib repository has with documentation on their configuration.
The batch processor batches all 3 signals, improving compression and reducing the number of outgoing connections. The opentelemetry-collector-contrib repository has with documentation on their configuration.
The SUSE Observability exporter requires authentication using an api key, to configure that an is used. The opentelemetry-collector-contrib repository has with documentation on their configuration.
If the gRPC exporter doesn't work for you (see also ), you can switch to the, slightly less efficient, OTLP over HTTP protocol by using the otlphttp exporter instead. Replace all references to otlp/suse-observability with otlphttp/suse-observability (don't forget the references in the pipelines) and make sure to update the exporter config to:
The OTLP HTTP endpoint for SUSE Observability is different from the OTLP endpoint. Use the to find the correct URL.
The opentelemetry-collector-contrib repository has with documentation on their configuration.
There are many processors in the . Here we try to give an overview of commonly used processors and their capabilities. For more details and many more processors use the .
Some instrumentations or applications may generate a lot of telemetry data that is just noisy and unneeded for your use-case. The can be used to drop the data that you don't need in the collector, to avoid sending the data to SUSE Observability. For example to drop all the data of 1 specific service:
The filter processor uses the to define the filters.
The can change attributes of spans, logs or metrics.
The can modify attributes of a . For example to add a Kubernetes cluster name to every resource:
For changing metric names and other metric specific information there is also the .
The can be used to, for example, set a span status:
It supports many more transformations, like modifying the span name, converting metric types or modifying log events. See it's for all the possibilities. It uses the to define the filters.
The collector is the ideal place to remove or obfuscate sensitive data, because it sits right between your applications and SUSE Observability and has processors to . Next to the filtering and transformation capabilities already discussed there is also a available that can mask attribute values that match a block list. It can also remove attributes that don't match a specified list of allowed attributes, however using this can quickly result in dropping most attributes resulting in very limited observability capabilities. Note that it does not process resource attributes.
The getting started guides enable snappy compression on the collector, this is not the best compression but uses less CPU resources than gzip. If you removed the compression you can enable it again, or you can switch to a compression algorithm that offers a better . The same compression types are available for gRPC and HTTP protocols.
The batch size is defined in number of spans, metric data points, or log records (not in bytes), so you might need some experimentation to find the correct setting for your situation. For more details please refer to the .