LogoLogo
StackState.comDownloadSupportExplore playground
SUSE Observability
SUSE Observability
  • SUSE Observability docs!
  • Docs for all SUSE Observability products
  • 🚀Get started
    • Quick start guide
    • SUSE Observability walk-through
    • SUSE Rancher Prime
      • Air-gapped
      • Agent Air-gapped
    • SUSE Cloud Observability
  • 🦮Guided troubleshooting
    • What is guided troubleshooting?
    • YAML Configuration
    • Changes
    • Logs
  • 🚨Monitors and alerts
    • Monitors
    • Out of the box monitors for Kubernetes
    • Notifications
      • Configure notifications
      • Notification channels
        • Slack
        • Teams
        • Webhook
        • Opsgenie
      • Troubleshooting
    • Customize
      • Add a monitor using the CLI
      • Derived State monitor
      • Override monitor arguments
      • Write a remediation guide
  • 📈Metrics
    • Explore Metrics
    • Custom charts
      • Adding custom charts to components
      • Writing PromQL queries for representative charts
      • Troubleshooting custom charts
    • Advanced Metrics
      • Grafana Datasource
      • Prometheus remote_write
      • OpenMetrics
  • 📑Logs
    • Explore Logs
    • Log Shipping
  • 🔭Traces
    • Explore Traces
  • 📖Health
    • Health synchronization
    • Send health data over HTTP
      • Send health data
      • Repeat Snapshots JSON
      • Transactional Increments JSON
    • Debug health synchronization
  • 🔍Views
    • Kubernetes views
    • Custom views
    • Component views
    • Explore views
    • View structure
      • Overview perspective
      • Highlights perspective
      • Topology perspective
      • Events perspective
      • Metrics perspective
      • Traces perspective
      • Filters
      • Keyboard shortcuts
    • Timeline and time travel
  • 🕵️Agent
    • Network configuration
      • Proxy Configuration
    • Using a custom registry
    • Custom Secret Management
      • Custom Secret Management (Deprecated)
    • Request tracing
      • Certificates for sidecar injection
  • 🔭Open Telemetry
    • Overview
    • Getting started
      • Concepts
      • Kubernetes
      • Kubernetes Operator
      • Linux
      • AWS Lambda
    • Open telemetry collector
      • Sampling
      • SUSE Observability OTLP APIs
    • Instrumentation
      • Java
      • Node.js
        • Auto-instrumentation of Lambdas
      • .NET
      • SDK Exporter configuration
    • Troubleshooting
  • CLI
    • SUSE Observability CLI
  • 🚀Self-hosted setup
    • Install SUSE Observability
      • Requirements
      • Kubernetes / OpenShift
        • Kubernetes install
        • OpenShift install
        • Alibaba Cloud ACK install
        • Required Permissions
        • Override default configuration
        • Configure storage
        • Exposing SUSE Observability outside of the cluster
      • Initial run guide
      • Troubleshooting
        • Advanced Troubleshooting
        • Support Package (Logs)
    • Configure SUSE Observability
      • Slack notifications
      • E-mail notifications
      • Stackpacks
      • Advanced
        • Analytics
    • Release Notes
      • v2.0.0 - 11/Sep/2024
      • v2.0.1 - 18/Sep/2024
      • v2.0.2 - 01/Oct/2024
      • v2.1.0 - 29/Oct/2024
      • v2.2.0 - 09/Dec/2024
      • v2.2.1 - 10/Dec/2024
      • v2.3.0 - 30/Jan/2025
      • v2.3.1 - 17/Mar/2025
      • v2.3.2 - 22/Apr/2025
      • v2.3.3 - 07/May/2025
    • Upgrade SUSE Observability
      • Migration from StackState
      • Steps to upgrade
      • Version-specific upgrade instructions
    • Uninstall SUSE Observability
    • Air-gapped
      • SUSE Observability air-gapped
      • SUSE Observability Kubernetes Agent air-gapped
    • Data management
      • Backup and Restore
        • Kubernetes backup
        • Configuration backup
      • Data retention
      • Clear stored data
    • Security
      • Authentication
        • Authentication options
        • Single password
        • File-based
        • LDAP
        • Open ID Connect (OIDC)
          • Microsoft Entra ID
        • KeyCloak
        • Service tokens
        • Troubleshooting
      • RBAC
        • Role-based Access Control
        • Permissions
        • Roles
        • Scopes
      • Self-signed certificates
      • External secrets
  • 🔐Security
    • Service Tokens
    • API Keys
  • ☁️SaaS
    • User Management
  • Reference
    • SUSE Observability Query Language (STQL)
    • Chart units
    • Topology Identifiers
Powered by GitBook
LogoLogo

Legal notices

  • Privacy
  • Cookies
  • Responsible disclosure
  • SOC 2/SOC 3
On this page
  • Observability through load balancers, service meshes and between clusters
  • How does it work
  • Enabling the trace header injection sidecar
  • Disabling trace header injection
  • Overhead
  • Add the trace header id to an existing proxy
  • Add the trace header id in envoy
  • Istio
  • Add the trace header id with envoy filter
  • Instrument your application
  • Supported systems/technologies
  • Known Issues
  • No sidecar is injected for my pods
  1. Agent

Request tracing

SUSE Observability

PreviousCustom Secret Management (Deprecated)NextCertificates for sidecar injection

Last updated 7 months ago

Observability through load balancers, service meshes and between clusters

SUSE Observability can observe connections between services and pods in different Clusters, or when the connections go through a Service Mesh or Load Balancer. Observing these connections is done through request tracing. Traced requests will result in connections in the , to give insight in the dependencies across an application and help with finding the root cause of an incident.

How does it work

Request tracing is done by injecting a unique header (the X-Request-ID header) into all HTTP traffic. This unique header is observed at both client and server through an eBPF probe installed with the SUSE Observability Agent. These observations are sent to SUSE Observability, which uses the observations to understand which clients and server are connected.

The X-Request-Id headers are by a sidecar proxy that can be automatically injected by the SUSE Observability Agent. The sidecar gets injected by a , which injects the sidecar into every pod for which the http-header-injector.stackstate.io/inject: enabled annotation is defined. Sidecar injection is not supported on OpenShift.

It's also possible to add the X-Request-Id header if your application , is deployed to an enabled Kubernetes cluster or through . Advantage of this is that the extra sidecar proxy isn't needed.

Enabling the trace header injection sidecar

Enabling trace header injection is a two-step process:

  1. Install the mutating webhook into the cluster by adding --set httpHeaderInjectorWebhook.enabled=true to the helm upgrade invocation when installing the SUSE Observability agent. By default the sidecar injector generates its own self-signed certificate, requiring cluster roles to install these into the cluster. It is also possible to in a more restricted environment.

  2. For every pod that has a endpoint which processes http(s) requests, place the annotation http-header-injector.stackstate.io/inject: enabled to have the sidecar injected.

Enabling the mutating webhook will only take effect upon pod restart

If the annotation is placed before the webhook is installed. Installing the webhook has no effect until the pods get restarted.

Disabling trace header injection

Disabling the trace header injection can be done with the reverse process:

  1. Remove the http-header-injector.stackstate.io/inject: enabled annotation from all pods.

  2. Redeploy the SUSE Observability Agent without the --set httpHeaderInjectorWebhook.enabled=true setting.

Disabling the mutating webhook will only take effect upon pod restart

If step 1 is skipped and only the mutating webhook is disabled, all pods need a restart for the sidecar to be removed.

Overhead

Request tracing adds a small, fixed amount of CPU overhead for each HTTP request header that gets injected and observed. The exact amount is dependent on the system that it's ran on, so it's advised to enable this feature first in an acceptance environment to observe the impact before moving to production. The sidecar proxy takes a minimum of 25Mb of memory per pod it's deployed with, up to a maximum of 40Mb.

Add the trace header id to an existing proxy

To add the X-Request-Id header from an existing proxy, two properties are important:

  1. Each request/response pair has to get a unique ID.

  2. The X-Request-Id header should be added to both request and response, to be observed on both client and server.

Add the trace header id in envoy

Istio

Add the trace header id with envoy filter

Use kubectl to apply the following definition to the Kubernetes cluster,

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: responsed-x-request-id-always
  namespace: istio-system
spec:
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: ANY
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
      patch:
        operation: MERGE
        value:
          typed_config:
            '@type': >-
              type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            always_set_request_id_in_response: true
            generate_request_id: true
            preserve_external_request_id: true
  priority: 0

Instrument your application

It's also possible to add the X-Request-Id header form either the client side to each request, or on the server side to each response. It's important to ensure each request/response gets a unique X-Request-Id value. Also, the X-Request-Id requires that if an ID is already present in a request, the response should contain that same ID.

Supported systems/technologies

  • HTTP/1.0 and HTTP/1.1 with keepAlive

  • Trace header injection and trace observation on unencrypted traffic

  • Trace observation for OpenSSL Encrypted traffic

  • Trace header injection alongside LinkerD

  • Any LoadBalancer that forwards the X-Request-Id header in requests and responses

  • Any cross-cluster networking solution that forwards the X-Request-Id header in requests and responses

Known Issues

No sidecar is injected for my pods

To make sure you setup is ok, first validate the following steps were taken:

  • The --set httpHeaderInjectorWebhook.enabled=true flag was set during installation of the agent

  • The pod has http-header-injector.stackstate.io/inject: enabled set

  • The pod was restarted

If this does not resolve the issue, the following could be the issue:

Cluster networking policies

The cluster can have networking policies setup, preventing the kubernetes control-plane apiserver from contacting the mutatingvalidationwebhook which injects the sidecar. To validate this, look at the logs of the kube-apiserver, which is either in the kube-system namespace or could be managed by your cloud provider. An error like the following should be found in those logs:

Failed calling webhook, failing open stackstate-agent-http-header-injector-webhook.stackstate.io: failed calling webhook "stackstate-agent-http-header-injector-webhook.stackstate.io": failed to call webhook: Post "https://stackstate-agent-http-header-injector.monitoring.svc:8443/mutate?timeout=10s": context deadline exceeded

If this happens, be sure to adapt your cluster network policies such that the apiserver can reach the mutatingvalidationwebhook.

In envoy, the X-Request-Id header can be enabled by setting generate_request_id: true and always_set_request_id_in_response: true for

An can be used to set the trace header for Envoy.

🕵️
http connections
Envoy Filter
topology perspective
mutating webhook
manage your own certificates
injected
already has a proxy or LoadBalancer
Istio service mesh
instrumenting your own code