LogoLogo
StackState.comDownloadSupportExplore playground
SUSE Observability
SUSE Observability
  • SUSE Observability docs!
  • Docs for all SUSE Observability products
  • 🚀Get started
    • Quick start guide
    • SUSE Observability walk-through
    • SUSE Rancher Prime
      • Air-gapped
      • Agent Air-gapped
    • SUSE Cloud Observability
  • 🦮Guided troubleshooting
    • What is guided troubleshooting?
    • YAML Configuration
    • Changes
    • Logs
  • 🚨Monitors and alerts
    • Monitors
    • Out of the box monitors for Kubernetes
    • Notifications
      • Configure notifications
      • Notification channels
        • Slack
        • Teams
        • Webhook
        • Opsgenie
      • Troubleshooting
    • Customize
      • Add a monitor using the CLI
      • Derived State monitor
      • Override monitor arguments
      • Write a remediation guide
  • 📈Metrics
    • Explore Metrics
    • Custom charts
      • Adding custom charts to components
      • Writing PromQL queries for representative charts
      • Troubleshooting custom charts
    • Advanced Metrics
      • Grafana Datasource
      • Prometheus remote_write
      • OpenMetrics
  • 📑Logs
    • Explore Logs
    • Log Shipping
  • 🔭Traces
    • Explore Traces
  • 📖Health
    • Health synchronization
    • Send health data over HTTP
      • Send health data
      • Repeat Snapshots JSON
      • Transactional Increments JSON
    • Debug health synchronization
  • 🔍Views
    • Kubernetes views
    • Custom views
    • Component views
    • Explore views
    • View structure
      • Overview perspective
      • Highlights perspective
      • Topology perspective
      • Events perspective
      • Metrics perspective
      • Traces perspective
      • Filters
      • Keyboard shortcuts
    • Timeline and time travel
  • 🕵️Agent
    • Network configuration
      • Proxy Configuration
    • Using a custom registry
    • Custom Secret Management
      • Custom Secret Management (Deprecated)
    • Request tracing
      • Certificates for sidecar injection
  • 🔭Open Telemetry
    • Overview
    • Getting started
      • Concepts
      • Kubernetes
      • Kubernetes Operator
      • Linux
      • AWS Lambda
    • Open telemetry collector
      • Sampling
      • SUSE Observability OTLP APIs
    • Instrumentation
      • Java
      • Node.js
        • Auto-instrumentation of Lambdas
      • .NET
      • SDK Exporter configuration
    • Troubleshooting
  • CLI
    • SUSE Observability CLI
  • 🚀Self-hosted setup
    • Install SUSE Observability
      • Requirements
      • Kubernetes / OpenShift
        • Kubernetes install
        • OpenShift install
        • Alibaba Cloud ACK install
        • Required Permissions
        • Override default configuration
        • Configure storage
        • Exposing SUSE Observability outside of the cluster
      • Initial run guide
      • Troubleshooting
        • Advanced Troubleshooting
        • Support Package (Logs)
    • Configure SUSE Observability
      • Slack notifications
      • E-mail notifications
      • Stackpacks
      • Advanced
        • Analytics
    • Release Notes
      • v2.0.0 - 11/Sep/2024
      • v2.0.1 - 18/Sep/2024
      • v2.0.2 - 01/Oct/2024
      • v2.1.0 - 29/Oct/2024
      • v2.2.0 - 09/Dec/2024
      • v2.2.1 - 10/Dec/2024
      • v2.3.0 - 30/Jan/2025
      • v2.3.1 - 17/Mar/2025
      • v2.3.2 - 22/Apr/2025
      • v2.3.3 - 07/May/2025
    • Upgrade SUSE Observability
      • Migration from StackState
      • Steps to upgrade
      • Version-specific upgrade instructions
    • Uninstall SUSE Observability
    • Air-gapped
      • SUSE Observability air-gapped
      • SUSE Observability Kubernetes Agent air-gapped
    • Data management
      • Backup and Restore
        • Kubernetes backup
        • Configuration backup
      • Data retention
      • Clear stored data
    • Security
      • Authentication
        • Authentication options
        • Single password
        • File-based
        • LDAP
        • Open ID Connect (OIDC)
          • Microsoft Entra ID
        • KeyCloak
        • Service tokens
        • Troubleshooting
      • RBAC
        • Role-based Access Control
        • Permissions
        • Roles
        • Scopes
      • Self-signed certificates
      • External secrets
  • 🔐Security
    • Service Tokens
    • API Keys
  • ☁️SaaS
    • User Management
  • Reference
    • SUSE Observability Query Language (STQL)
    • Chart units
    • Topology Identifiers
Powered by GitBook
LogoLogo

Legal notices

  • Privacy
  • Cookies
  • Responsible disclosure
  • SOC 2/SOC 3
On this page
  • Overview
  • How to
  • How to build my annotation
  • What monitors allow overriding arguments?
  • Build an override for a custom monitor
  1. Monitors and alerts
  2. Customize

Override monitor arguments

SUSE Observability

PreviousDerived State monitorNextWrite a remediation guide

Last updated 7 months ago

Overview

SUSE Observability provides , which provide monitoring on common issues that can occur in a Kubernetes cluster. Those monitors work with certain default arguments that suit most of the use cases but sometimes there's need to adapt its behaviour by overriding some of such default arguments like threshold or failureState. The mechanism to declare the overrides is via kubernetes resource annotations that denote to which monitor and component they should apply. For example we could override the failureState for the Available service endpoints monitor for a specific service where we want to signal a CRITICAL state when it fails rather than the default DEVIATING.

How to

As an example the steps will override the arguments for the Available service endpoints monitor of Kubernetes HTTP services.

How to build my annotation

The override annotations keys for SUSE Observability monitors follow the following convention:

monitor.${owner}.stackstate.io/${monitorShorName}

The owner property represents who created such a monitor, for the out of the box monitors is kubernetes-v2, and the monitorShorName property represents the id of the monitor and can be extracted from the identifier property of a monitor which can be read from the cli when listing or inspecting monitors

sts monitor list

ID              | STATUS  | IDENTIFIER                                                                          | NAME                                        | FUNCTION ID     | TAGS                                                                                  
8051105457030   | ENABLED | urn:stackpack:kubernetes-v2:shared:monitor:kubernetes-v2:service-available-endpoint | Available service endpoints                 | 233276809885571 | [services]         

In our example the identifier is urn:stackpack:kubernetes-v2:shared:monitor:kubernetes-v2:service-available-endpoint and the monitorShorName corresponds to the very last segment as in service-available-endpoint therefore the annotation key is:

monitor.kubernetes-v2.stackstate.io/service-available-endpoint

the annotation payload is a JSON object where the following optional arguments can be defined:

  • threshold: optional.A numeric threshold to compare against.

  • failureState: optional. Either "CRITICAL" or "DEVIATING". "CRITICAL" will show as read in SUSE Observability and "DEVIATING" as orange, to denote different severity.

  • enabled: optional. Boolean that determines if the monitor would produce a health state for that component.

The full annotation then would look like

    monitor.kubernetes-v2.stackstate.io/service-available-endpoint: |-
      {
        "threshold": 0.0,
        "failureState": "CRITICAL",
        "enabled": true
      }

What monitors allow overriding arguments?

Build an override for a custom monitor

monitor.custom.stackstate.io/${monitorShortName}

The example uses the identifier urn:custom:monitor:deployment-has-replicas therefore the annotation key would be

monitor.custom.stackstate.io/deployment-has-replicas

And the full annotation would look like

    monitor.custom.stackstate.io/deployment-has-replicas: |-
      {
        "threshold": 0.0,
        "failureState": "CRITICAL"
        "enabled": true
      }

Any custom threshold monitor created using the guide at is suitable to override arguments, as an identifier for a custom monitor is structured as urn:custom:monitor:{monitorShortName}and the override annotation key for such an identifier is expected to be

🚨
monitors out of the box
Build an override annotation
What monitors allow overriding arguments?
Build an override for a custom monitor
Add a threshold monitor to components using the CLI
Available service endpoints
Cpu limits resourcequota
Cpu requests resourcequota
Memory limits resourcequota
Memory requests resourcequota
Node Disk Pressure
Node Memory Pressure
Node PID Pressure
Node Readiness
Out of memory for containers
the example shows