LogoLogo
StackState.comDownloadSupportExplore playground
StackState v6.0
StackState v6.0
  • StackState docs!
  • Docs for all StackState products
  • 🚀Get started
    • Quick start guide
    • StackState walk-through
    • SUSE Rancher Prime
      • Air-gapped
      • Agent Air-gapped
  • 🦮Guided troubleshooting
    • What is guided troubleshooting?
    • YAML Configuration
    • Changes
    • Logs
  • 🚨Monitors and alerts
    • Monitors
    • Out of the box monitors for Kubernetes
    • Notifications
      • Configure notifications
      • Notification channels
        • Slack
        • Teams
        • Webhook
        • Opsgenie
      • Troubleshooting
    • Customize
      • Add a monitor using the CLI
      • Override monitor arguments
      • Write a remediation guide
  • 📈Metrics
    • Explore Metrics
    • Custom charts
      • Adding custom charts to components
      • Writing PromQL queries for representative charts
      • Troubleshooting custom charts
    • Advanced Metrics
      • Grafana Datasource
      • Prometheus remote_write
      • OpenMetrics
  • 📑Logs
    • Explore Logs
    • Log Shipping
  • 🔭Traces
    • Explore Traces
  • 📖Health
    • Health synchronization
    • Send health data over HTTP
      • Send health data
      • Repeat Snapshots JSON
      • Repeat States JSON
      • Transactional Increments JSON
    • Debug health synchronization
  • 🔍Views
    • Kubernetes views
    • Custom views
    • Component views
    • Explore views
    • View structure
      • Filters
      • Overview perspective
      • Highlights perspective
      • Topology perspective
      • Events perspective
      • Metrics perspective
      • Traces perspective
    • Timeline and time travel
  • 🕵️Agent
    • Network configuration
      • Proxy Configuration
    • Using a custom registry
    • Custom Secret Management
    • Request tracing
      • Certificates for sidecar injection
  • 🔭Open Telemetry
    • Getting started
    • Open telemetry collector
    • Languages
      • Generic Exporter configuration
      • Java
      • Node.js
      • .NET
      • Verify the results
    • Troubleshooting
  • CLI
    • StackState CLI
  • 🚀Self-hosted setup
    • Install StackState
      • Requirements
      • Kubernetes / OpenShift
        • Kubernetes install
        • OpenShift install
        • Required Permissions
        • Non-high availability setup
        • Small profile setup
        • Override default configuration
        • Configure storage
        • Exposing StackState outside of the cluster
      • Initial run guide
      • Troubleshooting
        • Logs
    • Configure StackState
      • Slack notifications
      • Stackpacks
    • Release Notes
      • v1.11.0 - 18/07/2024
      • v1.11.3 - 15/08/2024
      • v1.11.4 - 29/08/2024
      • v1.12.0 - 24/10/2024
      • v1.12.1 - 08/11/2024
    • Upgrade StackState
      • Steps to upgrade
      • Version-specific upgrade instructions
    • Uninstall StackState
    • Air-gapped
      • StackState air-gapped
      • StackState Kubernetes Agent air-gapped
    • Data management
      • Backup and Restore
        • Kubernetes backup
        • Configuration backup
      • Data retention
      • Clear stored data
    • Security
      • Authentication
        • Authentication options
        • File-based
        • LDAP
        • Open ID Connect (OIDC)
        • KeyCloak
        • Service tokens
      • RBAC
        • Role-based Access Control
        • Permissions
        • Roles
        • Scopes
      • Self-signed certificates
  • 🔐Security
    • Service Tokens
    • Ingestion API Keys
  • ☁️SaaS
    • User Management
  • Reference
    • StackState Query Language (STQL)
    • Chart units
Powered by GitBook
LogoLogo

Legal notices

  • Privacy
  • Cookies
  • Responsible disclosure
  • SOC 2/SOC 3
On this page
  • Overview
  • How to
  • How to build my annotation
  • What monitors allow overriding arguments?
  • Build an override for a custom monitor
  1. Monitors and alerts
  2. Customize

Override monitor arguments

StackState v6.0

PreviousAdd a monitor using the CLINextWrite a remediation guide

Last updated 10 months ago

Overview

StackState provides , which provide monitoring on common issues that can occur in a Kubernetes cluster. Those monitors work with certain default arguments that suit most of the use cases but sometimes there's need to adapt its behaviour by overriding some of such default arguments like threshold or failureState. The mechanism to declare the overrides is via kubernetes resource annotations that denote to which monitor and component they should apply. For example we could override the failureState for the Available service endpoints monitor for a specific service where we want to signal a CRITICAL state when it fails rather than the default DEVIATING.

How to

As an example the steps will override the arguments for the Available service endpoints monitor of Kubernetes HTTP services.

How to build my annotation

The override annotations keys for StackState monitors follow the following convention:

monitor.${owner}.stackstate.io/${monitorShorName}

The owner property represents who created such a monitor, for the out of the box monitors is kubernetes-v2, and the monitorShorName property represents the id of the monitor and can be extracted from the identifier property of a monitor which can be read from the cli when listing or inspecting monitors

sts monitor list

ID              | STATUS  | IDENTIFIER                                                                          | NAME                                        | FUNCTION ID     | TAGS                                                                                  
8051105457030   | ENABLED | urn:stackpack:kubernetes-v2:shared:monitor:kubernetes-v2:service-available-endpoint | Available service endpoints                 | 233276809885571 | [services]         

In our example the identifier is urn:stackpack:kubernetes-v2:shared:monitor:kubernetes-v2:service-available-endpoint and the monitorShorName corresponds to the very last segment as in service-available-endpoint therefore the annotation key is:

monitor.kubernetes-v2.stackstate.io/service-available-endpoint

the annotation payload is a JSON object where the following optional arguments can be defined:

  • threshold: optional.A numeric threshold to compare against.

  • failureState: optional. Either "CRITICAL" or "DEVIATING". "CRITICAL" will show as read in StackState and "DEVIATING" as orange, to denote different severity.

  • enabled: optional. Boolean that determines if the monitor would produce a health state for that component.

The full annotation then would look like

    monitor.kubernetes-v2.stackstate.io/service-available-endpoint: |-
      {
        "threshold": 0.0,
        "failureState": "CRITICAL",
        "enabled": true
      }

What monitors allow overriding arguments?

Build an override for a custom monitor

monitor.custom.stackstate.io/${monitorShortName}

The example uses the identifier urn:custom:monitor:deployment-has-replicas therefore the annotation key would be

monitor.custom.stackstate.io/deployment-has-replicas

And the full annotation would look like

    monitor.custom.stackstate.io/deployment-has-replicas: |-
      {
        "threshold": 0.0,
        "failureState": "CRITICAL"
        "enabled": true
      }

Any custom threshold monitor created using the guide at is suitable to override arguments, as an identifier for a custom monitor is structured as urn:custom:monitor:{monitorShortName}and the override annotation key for such an identifier is expected to be

🚨
monitors out of the box
Build an override annotation
What monitors allow overriding arguments?
Build an override for a custom monitor
Add a threshold monitor to components using the CLI
Available service endpoints
Cpu limits resourcequota
Cpu requests resourcequota
Memory limits resourcequota
Memory requests resourcequota
Node Disk Pressure
Node Memory Pressure
Node PID Pressure
Node Readiness
Out of memory for containers
the example shows