Override monitor arguments

StackState Kubernetes Troubleshooting
Overview
StackState provides monitors out of the box, which provide monitoring on common issues that can occur in a Kubernetes cluster. Those monitors work with certain default arguments that suit most of the use cases but sometimes there's need to adapt its behaviour by overriding some of such default arguments like threshold or failureState. The mechanism to declare the overrides is via kubernetes resource annotations that denote to which monitor and component they should apply. For example we could override the failureState for the Available service endpoints monitor for a specific service where we want to signal a CRITICAL state when it fails rather than the default DEVIATING.
How to
​Build an override annotation​
​What monitors allow overriding arguments?​
​Build an override for a custom monitor​
As an example the steps will override the arguments for the Available service endpoints monitor of Kubernetes HTTP services.
How to build my annotation
The override annotations keys for StackState monitors follow the following convention:
monitor.${owner}.stackstate.io/${monitorShorName}
The owner property represents who created such a monitor, for the out of the box monitors is kubernetes-v2, and the monitorShorName property represents the id of the monitor and can be extracted from the identifier property of a monitor which can be read from the cli when listing or inspecting monitors
sts monitor list
​
ID              | STATUS  | IDENTIFIER                                                                          | NAME                                        | FUNCTION ID     | TAGS                                                                                  
8051105457030   | ENABLED | urn:stackpack:kubernetes-v2:shared:monitor:kubernetes-v2:service-available-endpoint | Available service endpoints                 | 233276809885571 | [services]         
In our example the identifier is urn:stackpack:kubernetes-v2:shared:monitor:kubernetes-v2:service-available-endpoint and the monitorShorName corresponds to the very last segment as in service-available-endpoint therefore the annotation key is:
monitor.kubernetes-v2.stackstate.io/service-available-endpoint
the annotation payload is a JSON object where the following optional arguments can be defined:
threshold: optional.A numeric threshold to compare against.
failureState: optional. Either "CRITICAL" or "DEVIATING". "CRITICAL" will show as read in StackState and "DEVIATING" as orange, to denote different severity.
enabled: optional. Boolean that determines if the monitor would produce a health state for that component.
The full annotation then would look like
    monitor.kubernetes-v2.stackstate.io/service-available-endpoint: |-
      {
        "threshold": 0.0,
        "failureState": "CRITICAL"
        "enabled": true
      }
What monitors allow overriding arguments?
​Available service endpoints​
​Node Disk Pressure​
​Node Memory Pressure​
​Node PID Pressure​
​Node Readiness​
​Out of memory for containers​
Build an override for a custom monitor
Any custom threshold monitor created using the guide at Add a threshold monitor to components using the CLI is suitable to override arguments, as the example shows an identifier for a custom monitor is structured as urn:custom:monitor:{monitorShortName}and the override annotation key for such an identifier is expected to be
monitor.custom.stackstate.io/${monitorShortName}
The example uses the identifier urn:custom:monitor:deployment-has-replicas therefore the annotation key would be
monitor.custom.stackstate.io/deployment-has-replicas
And the full annotation would look like
    monitor.custom.stackstate.io/deployment-has-replicas: |-
      {
        "threshold": 0.0,
        "failureState": "CRITICAL"
        "enabled": true
      }
Add a monitor using the CLI
Next - Metrics
Explore Metrics
Last modified 22d ago