Links

Override monitor arguments

StackState Kubernetes Troubleshooting

Overview

StackState provides monitors out of the box, which provide monitoring on common issues that can occur in a Kubernetes cluster. Those monitors work with certain default arguments that suit most of the use cases but sometimes there's need to adapt its behaviour by overriding some of such default arguments like threshold or failureState. The mechanism to declare the overrides is via kubernetes resource annotations that denote to which monitor and component they should apply. For example we could override the failureState for the Available service endpoints monitor for a specific service where we want to signal a CRITICAL state when it fails rather than the default DEVIATING.

How to

As an example the steps will override the arguments for the Available service endpoints monitor of Kubernetes HTTP services.

How to build my annotation

The override annotations keys for StackState monitors follow the following convention:
monitor.${owner}.stackstate.io/${monitorShorName}
The owner property represents who created such a monitor, for the out of the box monitors is kubernetes-v2, and the monitorShorName property represents the id of the monitor and can be extracted from the identifier property of a monitor which can be read from the cli when listing or inspecting monitors
sts monitor list
ID | STATUS | IDENTIFIER | NAME | FUNCTION ID | TAGS
8051105457030 | ENABLED | urn:stackpack:kubernetes-v2:shared:monitor:kubernetes-v2:service-available-endpoint | Available service endpoints | 233276809885571 | [services]
In our example the identifier is urn:stackpack:kubernetes-v2:shared:monitor:kubernetes-v2:service-available-endpoint and the monitorShorName corresponds to the very last segment as in service-available-endpoint therefore the annotation key is:
monitor.kubernetes-v2.stackstate.io/service-available-endpoint
the annotation payload is a JSON object where the following optional arguments can be defined:
  • threshold: optional.A numeric threshold to compare against.
  • failureState: optional. Either "CRITICAL" or "DEVIATING". "CRITICAL" will show as read in StackState and "DEVIATING" as orange, to denote different severity.
  • enabled: optional. Boolean that determines if the monitor would produce a health state for that component.
The full annotation then would look like
monitor.kubernetes-v2.stackstate.io/service-available-endpoint: |-
{
"threshold": 0.0,
"failureState": "CRITICAL"
"enabled": true
}

What monitors allow overriding arguments?

Build an override for a custom monitor

Any custom threshold monitor created using the guide at Add a threshold monitor to components using the CLI is suitable to override arguments, as the example shows an identifier for a custom monitor is structured as urn:custom:monitor:{monitorShortName}and the override annotation key for such an identifier is expected to be
monitor.custom.stackstate.io/${monitorShortName}
The example uses the identifier urn:custom:monitor:deployment-has-replicas therefore the annotation key would be
monitor.custom.stackstate.io/deployment-has-replicas
And the full annotation would look like
monitor.custom.stackstate.io/deployment-has-replicas: |-
{
"threshold": 0.0,
"failureState": "CRITICAL"
"enabled": true
}