How to develop Agent checks
StackState Self-hosted v5.0.x
This page describes StackState version 5.0.
This document covers how to create your first check with Agent v2 Check API. Following topics are covered in this document: the agent directory structure, configuring your check, writing your first check, sending topology, metrics, events, and service checks as well as how to add external python dependencies and putting it all together.
Installing Agent v2 StackPack
To install this StackPack navigate to StackState’s StackPacks page using left menu and locate the “StackState Agent V2” section. Click the Agent V2 icon and this opens the installation page. Click the INSTALL button and follow installation instructions provided by the StackPack.
Directory Structure
Before starting your first check it is worth understanding the checks directory structure. There are two places that you will need to add files for your check. The first is the checks.d
folder, which lives in your Agent root.
For all Linux systems, you should find it at:
For Windows Server >= 2008 you should find it at:
The configuration folder is conf.d
which also lives in your Agent root.
For Linux, you should find it at:
For Windows, you should find it at:
You can also add additional checks to a single directory, and point to it in StackState.yaml
:
For the remainder of this document these paths will be referred to as checks.d
and conf.d
.
Check Configuration
Each check has a configuration directory and file that will be placed in the conf.d
directory. Configuration is written using YAML. The folder name should match the name of the check (e.g.: example.py
and example.d
containing the conf.yaml
configuration file). We will be using the StackState "Skeleton" / bare essentials check and configuration as a starting point.
The configuration file for the "Skeleton" check has the following structure:
YAML files must use spaces instead of tabs.
init_config
The init_config section allows you to have an arbitrary number of global configuration options that will be available on every run of the check in self.init_config
.
instances
The instances section is a list of instances that this check will be run against. Your check(...)
method is run once per instance each collection interval. This means that every check will support multiple instances out of the box. A check instance is an object that should contain all configuration items needed to monitor a specific instance. An instance is passed into the execution of the check
method in the instance
parameter. collection_interval
can be added to define how often the check should be run. If the value is set to 30, it means that this check will be scheduled for collection every 30 seconds. If the check runtime exceeds the collection_interval
, the Agent will warn in the Agent log with Check <name> did not finish execution with the defined collection_interval time '<time>', skipping execution...
. The default is 40
, if no collection_interval
is specified. The collection_interval
setting has superseded the min_collection_interval
setting that was used before. The Agent will still accept the min_collection_interval
setting and interpret it as if the collection_interval
was specified.
To synchronize multiple instances in StackState you have to create a multi-tenant StackPack.
➡️ Learn more about developing StackPacks..
Setting up your check configuration
Create a directory named {your_check_name}.d
inside the conf.d
directory. The YAML configuration below can be used as a starting point for your check configuration. Save it to a file called conf.yaml
in the conf.d/{your_check_name}.d/
directory.
First Check
Now you can start defining your first check. The following "Skeleton" check can be used as a good starting point:
Setting up your check
You can now take the "Skeleton" Check snippet given above and save inside a file named: {your_check_name}.py
in the checks.d
directory.
Load values from the instance config
Values can be loaded from the instance config object in the following ways:
StackState Topology Snapshots
Components and relations can be sent as part of a snapshot. A snapshot represents the total state of some external topology. By putting components and relations in a snapshot, StackState will persist all the topology elements present in the snapshot, and remove everything else for the topology instance. Creating snapshots is facilitated by two functions, these are already in place in the StackState "Skeleton" check:
self.start_snapshot()
- used to start a snapshot. Internally, theAgentCheck
interface uses theget_instance_key
function to uniquely identify this topology instance.self.stop_snapshot()
- used to stop a snapshot. This should be done at the end of the check, after all data has been submitted.
Disable snapshots
You can disable snapshots by setting with_snapshots=False
when you define a TopologyInstance
. This replaces the start
/stop_snapshot()
invocations described above.
Send Topology
Topology data can be submitted using the self.component()
and self.relation()
functions in the AgentCheck
interface. The example below shows how to submit two components with a relation between them:
This creates two components in StackState. One for the host named this-host
and one for an application named some-application
. The domain
value is used in the horizontal grouping of the components in StackState and layer
is used for vertical grouping. The labels
, tags
and environment
add some metadata to the component and can also be used for filtering in StackState. An IS_HOSTED_ON
relation is created between some-application
and this-host
. The labels
and tags
fields can also be used on relations to add some metadata. The component types (Host
, Application
) and relation type (IS_HOSTED_ON
) will be automatically created in StackState and can later be used in the synchronization to create mappings for the different types.
The identifiers and the external identifier, for example some-application-unique-identifier
will be used as the StackState Id. The external identifer
should be unique within this integration.
Merge Identifiers
The identifiers
are used within StackState to merge components across different integrations. Components with a matching identifier will be merged within StackState.
Given the following example:
These two components will be merged into a single component called this-host
containing data from both integrations.
➡️ Learn more about the Agent Check Topology API
Send Metrics
The StackState Agent Check interface supports various types of metrics.
Metric data can be submitted using i.e. the self.gauge()
function, or the self.count()
function in the AgentCheck
interface. All metrics data is stored in the StackSate Metrics
data source that can be mapped to a metric telemetry stream for a component/relation in StackState:
![](https://docs.stackstate.com/~gitbook/image?url=https%3A%2F%2F2504486602-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F7R5SkC7htnAgui7G9EXA%252Fuploads%252Fgit-blob-0b86fa8214f94b8862f87adb01267f48bab14674%252Fv50_metricstelemetrystream.png%3Falt%3Dmedia&width=768&dpr=4&quality=100&sign=52e1f990&sv=1)
The example below submits a gauge metric system.cpu.usage
for our previously submitted this-host
component:
Note: It is important to have a tag or combination of tags that you can use to uniquely identify this metric and map it to the corresponding component within StackState.
➡️ Learn more about the Agent Check Metric API
Send Events
Events can be submitted using the self.event()
function in the AgentCheck
interface. Events data is stored in the StackState Generic Events
data source that can be mapped to a telemetry stream on a component in StackState:
![](https://docs.stackstate.com/~gitbook/image?url=https%3A%2F%2F2504486602-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F7R5SkC7htnAgui7G9EXA%252Fuploads%252Fgit-blob-f374b93a8cbee65d8937be831c662ae1d9b3319b%252Fv50_genericevents.png%3Falt%3Dmedia&width=768&dpr=4&quality=100&sign=5dc27370&sv=1)
The example below submits an event to StackState when a call to the instance that is monitored exceeds some configured timeout:
➡️ Learn more about the Agent Check Event API
Health Synchronization
StackState can ingest check states from external monitoring systems. Before getting started, you can read up on the core concepts of health Synchronization.
To setup a health synchronization stream within a check, the following function needs to be defined:
This function specifies what health synchronization stream this agent check will produce to. Having this function defined will enable the self.health
api on the Agent Check.
Health Synchronization Snapshots
Like with topology, health data is presented to StackState using snapshots. This allows for removal of health information (check states) when they no longer exist in the source monitoring system. Snapshots are created by two functions:
self.health.start_snapshot()
- used to start a health snapshot.self.health.stop_snapshot()
- used to stop the snapshot, signaling that all submitted data is complete. This should be done at the end of the check, after all data has been submitted. If exceptions occur in the check, or for some other reason not all data can be produced, this function should not be called.Send Check State
Check states can be sent through the health synchronization API using the self.health.check_state()
functions in the AgentCheck
interface.
In the example below, a check state is created in StackState with the health value CRITICAL.
The check is attached to the component or relation matching the
topology_element_identifier
.The
check_state_id
is used to distinguish check states within the current health stream.
Check states can be send through the health synchronization api, using the self.health.check_state()
functions in the AgentCheck
interface. The example below shows how to submit the data:
If after following the previous steps you health data does not show in StackState, there is a troubleshooting guide for the health synchronization.
Send in Stream Definitions and Health Checks
Stream Definitions and Health Checks for StackState can be sent in with Topology. This allows you to map telemetry streams with health checks onto components in your integration, leaving no work to be done in StackState. This example below sets up a metric stream called Host CPU Usage
with a Maximum Average
check in StackState on the this-host
component.
We create a MetricStream
on the system.cpu.usage
metric with some conditions specific to our component. We then create a maximum_average
check on our metric stream using this_host_cpu_usage.identifier
. The stream and check are then added to the streams and checks list in our this-host
component.
➡️ Learn more about the Agent Check Telemetry API
Send Service Checks
Service Checks are used to submit the state of the agent integration instance. Service checks can be submitted using the self.service_check
method in the AgentCheck
interface. Service check data is also stored in the StackState Generic Events
data source.
The example below submits a service check to StackState when it is verified that the check was configured correctly and it can communicate with the instance that is monitored:
The service check can produce the following states:
AgentCheck.OK
AgentCheck.WARNING
AgentCheck.CRITICAL
AgentCheck.UNKNOWN
➡️ Learn more about the Agent Check Service Check API
Add Python Dependencies
Sometimes your check may require some external dependencies. To solve this problem the StackState Agent is shipped with python and pip embedded. When installing the dependencies needed by your custom check you should use the embedded pip to do so. This executable for pip can be found here:
For Linux, you should find it at:
For Windows, you should find it at:
Test your Check
Custom Agent checks need to be called by the Agent. To test this, run:
For Linux:
For Windows:
This executes your check once and displays the results.
Once you are happy with the result of your check, you can restart the StackState Agent service and your check will be scheduled alongside the other agent checks.
For Linux:
For Windows:
Troubleshooting
To troubleshoot any issues in your custom check, you can run the following status command and find your check in the output:
For Linux:
For Windows:
If your issue continues, contact Support with the help page that lists the paths it installs.
See also
Last updated