Splunk Health
StackState Self-hosted v5.1.x
Last updated
StackState v5.1
StackState Self-hosted v5.1.x
Last updated
When the has been installed in StackState, you can configure the Splunk Health check on StackState Agent V2 to begin collecting Splunk health data.
Health is collected from Splunk by executing Splunk saved searches from StackState Agent V2. In order to receive Splunk health data in StackState, configuration needs to be added to both Splunk and StackState Agent V2:
- there should be at least one saved search that generates the health data you want to retrieve.
- a Splunk Health check should be configured to connect to your Splunk instance and execute the relevant Splunk saved searches.
The Splunk Health check on StackState Agent V2 will execute all configured Splunk saved searches periodically to retrieve a snapshot of the health at the current time.
To run the Splunk Health Agent check, you need to have:
A running Splunk instance.
The installed on your StackState instance.
must be installed on a single machine which can connect to Splunk and StackState.
StackState Agent V2 executes the Splunk saved searches configured in the and pushes retrieved data to StackState. The following fields from the results of a saved search are sent to StackState:
check_state_id (string)
Required. The unique identifier for the check state.
name (string)
Required. Display name for the check state.
health (string)
Required. The health value of the check state. Can be CLEAR, DEVIATING or CRITICAL.
topology_element_identifier (string)
Required. The identifier of the component/relation this check state belongs to.
message (string)
Extended message associated with the check state, supports markdown.
To enable the Splunk Health integration and begin collecting health data from your Splunk instance, the Splunk Health check must be configured on StackState Agent V2. The check configuration provides all details required for the Agent to connect to your Splunk instance and execute a Splunk saved search.
To configure the Splunk Health Agent check:
Edit the StackState Agent V2 configuration file /etc/stackstate-agent/conf.d/splunk_health.d/conf.yaml.
Under instances, add details of your Splunk instance:
url - The URL of your Splunk instance.
ignore_saved_search_errors - Set to false to return an error if one of the configured saved searches doesn't exist. Default true.
collection_interval - The interval at which the check is scheduled to run.
Under saved_searches, add details of each Splunk saved search that the check should execute to retrieve health information:
match - Regex used for selecting Splunk saved search queries. Default "health.*".
app - The Splunk app in which the saved searches are located. Default "search".
request_timeout_seconds - Default 10.
search_max_retry_count - Default 5.
search_seconds_between_retries - Default 1.
batch_size - Default 1000.
parameters - Used in the Splunk API request. The default parameters provided make sure the Splunk saved search query refreshes. Default force_dispatch: true and dispatch.now: true.
Save the configuration file.
Restart StackState Agent V2 to apply the configuration changes.
Incoming health data will be mapped to associated components and relations in the StackState UI as check states.
To disable the Splunk Health Agent check:
Remove or rename the Agent integration configuration file, for example:
Restart StackState Agent V2 to apply the configuration changes.
Example Agent V2 Splunk Health check configuration file:
authentication - How the Agent should authenticate with your Splunk instance. Choose either token-based (recommended) or basic authentication. For details, see .
name - The name of the to execute.
More advanced options can be found in the .
To more closely inspect what the synchronization is doing,
The configured collection_interval will be used as the . Make sure that the value set for the collection_interval matches the time that the check will take to run.