Splunk Health
StackState Self-hosted v5.1.x
Overview
When the Splunk StackPack has been installed in StackState, you can configure the Splunk Health check on StackState Agent V2 to begin collecting Splunk health data.
Health is collected from Splunk by executing Splunk saved searches from StackState Agent V2. In order to receive Splunk health data in StackState, configuration needs to be added to both Splunk and StackState Agent V2:
Splunk saved search - there should be at least one saved search that generates the health data you want to retrieve.
StackState Agent V2 Splunk Health check - a Splunk Health check should be configured to connect to your Splunk instance and execute the relevant Splunk saved searches.
The Splunk Health check on StackState Agent V2 will execute all configured Splunk saved searches periodically to retrieve a snapshot of the health at the current time.
Prerequisites
To run the Splunk Health Agent check, you need to have:
A running Splunk instance.
The Splunk StackPack installed on your StackState instance.
StackState Agent V2 must be installed on a single machine which can connect to Splunk and StackState.
Splunk saved search
Fields used
StackState Agent V2 executes the Splunk saved searches configured in the Splunk Health Agent check and pushes retrieved data to StackState. The following fields from the results of a saved search are sent to StackState:
check_state_id (string)
Required. The unique identifier for the check state.
name (string)
Required. Display name for the check state.
health (string)
Required. The health value of the check state. Can be CLEAR, DEVIATING or CRITICAL.
topology_element_identifier (string)
Required. The identifier of the component/relation this check state belongs to.
message (string)
Extended message associated with the check state, supports markdown.
Example Splunk query
Agent check
Configure the Splunk Health check
To enable the Splunk Health integration and begin collecting health data from your Splunk instance, the Splunk Health check must be configured on StackState Agent V2. The check configuration provides all details required for the Agent to connect to your Splunk instance and execute a Splunk saved search.
To configure the Splunk Health Agent check:
Edit the StackState Agent V2 configuration file
/etc/stackstate-agent/conf.d/splunk_health.d/conf.yaml.Under instances, add details of your Splunk instance:
url - The URL of your Splunk instance.
authentication - How the Agent should authenticate with your Splunk instance. Choose either token-based (recommended) or basic authentication. For details, see authentication configuration details.
ignore_saved_search_errors - Set to
falseto return an error if one of the configured saved searches doesn't exist. Defaulttrue.collection_interval - The interval at which the check is scheduled to run.
Under saved_searches, add details of each Splunk saved search that the check should execute to retrieve health information:
name - The name of the Splunk saved search to execute.
match - Regex used for selecting Splunk saved search queries. Default
"health.*".app - The Splunk app in which the saved searches are located. Default
"search".request_timeout_seconds - Default
10.search_max_retry_count - Default
5.search_seconds_between_retries - Default
1.batch_size - Default
1000.parameters - Used in the Splunk API request. The default parameters provided make sure the Splunk saved search query refreshes. Default
force_dispatch: trueanddispatch.now: true.
More advanced options can be found in the example configuration (github.com).
Save the configuration file.
Restart StackState Agent V2 to apply the configuration changes.
Incoming health data will be mapped to associated components and relations in the StackState UI as check states.
To more closely inspect what the synchronization is doing, use the StackState CLI
Disable the Agent check
To disable the Splunk Health Agent check:
Remove or rename the Agent integration configuration file, for example:
Restart StackState Agent V2 to apply the configuration changes.
See also
Last updated