Elasticsearch
StackState Self-hosted v5.1.x
Last updated
StackState v5.1
StackState Self-hosted v5.1.x
Last updated
StackState can be configured to pull data from your own Elasticsearch instance. The collected metrics or events data can then be added as a telemetry stream directly to a component or included as part of an integration, topology synchronisation or component/relation template.
To connect StackState to your Elasticsearch instance and retrieve telemetry data you will need to have:
A running Elasticsearch instance (version 7.3.x or earlier) reachable from StackState.
An Elasticsearch index to retrieve.
A time field with a timestamp in the Elasticsearch data.
An Elasticsearch data source should be added in StackState for each Elasticsearch index that you want to work with. Default settings that should work with most instances of Elasticsearch are already included, so you will only need to add details of your Elasticsearch instance and the index to be retrieved.
To add an Elasticsearch data source:
In the StackState UI, go to Settings > Telemetry Sources > Elasticsearch sources.
Click ADD ELASTICSEARCH DATA SOURCE.
Enter the required settings:
Name - the name to identify the Elasticsearch data source in StackState.
Base URL - the URL of the REST API of your Elasticsearch instance (default port 9200). Note that this must be reachable by StackState.
Index pattern - the Elasticsearch index to retrieve. It's possible to specify a pattern if the index is sliced by time. See .
Time zone - the timezone of the timestamps stored in the Elasticsearch documents. Required to ensure data is correctly processed by StackState.
Time field - the field in the Elasticsearch documents with the timestamp of the event/metric.
Time field format - the format of the value in the specified Time field. See .
A number of additional settings can be tweaked in non-standard use-cases. See the .
Click TEST CONNECTION to confirm that StackState can connect to Elasticsearch at the configured Base URL.
Click CREATE to save the Elasticsearch data source settings.
The new Elasticsearch data source will be listed on the Elasticsearch sources page and available as a data source when adding telemetry to components and relations.
You can find the index pattern used in an Elasticsearch instance with the command curl <elasticsearch_node>:<port>/_cat/indices?v.
In the example below, the index pattern would be [sts_internal_events-]yyyy.MM.dd.
The Time field format specified in the StackState Elasticsearch data source settings should fit the date format used in the Elasticsearch index that will be retrieved. Follow the instructions below to find the Elasticsearch index date format and then use this to identify the correct time field format.
Find the date format used in the Elasticsearch index
You can find the date format used in a specific Elasticsearch index with the command curl <elasticsearch_node>:<port>/<index_name>/_mapping?pretty. This returns the index mapping, which includes the format of any date values.
In the example below, the date format would be date.
Find the correct time field format
yyyy-MM-dd for the date format date.
yyyy-MM-dd’T’HH:mm:ss[.SSS]ZZZZZ for a timestamp pattern with millis that can have 1, 2 or 3 digits.
To reduce noise resulting from intermittent failure of the data source, StackState can be configured to only emit errors after they exist for a specified time.
Propagate errors only after (minutes) - the time after which errors should be reported.
Elasticsearch can optionally be accessed through Kibana. To do this, the following settings must be provided:
Base URL - the URL of the REST API endpoint provided by Kibana. Note that this must be reachable by StackState.
Kibana version - the Kibana version used, required for XSS mitigation.
The timeout settings can be tweaked when dealing with exceptionally large result sets or a slower Elasticsearch cluster:
Read timeout (seconds) - the timeout when retrieving data directly from Elasticsearch.
Request timeout ui (seconds) - the timeout when retrieving Elasticsearch data through the StackState API.
Wildcards can be slow in Elasticsearch. By default, StackState will escape wildcards used in telemetry query values. If required, this behavior can be disabled.
Support wildcards in values - when enabled, StackState will include wildcards (*) in telemetry query values. Note that wildcards can be used to filter for partial matches, it isn't possible to use a full wildcard. At least one other character must be specified in the filter value.
To find the correct time field format to specify in the StackState Elasticsearch data source settings, look up the date format from your Elasticsearch index in the . You may need to adjust the format provided to fit your implementation, for example:
Elasticsearch data sources can be used to add telemetry streams to components and relations in StackState. This can be done manually by directly to an element or as part of an integration or topology synchronization, for details see how to .
