Elasticsearch

Retrieve data from a custom Elasticsearch data source

This page describes StackState version 4.2.

The StackState 4.2 version range is End of Life (EOL) and no longer supported. We encourage customers still running the 4.2 version range to upgrade to a more recent release.

Go to the documentation for the latest StackState release.

Overview

StackState can be configured to pull data from your own Elasticsearch instance. The collected metrics or events data can then be added as a telemetry stream directly to a component or included as part of an integration, topology synchronisation or component/relation template.

Pull telemetry from a custom Elasticsearch instance

Prerequisites

To connect StackState to your Elasticsearch instance and retrieve telemetry data you will need to have:

• A running Elasticsearch instance reachable from StackState.

• An Elasticsearch index to retrieve.

• A time field with a timestamp in the Elasticsearch data.

Add an Elasticsearch data source to StackState

An Elasticsearch data source should be added in StackState for each Elasticsearch index that you want to work with. Default settings that should work with most instances of Elasticsearch are already included, so you will only need to add details of your Elasticsearch instance and the index to be retrieved.

To add an Elasticsearch data source:

1. In the StackState UI, go to Settings > Telemetry Sources > Elasticsearch sources.

2. Click on ADD ELASTICSEARCH DATA SOURCE.

3. Enter the required settings:

   • Name - the name to identify the Elasticsearch data source in StackState.

   • Base URL - the URL of the REST API of your Elasticsearch instance (default port 9200). Note that this must be reachable by StackState.

   • Index pattern - the Elasticsearch index to retrieve. It is possible to specify a pattern if the index is sliced by time. See how to find the Elasticsearch index pattern.

   • Time zone - the timezone of the timestamps stored in the elasticsearch documents. This is required to ensure data is correctly processed by StackState.

   • Time field - the field in the Elasticsearch documents that contains the timestamp of the event/metric.

   • Time field format - the format of the value in the specified Time field.

   • A number of additional settings can be tweaked in non-standard use-cases. See the advanced settings.

4. Click TEST CONNECTION to confirm that StackState can connect to Elasticsearch at the configured Base URL.

5. Click CREATE to save the Elasticsearch data source settings.

   • The new Elasticsearch data source will be listed on the Elasticsearch sources page and available as a data source when adding telemetry to components and relations.

Add Elasticsearch data source

Find the Elasticsearch index pattern

You can find the index pattern used in an Elasticsearch instance with the command curl <elasticsearch_node>:<port>/_cat/indices?v.

In the example below, the index pattern would be [sts_internal_events-]yyyy.MM.dd.

curl localhost:9200/_cat/indices?v

> health status index                          uuid                   pri rep docs.count docs.deleted store.size pri.store.size
> green  open   sts_internal_events-2020.10.04 KhhFdcOFR5StiWC3zAMEJg   1   0      48262            0     18.6mb         18.6mb
> green  open   sts_internal_events-2020.10.05 SiycHLkORrGe0tCUWbby2w   1   0      47456            0     18.7mb         18.7mb
> green  open   sts_internal_events-2020.10.06 INYPKojcSMWnyyjMQvTEow   1   0      40890            0     16.4mb         16.4mb
> green  open   sts_internal_events-2020.10.07 AagKIOInRaetkeQF8TO_rA   1   0      47125            0     18.3mb         18.3mb

Work with Elasticsearch data in StackState

Elasticsearch data sources can be used to add telemetry streams to components and relations in StackState. This can be done manually by adding a telemetry stream directly to an element or as part of an integration or topology synchronization, for details see how to add telemetry during topology synchronization.

Advanced settings

Error management

To reduce noise resulting from intermittent failure of the data source, StackState can be configured to only emit errors after they exist for a specified time.

• Propagate errors only after (minutes) - the time after which errors should be reported.

Kibana

Elasticsearch can optionally be accessed through Kibana. To do this, the following settings must be provided:

• Base URL - the URL of the REST API endpoint provided by Kibana. Note that this must be reachable by StackState.

• Kibana version - the Kibana version used. This is required for XSS mitigation.

Timeouts

The timeout settings can be tweaked when dealing with exceptionally large result sets or a slower Elasticsearch cluster:

• Read timeout (seconds) - the timeout when retrieving data directly from Elasticsearch.

• Request timeout ui (seconds) - the timeout when retrieving Elasticsearch data through the StackState API.

Wildcards

Wildcards can be slow in Elasticsearch. By default, StackState will escape wildcards used in telemetry query values. If required, this behavior can be disabled.

• Support wildcards in values - when enabled, StackState will include wildcards (*, ?) in telemetry query values.

See also

• Add a telemetry stream to a component or relation

• Add telemetry during topology synchronization