Send telemetry data over HTTP

StackState Self-hosted v4.6.x

This page describes StackState version 4.6.

Go to the documentation for the latest StackState release.

Overview

StackState can either pull telemetry from a data source or can receive pushed telemetry. Pushed telemetry is stored by StackState, while pulled telemetry is not. Pushed telemetry is stored for the duration of the configured retention period. This page describes how telemetry can be pushed.

There are several ways to send telemetry to StackState. A large number of integrations are provided out of the box that may help you get started. If there is no out of the box integration you can send telemetry to StackState using either HTTP or the StackState CLI.

StackState Receiver API

The StackState Receiver API accepts topology, telemetry and health data in a common JSON object. By default, the receiver API is hosted at:

https://<baseUrl>/receiver/stsAgent/intake?api_key=<API_KEY>

Both the baseUrl and API_KEY are set during StackState installation, for details see Kubernetes install - configuration parameters.

https://<baseUrl>:<receiverPort>/stsAgent/intake?api_key=<API_KEY>

Both the baseUrl and API_KEY are set during StackState installation, for details see Linux install - configuration parameters.

Common JSON object

Topology, telemetry and health data are sent to the receiver API via HTTP POST. There is a common JSON object used for all messages. One message can contain multiple metrics and multiple events.

{
  "collection_timestamp": 1548855554, // the epoch timestamp for the collection
  "events": {}, // see the section on "events", below
  "internalHostname": "localdocker.test", // the host that is sending this data
  "metrics": [], // see the section on "metrics", below
  "service_checks": [],
  "topologies": [], // used for sending topology data
  "health" // used for sending health data
}

Depending on your StackState configuration, received metrics or events that are too old will be ignored.

Metrics

Metrics can be sent to the StackState receiver API using the "metrics" property of the common JSON object.

JSON property: "metrics"

[
  "test.metric", // the metric name
  1548857152,
  10.0, // double - value of the metric
  {
    "hostname": "localdocker.test",
    "tags": [ 
      "tag_key1:tag_value1",
      "tag_key2:tag_value2"
    ],
    "type": "gauge"
  }
]

Every metric has the following details:

name - The metric name. Must not start with any of the following prefixes: host, labels, name, tags , timeReceived, timestamp, tags or values.
timestamp - The epoch timestamp of the metric.
value - The value of the metric.
hostname - The host this metric is from.
type - The type of metric. Can be gauge, count, rate, counter or raw.
tags - Optional. A list of key/value tags to associate with the metric.

The timestamp and value are used to plot the metric as a time series. The name and tags can be used to define a metric stream in StackState.

Send metrics to StackState

Multiple metrics can be sent in one JSON message via HTTP POST. For example:

curl -X POST \
 'http://<stackstateURL>/stsAgent/intake?api_key=<API_KEY>' \
 -H 'Content-Type: application/json' \
 -d '{
  "collection_timestamp": 1548857167,
  "events": {},
  "internalHostname": "localdocker.test",
  "metrics": [
    [
      "test.metric",
      1548857152,
      10.0,
      {
        "hostname": "localdocker.test",
        "tags": [
          "tag_key1:tag_value1",
          "tag_key2:tag_value2"
        ],
        "type": "gauge"
      }
    ],
    [
      "test.metric",
      1548857167,
      10.0,
      {
        "hostname": "localdocker.test",
        "tags": [
          "tag_key1:tag_value1",
          "tag_key2:tag_value2"
        ],
        "type": "gauge"
      }
    ]
  ],
  "service_checks": [],
  "topologies": []
}'

You can also send metrics to StackState using the StackState CLI metric send command.

Events

Events can be sent to the StackState receiver API using the "events" property of the common JSON object.

All events in StackState relate to a topology element or elements. Any properties of an event can be used to define a log stream in StackState.

JSON property: "events"

"event.test": [ // The event name
  {
    "context": {
      "category": "Changes",
      "data": { 
        "data_key1":"data_value1",
        "data_key2":"data_value2"
      },
      "element_identifiers": [
        "element_identifier1",
        "element_identifier2"
      ],
      "source": "source_system",
      "source_links": [
        {
          "title": "link_title",
          "url": "link_url"
        }
      ]
    },
    "event_type": "event_typeEvent",
    "msg_title": "event_title",
    "msg_text": "event_text",
    "source_type_name": "source_event_type",
    "tags": [
      "tag_key1:tag_value1",
      "tag_key2:tag_value2",
    ],
    "timestamp": 1607432944
  }
]

Events have the following details:

An event name. this must not start with any of the following prefixes: eventType, host, labels, message, name, tags, timeReceived, timestamp or title.
context - Optional. Includes details of the source system for an event. Events that contain a context will be visible in the StackState Events Perspective for views that contain a component with a matching source identifier. Events without a context will be available in StackState as a log stream:
- category - The event category. Can be Activities, Alerts, Anomalies, Changes or Others.
- element_identifiers - The identifiers for the topology element(s) the event relates to. These are used to bind the event to a topology element or elements.
- source - The name of the system from which the event originates, for example AWS, Kubernetes or JIRA.
- data - Optional. A list of key/value details about the event, for example a configuration version.
- source_identifier - Optional. The original identifier of the event in the source system.
- source_links - Optional. A list of links related to the event, for example a dashboard or the event in the source system.
event_type - Describes the event being sent. This should generally end with the suffix Event, for example ConfigurationChangedEvent, VersionChangedEvent.
msg_text - Required. The text body of the event.
msg_title - Required. The title of the event.
source_type_name - Optional. The source event type name.
tags - Optional. A list of key/value tags to associate with the event.
timestamp - The epoch timestamp for the event.

Send events to StackState

Multiple events can be sent in one JSON message via HTTP POST. You can also send a single event to StackState using the StackState CLI event send command. For example:

curl -X POST \
 'http://<stackstateURL>/stsAgent/intake?api_key=<API_KEY>' \
 -H 'Content-Type: application/json' \
 -d '{
  "collection_timestamp": 1548857342,
  "events": {
    "event.test01": [ // The event name
      {
        "context": {
          "category": "Changes",
          "data": { 
            "data_key1":"data_value1",
            "data_key2":"data_value2"
          },
          "element_identifiers": [
            "element_identifier1",
            "element_identifier2"
          ],
          "source": "source_system",
          "source_links": [
            {
              "title": "link_title",
              "url": "link_url"
            }
          ]
        },
        "event_type": "HealthStateChangedEvent",
        "msg_title": "event_title",
        "msg_text": "event_text",
        "source_type_name": "source_event_type",
        "tags": [
          "tag_key1:tag_value1",
          "tag_key2:tag_value2",
        ],
        "timestamp": 1607432944
      }
    ],
    "event.test02": [ // The event name
      {
        "context": {
          "category": "Changes",
          "data": { 
            "data_key1":"data_value1",
            "data_key2":"data_value2"
          },
          "element_identifiers": [
            "element_identifier1",
            "element_identifier2"
          ],
          "source": "source_system",
          "source_links": [
            {
              "title": "link_title",
              "url": "link_url"
            }
          ]
        },
        "event_type": "HealthStateChangedEvent",
        "msg_title": "event_title",
        "msg_text": "event_text",
        "source_type_name": "source_event_type",
        "tags": [
          "tag_key1:tag_value1",
          "tag_key2:tag_value2",
        ],
        "timestamp": 1607432944
      }
    ]
  "internalHostname": "localdocker.test",
  "metrics": [],
  "service_checks": [],
  "topologies": []
}'

sts event send "HealthStateChangedEvent" \
    --title "event_title" \
    -i "element_identifier1" "element_identifier2" \
    -s "source_system" \
    -c "Changes" \
    -d '{"data_key1":"data_value1", "data_key2":"data_value2"}' \
    --links "link_title1: link_url1" "link_title2: link_url2"