Push telemetry to StackState over HTTP
StackState can either pull telemetry from a data source or can receive pushed telemetry. Pushed telemetry is stored by StackState, while pulled telemetry is not. Pushed telemetry is stored for the duration of the configured retention period. This page describes how telemetry can be pushed.
There are several ways to send telemetry to StackState. A large number of integrations are provided out of the box that may help you get started. If there is no out of the box integration you can send telemetry to StackState using either HTTP or the StackState CLI.
The StackState receiver API is responsible for receiving both telemetry and topology. By default, the receiver API is hosted at:
https://<baseUrl>:<receiverPort>/stsAgent/intake?api_key=<API_KEY>
Both the baseUrl and API_KEY are set during StackState installation, for details see:
Telemetry is sent to the receiver API via HTTP POST and has a common JSON object for all messages. One message can contain multiple metrics and multiple events.
{"collection_timestamp": 1548855554, // the epoch timestamp for the collection"events": {}, // see the section on "events", below"internalHostname": "localdocker.test", // the host that is sending this data"metrics": [], // see the section on "metrics", below"service_checks": [],"topologies": [] // used for sending topology data}
Depending on your StackState configuration, received metrics or events that are too old will be ignored.
Metrics can be sent to the StackState receiver API using the "metrics" property of the common JSON object.
["test.metric", // the metric name1548857152,10.0, // double - value of the metric{"hostname": "localdocker.test","tags": ["tag_key1:tag_value1","tag_key2:tag_value2"],"type": "gauge"}]
Every metric has the following details:
name - The metric name. Must not start with any of the following prefixes:
host,labels,name,tags,timeReceived,timestamp,tagsorvalues.timestamp - The epoch timestamp of the metric.
value - The value of the metric.
hostname - The host this metric is from.
type - The type of metric. Can be
gauge,count,rate,counterorraw.tags - Optional. A list of key/value tags to associate with the metric.
The timestamp and value are used to plot the metric as a time series. The name and tags can be used to define a metric stream in StackState.
Multiple metrics can be sent in one JSON message via HTTP POST. For example:
curl -X POST \'http://<stackstateURL>/stsAgent/intake?api_key=<API_KEY>' \-H 'Content-Type: application/json' \-d '{"collection_timestamp": 1548857167,"events": {},"internalHostname": "localdocker.test","metrics": [["test.metric",1548857152,10.0,{"hostname": "localdocker.test","tags": ["tag_key1:tag_value1","tag_key2:tag_value2"],"type": "gauge"}],["test.metric",1548857167,10.0,{"hostname": "localdocker.test","tags": ["tag_key1:tag_value1","tag_key2:tag_value2"],"type": "gauge"}]],"service_checks": [],"topologies": []}'
You can also send metrics to StackState using the StackState CLI metric send command.
Events can be sent to the StackState receiver API using the "events" property of the common JSON object.
All events in StackState relate to a topology element or elements. Any of the event properties can be used to define a log stream in StackState.
"event.test": [ // The event name{"context": {"category": "Changes","data": {"data_key1":"data_value1","data_key2":"data_value2"},"element_identifiers": ["element_identifier1","element_identifier2"],"source": "source_system","source_links": [{"title": "link_title","url": "link_url"}]},"event_type": "event_typeEvent","msg_title": "event_title","msg_text": "event_text","source_type_name": "source_event_type","tags": ["tag_key1:tag_value1","tag_key2:tag_value2",],"timestamp": 1607432944}]
Events have the following details:
An event name. this must not start with any of the following prefixes:
eventType,host,labels,message,name,tags,timeReceived,timestamportitle.context - Includes details of the source system for an event:
category - The event category. Can be
Activities,Alerts,Anomalies,ChangesorOthers.element_identifiers - The identifiers for the topology element(s) the event relates to. These are used to bind the event to a topology element or elements.
source - The name of the system from which the event originates, for example AWS, Kubernetes or JIRA.
data - Optional. A list of key/value details about the event, for example a configuration version.
source_identifier - Optional. The original identifier of the event in the source system.
source_links - Optional. A list of links related to the event, for example a dashboard or the event in the source system.
event_type - Describes the event being sent. This should generally end with the suffix
Event, for exampleConfigurationChangedEvent,VersionChangedEvent.msg_text - Optional. The text body of the event.
msg_title - Optional. The title of the event.
source_type_name - Optional. The source event type name.
tags - Optional. A list of key/value tags to associate with the event.
timestamp - The epoch timestamp for the event.
Multiple events can be sent in one JSON message via HTTP POST. You can also send a single event to StackState using the StackState CLI event send command. For example:
curl -X POST \'http://<stackstateURL>/stsAgent/intake?api_key=<API_KEY>' \-H 'Content-Type: application/json' \-d '{"collection_timestamp": 1548857342,"events": {"event.test01": [ // The event name{"context": {"category": "Changes","data": {"data_key1":"data_value1","data_key2":"data_value2"},"element_identifiers": ["element_identifier1","element_identifier2"],"source": "source_system","source_links": [{"title": "link_title","url": "link_url"}]},"event_type": "HealthStateChangedEvent","msg_title": "event_title","msg_text": "event_text","source_type_name": "source_event_type","tags": ["tag_key1:tag_value1","tag_key2:tag_value2",],"timestamp": 1607432944}],"event.test02": [ // The event name{"context": {"category": "Changes","data": {"data_key1":"data_value1","data_key2":"data_value2"},"element_identifiers": ["element_identifier1","element_identifier2"],"source": "source_system","source_links": [{"title": "link_title","url": "link_url"}]},"event_type": "HealthStateChangedEvent","msg_title": "event_title","msg_text": "event_text","source_type_name": "source_event_type","tags": ["tag_key1:tag_value1","tag_key2:tag_value2",],"timestamp": 1607432944}]"internalHostname": "localdocker.test","metrics": [],"service_checks": [],"topologies": []}'
sts event send "HealthStateChangedEvent" \--title "event_title" \-i "element_identifier1" "element_identifier2" \-s "source_system" \-c "Changes" \-d '{"data_key1":"data_value1", "data_key2":"data_value2"}' \--links "link_title1: link_url1" "link_title2: link_url2"
​StackState CLI reference​
​StackState identifiers​
​Events Perspective​
​Events tutorial​