LogoLogo
StackState.comDownloadSupportExplore playground
StackState v5.1
StackState v5.1
  • Welcome to the StackState docs!
  • StackState self-hosted v5.1 docs
  • Getting Started
  • 🚀Setup
    • Install StackState
      • Requirements
      • Kubernetes / OpenShift
        • Kubernetes install
        • OpenShift install
        • Required Permissions
        • Non-high availability setup
        • Override default configuration
        • Configure storage
        • Configure Ingress
        • Install from custom image registry
        • Migrate from Linux install
      • Linux
        • Before you install
        • Download
        • Install StackState
        • Install with production configuration
        • Install with development configuration
        • Install with POC configuration
        • Set up a reverse proxy
        • Set up TLS without reverse proxy
      • Initial run guide
      • Troubleshooting
    • Upgrade StackState
      • Steps to upgrade
      • Version specific upgrade instructions
      • StackPack versions
      • StackState release notes
    • StackState Agent
      • About StackState Agent V3
      • Docker
      • Kubernetes / OpenShift
      • Linux
      • Windows
      • Advanced Agent configuration
      • Use an HTTP/HTTPS proxy
      • Agent V1 (legacy)
      • Migrate Agent V1 to Agent V2
        • Linux
        • Docker
    • StackState CLI
      • CLI: sts
      • CLI: stac (deprecated)
      • Comparison between CLIs
    • Data management
      • Backup and Restore
        • Kubernetes backup
        • Linux backup
        • Configuration backup
      • Data retention
      • Clear stored data
  • 👤Use
    • Concepts
      • The 4T data model
      • Components
      • Relations
      • Health state
      • Layers, Domains and Environments
      • Perspectives
      • Anomaly detection
      • StackState architecture
    • StackState UI
      • Explore mode
      • Filters
      • Views
        • About views
        • Configure the view health
        • Create and edit views
        • Visualization settings
      • Perspectives
        • Topology Perspective
        • Events Perspective
        • Traces Perspective
        • Metrics Perspective
      • Timeline and time travel
      • Analytics
      • Keyboard shortcuts
    • Checks and monitors
      • Checks
      • Add a health check
      • Anomaly health checks
      • Monitors
      • Manage monitors
    • Problem analysis
      • About problems
      • Problem lifecycle
      • Investigate a problem
      • Problem notifications
    • Metrics
      • Telemetry streams
      • Golden signals
      • Top metrics
      • Add a telemetry stream
      • Browse telemetry
      • Set telemetry stream priority
    • Events
      • About events
      • Event notifications
      • Manage event handlers
    • Glossary
  • 🧩StackPacks
    • About StackPacks
    • Add-ons
      • Autonomous Anomaly Detector
      • Health Forecast
    • Integrations
      • About integrations
      • 💠StackState Agent V2
      • 💠AWS
        • AWS
        • AWS ECS
        • AWS X-ray
        • StackState/Agent IAM role: EC2
        • StackState/Agent IAM role: EKS
        • Policies for AWS
        • AWS (legacy)
        • Migrate AWS (legacy) to AWS
      • 💠Dynatrace
      • 💠Kubernetes
      • 💠OpenShift
      • 💠OpenTelemetry
        • About instrumentations
        • AWS NodeJS Instrumentation
        • Manual Instrumentation
          • Prerequisites
          • Tracer and span mappings
          • Relations between components
          • Span health state
          • Merging components
          • Code examples
      • 💠ServiceNow
      • 💠Slack
      • 💠Splunk
        • Splunk
        • Splunk Events
        • Splunk Health
        • Splunk Metrics
        • Splunk Topology
      • 💠VMWare vSphere
      • Apache Tomcat
      • Azure
      • Cloudera
      • Custom Synchronization
      • DotNet APM
      • Elasticsearch
      • Humio
      • Java APM
      • JMX
      • Logz.io
      • MySQL
      • Nagios
      • OpenMetrics
      • PostgreSQL
      • Prometheus
      • SAP
      • SCOM
      • SolarWinds
      • Static Health
      • Static Topology
      • Traefik
      • WMI
      • Zabbix
    • Develop your own StackPacks
  • 🔧Configure
    • Topology
      • Component actions
      • Identifiers
      • Topology naming guide
      • Topology sources
      • Create a topology manually
      • Configure topology synchronizations
      • Enable email event notifications
      • Send topology data over HTTP
      • Set the topology filtering limit
      • Use a proxy for event handlers
      • Use tags
      • Tune topology synchronization
      • Debug topology synchronization
    • Telemetry
      • Add telemetry during topology synchronization
      • Data sources
        • Elasticsearch
        • Prometheus mirror
      • Send events over HTTP
      • Send metrics data over HTTP
      • Set the default telemetry interval
      • Debug telemetry synchronization
    • Traces
      • Set up traces
      • Advanced configuration for traces
    • Health
      • Health synchronization
      • Send health data over HTTP
        • Send health data
        • Repeat Snapshots JSON
        • Repeat States JSON
        • Transactional Increments JSON
      • Debug health synchronization
    • Anomaly Detection
      • Export anomaly feedback
      • Scale the AAD up and down
      • The AAD status UI
    • Security
      • Authentication
        • Authentication options
        • File based
        • LDAP
        • Open ID Connect (OIDC)
        • KeyCloak
        • Service tokens
      • RBAC
        • Role-based Access Control
        • Permissions
        • Roles
        • Scopes
        • Subjects
      • Secrets management
      • Self-signed certificates
      • Set up a security backend for Linux
      • Set up a security backend for Windows
    • Logging
      • Kubernetes logs
      • Linux logs
      • Enable logging for functions
  • 📖Develop
    • Developer guides
      • Agent checks
        • About Agent checks
        • Agent check API
        • Agent check state
        • How to develop Agent checks
        • Connect an Agent check to StackState
      • Custom functions and scripts
        • StackState functions
        • Check functions
        • Component actions
        • Event handler functions
        • ID extractor functions
        • Mapping functions
        • Monitor functions
        • Propagation functions
        • Template functions
        • View health state configuration functions
      • Custom Synchronization StackPack
        • About the Custom Synchronization StackPack
        • How to customize elements created by the Custom Synchronization StackPack
        • How to configure a custom synchronization
      • Integrate external services
      • Mirroring Telemetry
      • Monitors
        • Create monitors
        • Monitor STJ file format
      • StackPack development
        • How to create a StackPack
        • Packaging
        • How to get a template file
        • How to make a multi-instance StackPack
        • Prepare a multi-instance provisioning script
        • Upload a StackPack file
        • Prepare a shared template
        • Customize a StackPack
        • Prepare instance template files
        • Prepare a StackPack provisioning script
        • Resources in a StackPack
        • StackState Common Layer
      • Synchronizations and templated files
    • Reference
      • StackState OpenAPI docs
      • StackState Template JSON (STJ)
        • Using STJ
        • Template functions
      • StackState Markup Language (STML)
        • Using STML
        • STML Tags
      • StackState Query Language (STQL)
      • StackState Scripting Language (STSL)
        • Scripting in StackState
        • Script result: Async
        • Script result: Streaming
        • Time in scripts
        • Script APIs
          • Async - script API
          • Component - script API
          • HTTP - script API
          • Prediction - script API
          • StackPack - script API
          • Telemetry - script API
          • Time - script API
          • Topology - script API
          • UI - script API
          • View - script API
    • Tutorials
      • Create a simple StackPack
      • Push data to StackState from an external system
      • Send events to StackState from an external system
      • Set up a mirror to pull telemetry data from an external system
Powered by GitBook
LogoLogo

Legal notices

  • Privacy
  • Cookies
  • Responsible disclosure
  • SOC 2/SOC 3
On this page
  • Overview
  • Setup
  • Installation
  • Configuration
  • Validation
  • Data collected
  • Metrics
  • Events
  • Traces
  • See also
  1. StackPacks
  2. Integrations

OpenMetrics

StackState Self-hosted v5.1.x

PreviousNagiosNextPostgreSQL

Last updated 1 year ago

Overview

StackState can be configured to retrieve metrics from an OpenMetrics endpoint and push these to StackState.

OpenMetrics is a .

Setup

Installation

The OpenMetrics check is included in the .

Configuration

To enable the OpenMetrics integration and begin collecting metrics data from an OpenMetrics endpoint, the OpenMetrics check must be configured on StackState . The check configuration provides all details required for the Agent to connect to your OpenMetrics endpoint and retrieve the available metrics.

  1. Deploy the Agent on your Kubernetes or OpenShift cluster.

  2. Add the annotations below when launching a pod that exposes metrics via an OpenMetrics endpoint. Add the following:

    • <CONTAINER_NAME> - the name of the container that exposes the OpenMetrics. It's possible to process multiple endpoints in a single pod (that's why there is a list in the JSON).

    • prometheus_url - the path (often just metrics) and port at which the OpenMetrics endpoint is exposed.

    • namespace - all metrics collected here will get this as a dot-separated prefix.

    • metrics - use ["*"] to collect all available metrics. It's also possible to specify a list of metrics to be fetched. This should either be a string representing the metric name or a mapping to rename the metric<EXPOSED_METRIC>:<SENT_METRIC>

      ...
      metadata:
        annotations:
          ad.stackstate.com/<CONTAINER_NAME>.check_names: '["openmetrics"]'
          ad.stackstate.com/<CONTAINER_NAME>.init_configs: '[{}]'
          ad.stackstate.com/<CONTAINER_NAME>.instances: |
            [ 
              {
                "prometheus_url": "http://%%host%%:<METRICS_PORT>/<METRICS_PATH>",
                "namespace": "<METRICS_NAMESPACE>", 
                "metrics": ["*"] 
              } 
            ]
      ...
      # This already exists in the pod spec, the container name needs to match the container that is exposing the openmetrics endpoint
      spec:
        containers:
         - name: <CONTAINER_NAME>
      ...
  3. You can also add optional configuration and filters:

    • prometheus_metrics_prefix - prefix to add to exposed OpenMetrics metrics.

    • health_service_check - send a service check <NAMESPACE>.prometheus.health reporting the health of the OpenMetrics endpoint. Default true.

    • label_to_hostname - override the hostname with the value of one label.

    • label_joins - target a metric and retrieve it's label via a 1:1 mapping

    • labels_mapper - rename labels. Format is <LABEL_TO_RENAME>: <NEW_LABEL_NAME>.

    • type_overrides - override a type in the OpenMetrics the payload or type an untyped metric (these would be ignored by default). Supported <METRIC_TYPE> are gauge, count and rate. Format is <METRIC_NAME>: <METRIC_TYPE>.

    • tags - list of tags to attach to every metric, event and service check emitted by this integration.

    • send_histograms_buckets - send the histograms bucket. Default true.

    • send_monotonic_counter - set to true to convert counters to a rate (note that two runs are required to produce the first result). Set to false to send counters as a monotonic counter. Default true.

    • exclude_labels - list of labels to be excluded.

    • prometheus_timeout - set a timeout for the OpenMetrics query.

    • ssl_cert - If your OpenMetrics endpoint is secured, enter the path to the certificate and specify the private key in the ssl_private_key parameter, or give the path to a file containing both the certificate and the private key.

    • ssl_private_key - required if the certificate linked in ssl_cert doesn't include the private key. Note that the private key to your local certificate must be unencrypted.

    • ssl_ca_cert - the path to the trusted CA used for generating custom certificates.

    • extra_headers - a list of additional HTTP headers to send in queries to the OpenMetrics endpoint. Can be combined with autodiscovery template variables. For example, "Authorization: Bearer %%env_TOKEN%%".

  4. Wait for the Agent to collect data from the OpenMetrics endpoint and send it to StackState.

  1. Edit the StackState configuration file /etc/stackstate-agent/conf.d/openmetrics.d/conf.yaml to include details of the OpenMetrics endpoint and the metrics to be retrieved.

    • prometheus_url - the URL exposing metrics in the OpenMetrics format

    • namespace - the namespace to be prepended to all metrics.

    • metrics - a list of metrics to be fetched from the OpenMetrics endpoint at prometheus_url. Either a string representing the metric name or a mapping to rename the metric<EXPOSED_METRIC>:<SENT_METRIC>. This list should contain at least one metric.

    init_config:
      
    instances:
      - prometheus_url: <OPENMETRICS_ENDOINT>
        namespace: "service"
        metrics:
          - processor: cpu
          - memory: mem
          - io
      #  prometheus_metrics_prefix: <PREFIX>_
      #  health_service_check: true
      #  label_to_hostname: <LABEL>
      #  label_joins:
      #    target_metric:
      #      label_to_match: <MATCHED_LABEL>
      #      labels_to_get:
      #        - <EXTRA_LABEL_1>
      #        - <EXTRA_LABEL_2>
      #  labels_mapper:
      #    flavor: origin
      #  type_overrides:
      #    <METRIC_NAME>: <METRIC_TYPE>
      #  tags:
      #    - <KEY_1>:<VALUE_1>
      #    - <KEY_2>:<VALUE_2>
      #  send_histograms_buckets: true
      #  send_monotonic_counter: true
      #  exclude_labels:
      #    - timestamp
      #  prometheus_timeout: 10
      #  ssl_cert: "<CERT_PATH>"
      #  ssl_private_key: "<KEY_PATH>"
      #  ssl_ca_cert: "<CA_CERT_PATH>"
      #  extra_headers:
      #    <HEADER_NAME>: <HEADER_VALUE>
  2. You can also add optional configuration and filters:

    • prometheus_metrics_prefix - prefix to add to exposed OpenMetrics metrics.

    • health_service_check - send a service check <NAMESPACE>.prometheus.health reporting the health of the OpenMetrics endpoint. Default true.

    • label_to_hostname - override the hostname with the value of one label.

    • label_joins - target a metric and retrieve it's label via a 1:1 mapping

    • labels_mapper - rename labels. Format is <LABEL_TO_RENAME>: <NEW_LABEL_NAME>.

    • type_overrides - override a type in the OpenMetrics payload or type an untyped metric (these would be ignored by default). Supported <METRIC_TYPE> are gauge, count and rate.

    • tags - list of tags to attach to every metric, event and service check emitted by this integration.

    • send_histograms_buckets - send the histograms bucket. Default true.

    • send_monotonic_counter - set to true to convert counters to a rate (note that two runs are required to produce the first result). Set to false to send counters as a monotonic counter. Default true.

    • exclude_labels - list of labels to be excluded.

    • prometheus_timeout - set a timeout for the OpenMetrics query.

    • ssl_cert - If your OpenMetrics endpoint is secured, enter the path to the certificate and specify the private key in the ssl_private_key parameter, or give the path to a file containing both the certificate and the private key.

    • ssl_private_key - required if the certificate linked in ssl_cert doesn't include the private key. Note that the private key to your local certificate must be unencrypted.

    • ssl_ca_cert - the path to the trusted CA used for generating custom certificates.

    • extra_headers - a list of additional HTTP headers to send in queries to the OpenMetrics endpoint. Can be combined with autodiscovery template variables. For example, "Authorization: Bearer %%env_TOKEN%%".

  3. Once the Agent has restarted, wait for the Agent to collect data from the OpenMetrics endpoint and send it to StackState.

Validation

Run the Agent status subcommand and look for openmetrics under the Checks section.

Data collected

Metrics

Events

The OpenMetrics integration doesn't retrieve any events data.

Traces

The OpenMetrics integration doesn't retrieve trace data.

See also

Example OpenMetrics Agent check configuration file:

to apply the configuration changes.

Follow the instructions on the Kubernetes Agent page to

By default, all metrics are retrieved from the specified OpenMetrics endpoint and available in the StackState multi metrics data source. To optimize performance, a maximum of 2000 metrics will be retrieved. If the check is attempting to retrieve more than 2000 metrics, add a metrics filter to the to ensure that all important metrics can be retrieved within the limit.

Retrieved metrics won't automatically be mapped to topology elements. They can be browsed using the or added to a component as a telemetry stream. Select the data source StackState Multi Metrics and type the configured namespace in the Select box to get a full list of all available metrics.

🧩
openmetrics/conf.yaml.example (github.com)
telemetry inspector
StackState Agent V2 StackPack
configuration
Agent V2 StackPack
StackState curated integration
track the status of the OpenMetrics check
Restart StackState Agent V3
Inspect OpenMetrics telemetry