Prometheus mirror

StackState Self-hosted v5.1.x

Overview

Prometheus mirror is a gateway between StackState and Prometheus that enables Prometheus telemetry in StackState.

Prerequisites

The Prometheus Mirror has the following prerequisites:

• The mirror must be reachable from StackState

• Prometheus must be reachable from the mirror without authentication

Helm

The Prometheus mirror is available via the StackState helm repository. Configure your helm with these 2 commands:

helm repo add stackstate https://helm.stackstate.io
helm repo update

Install the Prometheus mirror with the following command:

helm install prometheusmirror stackstate/prometheusmirror --set global.apiKey=API_KEY

Alternatively, you can use the Docker container directly:

docker pull stackstate/prometheusmirror:latest

Mirror configuration

The Prometheus mirror is configured using the following parameters:

• global.apiKey - the API key used to authenticate communication between the mirror and StackState

• workers - number of workers processes (default: 20)

• port - the port the mirror is listening on (default: 9900)

StackState configuration

In order to start using Prometheus mirror in StackState one has to create Mirror Datasource

Configure Mirror Datasource

Create a new Mirror datasource:

• DataSourceUrl - points to the Prometheus mirror endpoint, for example http://prometheusmirror.stackstate.svc.cluster.local:9900/

• API Key - should be the same key as specified by global.apiKey mirror configuration

• Connection Details JSON - the mirror configuration json, for example:

    {
      "host": "<prometheus host>",
      "port": <prometheus port>,
      "requestTimeout": 20000
    }

  Prometheus host/port refers to the actual Prometheus host/port (not the mirror).

Query Configuration

Prometheus Counter

Counter queries fetch counter metrics from Prometheus. The retrieved counter values are transformed to a rate.

The following are sample parameters for a counter query:

• __counter__ = go_memstats_lookups_total

• job = payment-service

• name = payment

• instance = 127.0.0.1:80

Prometheus Gauge

Gauge queries fetch gauge metrics from Prometheus.

The following are sample parameters for a gauge query:

• __gauge__ = go_gc_duration_seconds

• job = payment-service

• name = payment

• instance = 127.0.0.1:80

Prometheus Histogram and Summary

Prometheus histogram and summary queries are not supported from the query interface. They still can be configured using tilda-query.

Tilde query ~

The query allows arbitrary Prometheus queries, for example:

~ = histogram_quantile(0.95, sum(rate(request_duration_seconds_bucket{instance='127.0.0.1:80', name='payment-service'}[1m])) by (name, le)) * 1000

See also

• Developer guide - mirroring telemetry

• Developer tutorial - Set up a mirror to pull telemetry data from an external system