Stackstate-Kubernetes Integration

Overview

Get metrics from your Kubelets in real time to:

  • Visualize your Kubernetes cluster performance
  • Collect performances metrics for your containers, pods, container namespaces
  • Create monitors on the status of your Kubelets
  • Ingest Kubernetes labels as tags in Stackstate.

Setup

Installation

This check is packaged with the Agent, so simply install the agent.

Configuration

Configure the agent by editing the kubernetes.yaml file in conf.d:

init_config:

instances:
  # The kubernetes check retrieves metrics from cadvisor running under kubelet.
  # By default we will assume we're running under docker and will use the address
  # of the default router to reach the cadvisor api.
  #
  # To override, e.g. in the case of a standalone cadvisor instance, use the following:
  #
  # host: localhost
  # port: 4194
  # method: http
  - port: 4194

  # use_histogram controls whether we send detailed metrics, i.e. one per container.
  # When false, we send detailed metrics corresponding to individual containers, tagging by container id
  # to keep them unique.
  # When true, we aggregate data based on container image.
  # Defaults to false

  # use_histogram: True
  #
  # kubelet_port: 10255
  #
  # We can define a whitelist of patterns that permit publishing raw metrics.
  # enabled_rates:
  #   - cpu.*
  #   - network.*
  #
  # enabled_gauges:
  #   - filesystem.*
  #
  # Custom tags that should be applied to kubernetes metrics
  # tags:
  #  - optional_tag1
  #  - optional_tag2

Since the agent is deployed as a docker container, refer to the Agent container documentation.

Kubernetes State Metrics

If you are running Kubernetes >= 1.2.0, you can use the kube-state-metrics project to provide additional metrics (identified by the kubernetes_state prefix in the metrics list below) to Stackstate.

To run kube-state-metrics, create a kube-state-metrics.yaml file using the following manifest to deploy the kube-state-metrics service:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kube-state-metrics
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      containers:
      - name: kube-state-metrics
        image: gcr.io/google_containers/kube-state-metrics:v0.3.0
        ports:
        - name: metrics
          containerPort: 8080
        resources:
          requests:
            memory: 30Mi
            cpu: 100m
          limits:
            memory: 50Mi
            cpu: 200m
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: 'true'
  labels:
    app: kube-state-metrics
  name: kube-state-metrics
spec:
  ports:
  - name: metrics
    port: 8080
    targetPort: metrics
    protocol: TCP
  selector:
    app: kube-state-metrics

Then deploy it by running:

kubectl create -f kube-state-metrics.yaml

The manifest above uses Google’s publicly available kube-state-metrics container. If you would like to build your own, you can do so by:

  1. Clone the kube-state-metrics Github repository
  2. Run make container to build the container
  3. Run kubectl apply -f kubernetes

If you configure your Kubernetes State Metrics service to run on a different URL or port, you can configure the StackState Agent by setting the kube_state_url parameter in conf.d/kubernetes_state.yaml, then restarting the agent. For more information, see the kubernetes_state.yaml.example file. If you have enabled Autodiscovery, the kube state URL will be configured and managed automatically.

Validation

To verify the Stackstate agent is running in your environment as a daemonset, execute:

kubectl get daemonset

If the agent is deployed you will see similar output to the text below, where desired and current are equal to the number of running nodes in your cluster.

NAME       DESIRED   CURRENT   NODE-SELECTOR   AGE
sts-agent   3         3         <none>          11h

Data Collected

Metrics

kubernetes.cpu.capacity
(gauge)
The number of cores in this machine.
shown as
kubernetes.cpu.usage.total
(gauge)
The percentage of CPU time used
shown as percent_nano
kubernetes.cpu.limits
(gauge)
The limit of cpu cores set
shown as cpu
kubernetes.cpu.requests
(gauge)
The requested cpu cores
shown as cpu
kubernetes.filesystem.usage
(gauge)
The amount of disk used
shown as byte
kubernetes.filesystem.usage_pct
(gauge)
The percentage of disk used
shown as fraction
kubernetes.memory.capacity
(gauge)
The amount of memory (in bytes) in this machine
shown as byte
kubernetes.memory.limits
(gauge)
The limit of memory set
shown as byte
kubernetes.memory.requests
(gauge)
The requested memory
shown as byte
kubernetes.memory.usage
(gauge)
The amount of memory used
shown as byte
kubernetes.network.rx_bytes
(gauge)
The amount of bytes per second received
shown as byte
kubernetes.network.tx_bytes
(gauge)
The amount of bytes per second transmitted
shown as byte
kubernetes.network_errors
(gauge)
The amount of network errors per second
shown as error

Further Reading

Knowledge Base

Limitations

Please be aware that Kubernetes relies on Heapster to report metrics, rather than the cgroup file directly. The collection interval for Heapster is unknown which can lead to innacurate time-related data, such as CPU usage. If you require more precise metrics, we recommend using the StackState-Docker Integration and that Heapster must be running in standalone mode.