Stackstate-Kubernetes Integration

Overview

Get metrics from your Kubelets in real time to:

  • Visualize your Kubernetes cluster performance
  • Collect performances metrics for your containers, pods, container namespaces
  • Create monitors on the status of your Kubelets
  • Ingest Kubernetes labels as tags in Stackstate.

Installation

The Kubernetes integration is deployed as a docker container along side your existing workloads.

Installation via DaemonSets (Kubernetes >=1.1.0)

If you are running Kubernetes >= 1.2.0, you can take advantage of DaemonSets to automatically deploy the Stackstate Agent on all your nodes. On clusters running 1.1.x you will need to explicitly enable the DaemonSets extension.

  1. Download the sts-agent.yaml manifest file.
  2. Launch sts-agent:

    kubectl create -f sts-agent.yaml

Manual Installation

If DaemonSets are not an option for your Kubernetes cluster, you will need to install the Stackstate agent as a sidecar container on each Kubernetes node.

docker run -d --name sts-agent -h `hostname` \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
  -e API_KEY='YOUR_API_KEY_HERE' -e KUBERNETES=yes stackstate/docker-sts-agent:latest

Configuration

Configure the agent by editing the kubernetes.yaml file in conf.d:

init_config:
  tags:
    - optional_tag1
    - optional_tag2

instances:
  # The kubernetes check retrieves metrics from cadvisor running under kubelet.
  # By default we will assume we're running under docker and will use the address
  # of the default router to reach the cadvisor api.
  #
  # To override, e.g. in the case of a standalone cadvisor instance, use the following:
  #
  # host: localhost
  # port: 4194
  # method: http
  - port: 4194

  # use_histogram controls whether we send detailed metrics, i.e. one per container.
  # When false, we send detailed metrics corresponding to individual containers, tagging by container id
  # to keep them unique.
  # When true, we aggregate data based on container image.
  # Defaults to false
  #
  # use_histogram: True
  #
  # kubelet_port: 10255
  #
  # We can define a whitelist of patterns that permit publishing raw metrics.
  # enabled_rates:
  #   - cpu.*
  #   - network.*
  #
  # enabled_gauges:
  #   - filesystem.*

Since the agent is deployed as a docker container, refer to the Agent container documentation.

For more details about configuring this integration refer to the following file(s) on GitHub:

Kubernetes State Metrics

If you are running Kubernetes >= 1.2.0, you can use the kube-state-metrics project to provide additional metrics (identified by the kubernetes_state prefix in the metrics list below) to Stackstate.

To run kube-state-metrics, create a kube-state-metrics.yaml file using the following manifest to deploy the kube-state-metrics service:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kube-state-metrics
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      containers:
      - name: kube-state-metrics
        image: gcr.io/google_containers/kube-state-metrics:v0.3.0
        ports:
        - name: metrics
          containerPort: 8080
        resources:
          requests:
            memory: 30Mi
            cpu: 100m
          limits:
            memory: 50Mi
            cpu: 200m
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: 'true'
  labels:
    app: kube-state-metrics
  name: kube-state-metrics
spec:
  ports:
  - name: metrics
    port: 8080
    targetPort: metrics
    protocol: TCP
  selector:
    app: kube-state-metrics

Then deploy it by running:

kubectl create -f kube-state-metrics.yaml

The manifest above uses Google’s publicly available kube-state-metrics container. If you would like to build your own, you can do so by:

  1. Clone the kube-state-metrics Github repository
  2. Run make container to build the container
  3. Run kubectl apply -f kubernetes

If you configure your Kubernetes State Metrics service to run on a different URL or port, you can configure the Stackstate Agent by setting the kube_state_url parameter in conf.d/kubernetes_state.yaml, then restarting the agent. For more information, see the kubernetes_state.yaml.example file. If you have enabled Service Discovery, the kube state URL will be configured and managed automatically.

Validation

To verify the Stackstate agent is running in your environment as a daemonset, execute:

kubectl get daemonset

If the agent is deployed you will see similar output to the text below, where desired and current are equal to the number of running nodes in your cluster.

NAME       DESIRED   CURRENT   NODE-SELECTOR   AGE
sts-agent   3         3         <none>          11h

Limitations

Please be aware that Kubernetes relies on Heapster to report metrics, rather than the cgroup file directly. The collection interval for Heapster is unknown which can lead to innacurate time-related data, such as CPU usage. If you require more precise metrics, we recommend using the Stackstate-Docker Integration.

Metrics

kubernetes.cpu.capacity
(gauge)
The number of cores in this machine.
kubernetes.cpu.usage.total
(gauge)
The percentage of CPU time used
shown as percent_nano
kubernetes.cpu.limits
(gauge)
The limit of cpu cores set
shown as cpu
kubernetes.cpu.requests
(gauge)
The requested cpu cores
shown as cpu
kubernetes.filesystem.usage
(gauge)
The amount of disk used
shown as byte
kubernetes.filesystem.usage_pct
(gauge)
The percentage of disk used
shown as fraction
kubernetes.memory.capacity
(gauge)
The amount of memory (in bytes) in this machine
shown as byte
kubernetes.memory.limits
(gauge)
The limit of memory set
shown as byte
kubernetes.memory.requests
(gauge)
The requested memory
shown as byte
kubernetes.memory.usage
(gauge)
The amount of memory used
shown as byte
kubernetes.network.rx_bytes
(gauge)
The amount of bytes per second received
shown as byte/second
kubernetes.network.tx_bytes
(gauge)
The amount of bytes per second transmitted
shown as byte/second
kubernetes.network_errors
(gauge)
The amount of network errors per second
shown as error/second
kubernetes_state.deployment.replicas_available
(gauge)
The number of available replicas per deployment
kubernetes_state.deployment.replicas_unavailable
(gauge)
The number of unavailable replicas per deployment
kubernetes_state.deployment.replicas_updated
(gauge)
The number of updated replicas per deployment
kubernetes_state.deployment.replicas_desired
(gauge)
The number of desired replicas per deployment
kubernetes_state.node.cpu_capacity
(gauge)
The total CPU resources of the node
shown as cpu
kubernetes_state.node.memory_capacity
(gauge)
The total memory resources of the node
shown as byte
kubernetes_state.node.pods_capacity
(gauge)
The total pod resources of the node
kubernetes_state.node.cpu_allocatable
(gauge)
The CPU resources of a node that are available for scheduling
shown as cpu
kubernetes_state.node.memory_allocatable
(gauge)
The memory resources of a node that are available for scheduling
shown as byte
kubernetes_state.node.pods_allocatable
(gauge)
The pod resources of a node that are available for scheduling
kubernetes_state.node.unschedulable
(gauge)
Whether a node can schedule new pods