LogoLogo
StackState.comDownloadSupportExplore playground
StackState v5.1
StackState v5.1
  • Welcome to the StackState docs!
  • StackState self-hosted v5.1 docs
  • Getting Started
  • 🚀Setup
    • Install StackState
      • Requirements
      • Kubernetes / OpenShift
        • Kubernetes install
        • OpenShift install
        • Required Permissions
        • Non-high availability setup
        • Override default configuration
        • Configure storage
        • Configure Ingress
        • Install from custom image registry
        • Migrate from Linux install
      • Linux
        • Before you install
        • Download
        • Install StackState
        • Install with production configuration
        • Install with development configuration
        • Install with POC configuration
        • Set up a reverse proxy
        • Set up TLS without reverse proxy
      • Initial run guide
      • Troubleshooting
    • Upgrade StackState
      • Steps to upgrade
      • Version specific upgrade instructions
      • StackPack versions
      • StackState release notes
    • StackState Agent
      • About StackState Agent V3
      • Docker
      • Kubernetes / OpenShift
      • Linux
      • Windows
      • Advanced Agent configuration
      • Use an HTTP/HTTPS proxy
      • Agent V1 (legacy)
      • Migrate Agent V1 to Agent V2
        • Linux
        • Docker
    • StackState CLI
      • CLI: sts
      • CLI: stac (deprecated)
      • Comparison between CLIs
    • Data management
      • Backup and Restore
        • Kubernetes backup
        • Linux backup
        • Configuration backup
      • Data retention
      • Clear stored data
  • 👤Use
    • Concepts
      • The 4T data model
      • Components
      • Relations
      • Health state
      • Layers, Domains and Environments
      • Perspectives
      • Anomaly detection
      • StackState architecture
    • StackState UI
      • Explore mode
      • Filters
      • Views
        • About views
        • Configure the view health
        • Create and edit views
        • Visualization settings
      • Perspectives
        • Topology Perspective
        • Events Perspective
        • Traces Perspective
        • Metrics Perspective
      • Timeline and time travel
      • Analytics
      • Keyboard shortcuts
    • Checks and monitors
      • Checks
      • Add a health check
      • Anomaly health checks
      • Monitors
      • Manage monitors
    • Problem analysis
      • About problems
      • Problem lifecycle
      • Investigate a problem
      • Problem notifications
    • Metrics
      • Telemetry streams
      • Golden signals
      • Top metrics
      • Add a telemetry stream
      • Browse telemetry
      • Set telemetry stream priority
    • Events
      • About events
      • Event notifications
      • Manage event handlers
    • Glossary
  • 🧩StackPacks
    • About StackPacks
    • Add-ons
      • Autonomous Anomaly Detector
      • Health Forecast
    • Integrations
      • About integrations
      • 💠StackState Agent V2
      • 💠AWS
        • AWS
        • AWS ECS
        • AWS X-ray
        • StackState/Agent IAM role: EC2
        • StackState/Agent IAM role: EKS
        • Policies for AWS
        • AWS (legacy)
        • Migrate AWS (legacy) to AWS
      • 💠Dynatrace
      • 💠Kubernetes
      • 💠OpenShift
      • 💠OpenTelemetry
        • About instrumentations
        • AWS NodeJS Instrumentation
        • Manual Instrumentation
          • Prerequisites
          • Tracer and span mappings
          • Relations between components
          • Span health state
          • Merging components
          • Code examples
      • 💠ServiceNow
      • 💠Slack
      • 💠Splunk
        • Splunk
        • Splunk Events
        • Splunk Health
        • Splunk Metrics
        • Splunk Topology
      • 💠VMWare vSphere
      • Apache Tomcat
      • Azure
      • Cloudera
      • Custom Synchronization
      • DotNet APM
      • Elasticsearch
      • Humio
      • Java APM
      • JMX
      • Logz.io
      • MySQL
      • Nagios
      • OpenMetrics
      • PostgreSQL
      • Prometheus
      • SAP
      • SCOM
      • SolarWinds
      • Static Health
      • Static Topology
      • Traefik
      • WMI
      • Zabbix
    • Develop your own StackPacks
  • 🔧Configure
    • Topology
      • Component actions
      • Identifiers
      • Topology naming guide
      • Topology sources
      • Create a topology manually
      • Configure topology synchronizations
      • Enable email event notifications
      • Send topology data over HTTP
      • Set the topology filtering limit
      • Use a proxy for event handlers
      • Use tags
      • Tune topology synchronization
      • Debug topology synchronization
    • Telemetry
      • Add telemetry during topology synchronization
      • Data sources
        • Elasticsearch
        • Prometheus mirror
      • Send events over HTTP
      • Send metrics data over HTTP
      • Set the default telemetry interval
      • Debug telemetry synchronization
    • Traces
      • Set up traces
      • Advanced configuration for traces
    • Health
      • Health synchronization
      • Send health data over HTTP
        • Send health data
        • Repeat Snapshots JSON
        • Repeat States JSON
        • Transactional Increments JSON
      • Debug health synchronization
    • Anomaly Detection
      • Export anomaly feedback
      • Scale the AAD up and down
      • The AAD status UI
    • Security
      • Authentication
        • Authentication options
        • File based
        • LDAP
        • Open ID Connect (OIDC)
        • KeyCloak
        • Service tokens
      • RBAC
        • Role-based Access Control
        • Permissions
        • Roles
        • Scopes
        • Subjects
      • Secrets management
      • Self-signed certificates
      • Set up a security backend for Linux
      • Set up a security backend for Windows
    • Logging
      • Kubernetes logs
      • Linux logs
      • Enable logging for functions
  • 📖Develop
    • Developer guides
      • Agent checks
        • About Agent checks
        • Agent check API
        • Agent check state
        • How to develop Agent checks
        • Connect an Agent check to StackState
      • Custom functions and scripts
        • StackState functions
        • Check functions
        • Component actions
        • Event handler functions
        • ID extractor functions
        • Mapping functions
        • Monitor functions
        • Propagation functions
        • Template functions
        • View health state configuration functions
      • Custom Synchronization StackPack
        • About the Custom Synchronization StackPack
        • How to customize elements created by the Custom Synchronization StackPack
        • How to configure a custom synchronization
      • Integrate external services
      • Mirroring Telemetry
      • Monitors
        • Create monitors
        • Monitor STJ file format
      • StackPack development
        • How to create a StackPack
        • Packaging
        • How to get a template file
        • How to make a multi-instance StackPack
        • Prepare a multi-instance provisioning script
        • Upload a StackPack file
        • Prepare a shared template
        • Customize a StackPack
        • Prepare instance template files
        • Prepare a StackPack provisioning script
        • Resources in a StackPack
        • StackState Common Layer
      • Synchronizations and templated files
    • Reference
      • StackState OpenAPI docs
      • StackState Template JSON (STJ)
        • Using STJ
        • Template functions
      • StackState Markup Language (STML)
        • Using STML
        • STML Tags
      • StackState Query Language (STQL)
      • StackState Scripting Language (STSL)
        • Scripting in StackState
        • Script result: Async
        • Script result: Streaming
        • Time in scripts
        • Script APIs
          • Async - script API
          • Component - script API
          • HTTP - script API
          • Prediction - script API
          • StackPack - script API
          • Telemetry - script API
          • Time - script API
          • Topology - script API
          • UI - script API
          • View - script API
    • Tutorials
      • Create a simple StackPack
      • Push data to StackState from an external system
      • Send events to StackState from an external system
      • Set up a mirror to pull telemetry data from an external system
Powered by GitBook
LogoLogo

Legal notices

  • Privacy
  • Cookies
  • Responsible disclosure
  • SOC 2/SOC 3
On this page
  • Overview
  • Setup
  • Prerequisites
  • Supported container runtimes
  • Install
  • Deploy: Agent and Cluster Agent
  • Configure kube-state-metrics (stackstate/stackstate-agent helm chart versions prior to and including version 3.3.0)
  • Configure kube-state-metrics (stackstate/stackstate-agent helm chart versions later than and including version 3.5.0)
  • Status
  • Integration details
  • Data retrieved
  • REST API endpoints
  • Component actions
  • Kubernetes views in StackState
  • Open source
  • Troubleshooting
  • Uninstall
  • Release notes
  • See also
  1. StackPacks
  2. Integrations

Kubernetes

StackState Self-hosted v5.1.x

PreviousDynatraceNextOpenShift

Last updated 1 year ago

Overview

The Kubernetes integration is used to create a near real-time synchronization of topology and associated internal services from a Kubernetes cluster to StackState. This StackPack allows monitoring of the following:

  • Workloads

  • Nodes, pods, containers and services

  • Configmaps, secrets and volumes

Kubernetes is a .

The Kubernetes integration collects topology data in a Kubernetes cluster as well as metrics and events.

  • In StackState:

Setup

Prerequisites

The following prerequisites are required to install the Kubernetes StackPack and deploy the StackState Agent and Cluster Agent:

  • A Kubernetes Cluster must be up and running.

  • A recent version of Helm 3.

  • A user with permissions to create privileged pods, ClusterRoles and ClusterRoleBindings:

    • ClusterRole and ClusterRoleBinding are needed to grant StackState Agents permissions to access the Kubernetes API.

    • StackState Agents need to run in a privileged pod to be able to gather information on network connections and host information.

Supported container runtimes

From StackState Agent V2.16, the following container runtimes are supported:

  • containerd

  • CRI-O

  • Docker

Note that versions of StackState Agent prior to v2.16 support the Docker container runtime only.

Install

Install the Kubernetes StackPack from the StackState UI StackPacks > Integrations screen. You will need to enter the following details:

  • Kubernetes Cluster Name - A name to identify the cluster. This doesn't need to match the cluster name used in kubeconfig, however, that's usually a good candidate for a unique name.

If the Agent StackPack isn't already installed, this will be automatically installed together with the Kubernetes StackPack. StackState requires the Agent StackPack to work with the StackState Agent, which will need to be deployed on each node in the Kubernetes cluster.

Deploy: Agent and Cluster Agent

For the Kubernetes integration to retrieve topology, events and metrics data, you will need to have the following installed on your Kubernetes cluster:

  • StackState Agent V3 on each node in the cluster

  • StackState Cluster Agent on one node

  • StackState Checks Agent on one node

  • kube-state-metrics

Configure kube-state-metrics (stackstate/stackstate-agent helm chart versions prior to and including version 3.3.0)

The default URL that the kubernetes_state check uses is:

http://<release-name>-kube-state-metrics.<namespace>.svc.cluster.local:8080/metrics

If an alternative kube-state-metrics pod (i.e. Prometheus) is installed, the default StackState kube-state-metrics pod can be disabled and the kubernetes_state check redirected to the alternative service:

  1. Update the values.yaml file used to deploy the checks-agent, for example:

    dependencies:
      kubeStateMetrics:
        enabled: false
    checksAgent:
      kubeStateMetrics:
        url: http://YOUR_KUBE_STATE_METRICS_SERVICE_NAME:8080/metrics
  2. Deploy the cluster_agent using the updated values.yaml:

    helm upgrade --install \
    --namespace stackstate \
    --create-namespace \
    --set-string 'stackstate.apiKey'='<STACKSTATE_RECEIVER_API_KEY>' \
    --set-string 'stackstate.cluster.name'='<KUBERNETES_CLUSTER_NAME>' \
    --set-string 'stackstate.cluster.authToken=<CLUSTER_AUTH_TOKEN>' \
    --set-string 'stackstate.url'='<STACKSTATE_RECEIVER_API_ADDRESS>' \
    --values values.yaml \
    stackstate-agent stackstate/stackstate-agent

Configure kube-state-metrics (stackstate/stackstate-agent helm chart versions later than and including version 3.5.0)

The kube_state_metrics Python integration has been deprecated and removed in favour of the kube_state_core check. The key difference lies in the deployment of the helm chart. There is no longer a need for running a kube_state_state_metrics pod, as the kube_state_core check is now part of the StackState Agent. The kube_state_core check queries the relevant Kubernetes api endpoints directly, and sends the metrics to StackState, without the need for a separate kube-state-metrics pod. It is therefore also no longer required to set a value for the kubeStateMetrics.url in the values.yaml file.

If an alternative kube-state-metrics pod (i.e. Prometheus) is installed, it can remain and will not be affected by the agent deployment.

Status

To check the status of the Kubernetes integration, check that the StackState Cluster Agent (cluster-agent) pod, StackState Checks Agent pod (checks-agent) and all of the StackState Agent (node-agent) pods have status ready.

❯ kubectl get deployment,daemonset --namespace stackstate

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/stackstate-agent-cluster-agent       1/1     1            1           5h14m
deployment.apps/stackstate-agent-checks-agent        1/1     1            1           5h14m
NAME                                                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/stackstate-agent-node-agent           10        10        10      10           10          <none>          5h14m

Integration details

Data retrieved

The Kubernetes integration retrieves the following data:

Events

Kubernetes events

The Kubernetes integration retrieves all events from the Kubernetes cluster. The table below shows which event category will be assigned to each event type in StackState:

StackState event category
Kubernetes events

Activities

BackOff ContainerGCFailed ExceededGracePeriod FileSystemResizeSuccessful ImageGCFailed Killing NodeAllocatableEnforced NodeNotReady NodeSchedulable Preempting Pulling Pulled Rebooted Scheduled Starting Started SuccessfulAttachVolume SuccessfulDetachVolume SuccessfulMountVolume SuccessfulUnMountVolume VolumeResizeSuccessful

Alerts

NotTriggerScaleUp

Changes

Created (created container) NodeReady SandboxChanged SuccessfulCreate SuccessfulDelete Completed

Others

All other events

Object change events

The Kubernetes integration will detect changes in Kubernetes objects and will create an event of type Element Properties Change with a diff with a YAML representation of the changed object.

Changes will be detected in the following object types:

  • ConfigMap

  • CronJob

  • DaemonSet

  • Deployment

  • Ingress

  • Job

  • Namespace

  • Node

  • PersistentVolume

  • Pod

  • ReplicaSet

  • Secret (a hash of the content will be compared)

  • Service

  • StatefulSet

Note that, to reduce noise of changes, the following object properties won't be compared:

  • metadata

    • managedFields

    • resourceVersion

    • annotations

      • kubectl.kubernetes.io/last-applied-configuration

  • status (except for Node, Pod and PersistentVolume objects)

Metrics

The Kubernetes integration makes all metrics from the Kubernetes cluster available in StackState. Relevant metrics are automatically mapped to the associated components.

All retrieved metrics can be browsed or added to a component as a telemetry stream. Select the data source StackState Metrics and type kubernetes in the Select box to get a full list of all available metrics.

Topology

The Kubernetes integration retrieves components and relations for the Kubernetes cluster.

StackState Agent versions prior to 2.16: Topology information is only gathered from Kubernetes clusters that use the Docker container runtime.

Components

The following Kubernetes topology data is available in StackState as components:

Cluster

Ingress

Namespace

Job

Node

Persistent Volume

Pod

ReplicaSet

Container

Secret

ConfigMap

Service

CronJob

StatefulSet

DaemonSet

Volume

Deployment

Relations

The following relations between components are retrieved:

  • Container → PersistentVolume, Volume

  • CronJob → Job

  • DaemonSet → Pod

  • Deployment → ReplicaSet

  • Job → Pod

  • Ingress → Service

  • Namespace → CronJob, DaemonSet, Deployment, Job, ReplicaSet, Service, StatefulSet

  • Node → Cluster relation

  • Pod → ConfigMap, Container, Deployment, Node, Secret, Volume

  • ReplicaSet → Pod

  • Service → ExternalService, Pod

  • StatefulSet → Pod

  • Direct communication between processes

  • Process → Process communication via Kubernetes service

  • Process → Process communication via headless Kubernetes service

Traces

The Kubernetes integration doesn't retrieve any traces data.

Tags

All tags defined in Kubernetes will be retrieved and added to the associated components and relations in StackState as labels.

The following labels will also be added to imported elements as relevant:

  • image_name

  • kube_cluster_name

  • kube_container_name

  • kube_cronjob

  • kube_daemon_set

  • kube_deployment

  • kube_job

  • kube_namespace

  • kube_replica_set

  • kube_replication_controller

  • kube_stateful_set

  • pod_name

  • pod_phase

REST API endpoints

The StackState Agent talks to the kubelet and kube-state-metrics API.

The StackState Agent and Cluster Agent connect to the Kubernetes API to retrieve cluster wide information and Kubernetes events. The following API endpoints used:

Resource type
REST API endpoint

Cluster > ComponentStatus

GET /api/v1/componentstatuses

Cluster > Event

GET /apis/events.k8s.io/v1/events

Cluster > Namespace

GET /api/v1/namespaces

Cluster > Node

GET /api/v1/nodes

Services > Endpoints

GET /api/v1/namespaces/{namespace}/endpoints

Services > Ingress

GET /apis/networking.k8s.io/v1/namespaces/{namespace}/ingresses

Services > Service

GET /api/v1/namespaces/{namespace}/services

Workloads > CronJob

GET /apis/batch/v1beta1/namespaces/{namespace}/cronjobs

Workloads > DaemonSet

GET /apis/apps/v1/namespaces/{namespace}/daemonsets

Workloads > Deployment

GET /apis/apps/v1/namespaces/{namespace}/deployments

Workloads > Job

GET /apis/batch/v1/namespaces/{namespace}/jobs

Workloads > Pod

GET /api/v1/namespaces/{namespace}/pods

Workloads > ReplicaSet

GET /apis/apps/v1/namespaces/{namespace}/replicasets

Workloads > StatefulSet

GET /apis/apps/v1/namespaces/{namespace}/statefulsets

Config and Storage > ConfigMap

GET /api/v1/namespaces/{namespace}/configmaps

Config and Storage > Secret

GET /api/v1/secrets

Config and Storage > PersistentVolume

GET /api/v1/persistentvolumes

Config and Storage > PersistentVolumeClaimSpec

GET /api/v1/namespaces/{namespace}/persistentvolumeclaims

Config and Storage > VolumeAttachment

GET /apis/storage.k8s.io/v1/volumeattachments

/version

/healthz

Component actions

Action
Available for component types
Description

Show configuration and storage

pods containers

Display the selected pod or container with its configmaps, secrets and volumes

Show dependencies (deep)

deployment replicaset replicationcontroller statefulset daemonset job cronjob pod

Displays all dependencies (up to 6 levels deep) of the selected pod or workload.

Show pods

deployment replicaset replicationcontroller statefulset daemonset job cronjob

Displays the pods for the selected workload.

Show pods and services

namespace

Opens a view for the pods/services in the selected namespace

Show services

namespace

Open a view for the service and ingress components in the selected namespace

Show workloads

namespace

Show workloads in the selected namespace

Details of installed actions can be found in the StackState UI Settings > Actions > Component Actions screen.

Kubernetes views in StackState

When the Kubernetes integration is enabled, the following Kubernetes views are available in StackState for each cluster:

  • Kubernetes - Applications -

  • Kubernetes - Infrastructure -

  • Kubernetes - Namespaces -

  • Kubernetes - Workload Controllers -

Open source

The code for the StackState Agent Kubernetes check is open source and available on GitHub at:

Troubleshooting

Uninstall

To uninstall the Kubernetes StackPack, go to the StackState UI StackPacks > Integrations > Kubernetes screen and click UNINSTALL. All Kubernetes StackPack specific configuration will be removed from StackState.

Release notes

Kubernetes StackPack v3.10.2 (2022-09-28)

  • Improvement: Updated documentation.

Kubernetes StackPack v3.10.1 (2022-09-15)

  • Improvement: Added validation for cluster name

Kubernetes StackPack v3.9.13 (2022-06-21)

  • Bug Fix: Fixed description for services/ingresses view.

Kubernetes StackPack v3.9.12 (2022-06-03)

  • Improvement: Updated documentation.

Kubernetes StackPack v3.9.11 (2022-05-23)

  • Bug Fix: Fixed broken link in integration StackState Agent V2 integration documentation.

Kubernetes StackPack v3.9.10 (2022-04-11)

  • Bug Fix: Show kubernetes view names on StackPack instance

Kubernetes StackPack v3.9.9 (2022-03-02)

  • Improvement: Documentation for agent.containerRuntime.customSocketPath option.

Kubernetes StackPack v3.9.8 (2021-11-30)

  • Bug Fix: Support nodes without instanceId

See also

Data is retrieved by the deployed and then pushed to StackState via the Agent StackPack and the Kubernetes StackPack.

is translated into components and relations.

defined in Kubernetes are added to components and relations in StackState.

is stored and accessible within StackState. Relevant metrics data is mapped to associated components and relations in StackState.

are available in the StackState UI Events Perspective. They're also included in the Event list in the right panel View summary tab and the details tabs - Component details and Direct relation details.

are created for every detected change to spec or metadata in Kubernetes objects

➡️ .

To integrate with other services, a separate instance of the should be deployed on a standalone VM. It isn't currently possible to configure a StackState Agent deployed on a Kubernetes cluster with checks that integrate with other services.

The kubernetes_state check is responsible for gathering metrics from kube-state-metrics and sending them to StackState. The kubernetes_state check runs in the by default and is configured in the .

All are retrieved from the Kubernetes cluster.

StackState Element Properties Change events will be generated for .

You can also see the current YAML definition of the object in the :

For further details, refer to the .

A number of are added to StackState when the Kubernetes StackPack is installed. They're available from the Actions section in the right panel details tab - Component details - when a Kubernetes component is selected or from the component context menu, displayed when you hover the mouse pointer over a Kubernetes component in the Topology Perspective

Troubleshooting steps for any known issues can be found in the .

See the Kubernetes Agent documentation for instructions on from your Kubernetes cluster.

🧩
💠
Deploy StackState Agents and kube-state-metrics
StackState Agent
Kubernetes API documentation (kubernetes.io)
https://github.com/StackVista/stackstate-agent/tree/master/pkg/collector/corechecks/cluster
https://github.com/stackvista/stackstate-agent
https://github.com/StackVista/stackstate-agent-integrations/tree/master/kubernetes
StackState support Knowledge base
Deploy StackState Agent V3, the Cluster Agent and kube-state-metrics
StackState Agent V3 StackPack
StackState Agent Kubernetes check (github.com)
StackState Agent Helm Chart (github.com)
Kubernetes API documentation (kubernetes.io)
Topology data
Tags
Metrics data
Kubernetes events
Object change events
Events
Metrics
Tags
Topology
Kubernetes events
changes detected in a Kubernetes object
StackState core integration
actions
or past
StackState Agents
StackState Checks Agent
StackState Cluster Agent
how to uninstall the StackState Cluster Agent and the StackState Agent
Data flow
Element Properties Change event
Kubernetes Component properties
Add a Kubernetes metrics stream to a component
Component properties